Sep 09 13:26:05.056679 ip-10-0-129-84 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 13:26:05.511542 ip-10-0-129-84 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 13:26:05.511542 ip-10-0-129-84 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 13:26:05.511542 ip-10-0-129-84 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 13:26:05.511542 ip-10-0-129-84 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 13:26:05.511542 ip-10-0-129-84 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 13:26:05.514427 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.514321    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 13:26:05.520095 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520060    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 13:26:05.520238 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520227    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 13:26:05.520321 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520293    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 13:26:05.520394 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520386    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 13:26:05.520451 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520445    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 13:26:05.520509 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520503    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 13:26:05.520568 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520555    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 13:26:05.520628 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520612    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 13:26:05.520682 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520629    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 13:26:05.520682 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520638    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 13:26:05.520682 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520643    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 13:26:05.520682 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520648    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 13:26:05.520682 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520653    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 13:26:05.520682 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520658    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 13:26:05.520682 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520663    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 13:26:05.520682 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520668    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 13:26:05.520682 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520673    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 13:26:05.520682 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520678    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 13:26:05.520682 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520682    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 13:26:05.520682 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520687    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 13:26:05.521153 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520692    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 13:26:05.521153 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520704    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 13:26:05.521153 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520709    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 13:26:05.521153 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520714    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 13:26:05.521153 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520718    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 13:26:05.521153 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520723    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 13:26:05.521153 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520730    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 13:26:05.521153 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520735    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 13:26:05.521153 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520739    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 13:26:05.521153 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520748    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 13:26:05.521153 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520752    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 13:26:05.521153 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520779    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 13:26:05.521153 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520895    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 13:26:05.521153 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520901    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 13:26:05.521153 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520905    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 13:26:05.521153 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520909    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 13:26:05.521153 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520914    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 13:26:05.521153 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520918    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 13:26:05.521153 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520923    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 13:26:05.521153 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520927    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 13:26:05.522002 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520931    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 13:26:05.522002 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520936    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 13:26:05.522002 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520940    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 13:26:05.522002 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520944    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 13:26:05.522002 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520948    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 13:26:05.522002 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520953    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 13:26:05.522002 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520958    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 13:26:05.522002 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520962    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 13:26:05.522002 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520970    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 13:26:05.522002 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520977    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 13:26:05.522002 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520983    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 13:26:05.522002 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520988    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 13:26:05.522002 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520994    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 13:26:05.522002 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.520999    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 13:26:05.522002 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521005    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 13:26:05.522002 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521009    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 13:26:05.522002 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521015    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 13:26:05.522002 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521022    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 13:26:05.522002 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521028    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 13:26:05.522002 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521032    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 13:26:05.522855 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521037    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 13:26:05.522855 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521041    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 13:26:05.522855 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521046    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 13:26:05.522855 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521050    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 13:26:05.522855 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521063    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 13:26:05.522855 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521067    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 13:26:05.522855 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521072    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 13:26:05.522855 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521077    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 13:26:05.522855 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521081    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 13:26:05.522855 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521086    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 13:26:05.522855 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521090    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 13:26:05.522855 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521094    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 13:26:05.522855 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521099    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 13:26:05.522855 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521103    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 13:26:05.522855 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521108    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 13:26:05.522855 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521112    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 13:26:05.522855 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521117    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 13:26:05.522855 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521121    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 13:26:05.522855 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521127    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 13:26:05.522855 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521131    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 13:26:05.523680 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521135    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 13:26:05.523680 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521139    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 13:26:05.523680 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521144    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 13:26:05.523680 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521151    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 13:26:05.523680 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521156    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 13:26:05.523680 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521163    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 13:26:05.523680 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521170    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 13:26:05.523680 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521175    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 13:26:05.523680 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521180    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 13:26:05.523680 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521184    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 13:26:05.523680 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521189    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 13:26:05.523680 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521194    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 13:26:05.523680 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521199    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 13:26:05.523680 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521203    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 13:26:05.523680 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521208    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 13:26:05.523680 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521212    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 13:26:05.523680 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521217    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 13:26:05.523680 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521221    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 13:26:05.523680 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.521226    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 13:26:05.524526 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522522    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 13:26:05.524526 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522534    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 13:26:05.524526 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522540    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 13:26:05.524526 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522545    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 13:26:05.524526 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522551    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 13:26:05.524526 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522555    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 13:26:05.524526 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522560    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 13:26:05.524526 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522564    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 13:26:05.524526 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522568    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 13:26:05.524526 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522573    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 13:26:05.524526 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522578    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 13:26:05.524526 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522582    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 13:26:05.524526 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522586    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 13:26:05.524526 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522591    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 13:26:05.524526 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522595    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 13:26:05.524526 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522599    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 13:26:05.524526 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522605    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 13:26:05.524526 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522612    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 13:26:05.524526 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522619    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 13:26:05.525465 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522624    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 13:26:05.525465 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522629    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 13:26:05.525465 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522634    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 13:26:05.525465 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522638    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 13:26:05.525465 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522643    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 13:26:05.525465 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522648    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 13:26:05.525465 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522652    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 13:26:05.525465 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522658    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 13:26:05.525465 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522662    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 13:26:05.525465 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522667    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 13:26:05.525465 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522671    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 13:26:05.525465 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522676    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 13:26:05.525465 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522680    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 13:26:05.525465 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522684    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 13:26:05.525465 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522688    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 13:26:05.525465 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522692    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 13:26:05.525465 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522696    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 13:26:05.525465 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522701    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 13:26:05.525465 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522705    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 13:26:05.525465 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522710    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 13:26:05.525985 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522715    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 13:26:05.525985 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522719    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 13:26:05.525985 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522724    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 13:26:05.525985 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522728    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 13:26:05.525985 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522732    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 13:26:05.525985 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522736    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 13:26:05.525985 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522740    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 13:26:05.525985 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522745    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 13:26:05.525985 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522749    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 13:26:05.525985 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522754    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 13:26:05.525985 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522758    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 13:26:05.525985 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522764    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 13:26:05.525985 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522770    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 13:26:05.525985 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522775    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 13:26:05.525985 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522779    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 13:26:05.525985 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522784    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 13:26:05.525985 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522788    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 13:26:05.525985 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522793    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 13:26:05.525985 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522797    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 13:26:05.525985 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522801    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 13:26:05.526511 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522805    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 13:26:05.526511 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522811    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 13:26:05.526511 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522818    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 13:26:05.526511 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522822    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 13:26:05.526511 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522827    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 13:26:05.526511 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522831    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 13:26:05.526511 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522836    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 13:26:05.526511 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522840    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 13:26:05.526511 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522845    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 13:26:05.526511 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522849    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 13:26:05.526511 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522853    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 13:26:05.526511 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522857    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 13:26:05.526511 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522861    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 13:26:05.526511 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522866    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 13:26:05.526511 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522870    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 13:26:05.526511 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522873    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 13:26:05.526511 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522877    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 13:26:05.526511 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522881    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 13:26:05.526511 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522885    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 13:26:05.526990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522893    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 13:26:05.526990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522899    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 13:26:05.526990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522904    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 13:26:05.526990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522909    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 13:26:05.526990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522914    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 13:26:05.526990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522920    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 13:26:05.526990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522924    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 13:26:05.526990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522928    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 13:26:05.526990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522933    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 13:26:05.526990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522937    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 13:26:05.526990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522942    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 13:26:05.526990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522946    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 13:26:05.526990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522951    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 13:26:05.526990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522955    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 13:26:05.526990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522959    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 13:26:05.526990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522963    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 13:26:05.526990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522967    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 13:26:05.526990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522972    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 13:26:05.526990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522976    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 13:26:05.527498 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522981    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 13:26:05.527498 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.522985    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 13:26:05.527498 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523684    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 13:26:05.527498 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523692    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 13:26:05.527498 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523696    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 13:26:05.527498 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523700    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 13:26:05.527498 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523704    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 13:26:05.527498 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523709    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 13:26:05.527498 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523713    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 13:26:05.527498 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523718    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 13:26:05.527498 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523722    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 13:26:05.527498 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523726    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 13:26:05.527498 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523730    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 13:26:05.527498 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523735    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 13:26:05.527498 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523739    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 13:26:05.527498 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523744    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 13:26:05.527498 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523748    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 13:26:05.527498 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523752    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 13:26:05.527498 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523758    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 13:26:05.527990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523762    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 13:26:05.527990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523767    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 13:26:05.527990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523781    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 13:26:05.527990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523785    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 13:26:05.527990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523789    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 13:26:05.527990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523793    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 13:26:05.527990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523797    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 13:26:05.527990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523804    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 13:26:05.527990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523810    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 13:26:05.527990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523815    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 13:26:05.527990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523823    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 13:26:05.527990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523828    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 13:26:05.527990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523833    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 13:26:05.527990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523837    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 13:26:05.527990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523842    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 13:26:05.527990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523846    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 13:26:05.527990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523850    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 13:26:05.527990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523854    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 13:26:05.527990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523858    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 13:26:05.527990 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523862    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 13:26:05.528527 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523867    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 13:26:05.528527 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523873    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 13:26:05.528527 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523877    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 13:26:05.528527 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523883    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 13:26:05.528527 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523889    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 13:26:05.528527 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523894    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 13:26:05.528527 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523899    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 13:26:05.528527 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523903    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 13:26:05.528527 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523908    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 13:26:05.528527 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523912    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 13:26:05.528527 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523917    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 13:26:05.528527 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523921    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 13:26:05.528527 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523934    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 13:26:05.528527 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523938    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 13:26:05.528527 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523942    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 13:26:05.528527 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523947    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 13:26:05.528527 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523951    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 13:26:05.528527 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523955    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 13:26:05.528527 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523960    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 13:26:05.528527 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523964    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 13:26:05.529020 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523969    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 13:26:05.529020 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523973    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 13:26:05.529020 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523979    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 13:26:05.529020 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523984    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 13:26:05.529020 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523988    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 13:26:05.529020 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523992    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 13:26:05.529020 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.523996    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 13:26:05.529020 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524000    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 13:26:05.529020 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524005    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 13:26:05.529020 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524009    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 13:26:05.529020 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524013    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 13:26:05.529020 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524017    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 13:26:05.529020 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524021    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 13:26:05.529020 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524025    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 13:26:05.529020 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524029    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 13:26:05.529020 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524033    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 13:26:05.529020 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524037    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 13:26:05.529020 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524044    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 13:26:05.529020 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524048    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 13:26:05.529020 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524052    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 13:26:05.529539 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524057    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 13:26:05.529539 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524062    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 13:26:05.529539 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524066    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 13:26:05.529539 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524071    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 13:26:05.529539 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524076    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 13:26:05.529539 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524081    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 13:26:05.529539 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524085    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 13:26:05.529539 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524089    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 13:26:05.529539 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524093    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 13:26:05.529539 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524100    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 13:26:05.529539 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524106    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 13:26:05.529539 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524111    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 13:26:05.529539 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524116    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 13:26:05.529539 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524120    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 13:26:05.529539 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524126    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 13:26:05.529539 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524130    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 13:26:05.529539 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524134    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 13:26:05.529539 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524138    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 13:26:05.529539 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524142    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 13:26:05.530057 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524146    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 13:26:05.530057 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524151    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 13:26:05.530057 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.524155    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 13:26:05.530057 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524271    2566 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 13:26:05.530057 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524342    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 13:26:05.530057 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524354    2566 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 13:26:05.530057 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524361    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 13:26:05.530057 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524369    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 13:26:05.530057 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524374    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 13:26:05.530057 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524381    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 13:26:05.530057 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524388    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 13:26:05.530057 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524393    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 13:26:05.530057 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524398    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 13:26:05.530057 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524406    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 13:26:05.530057 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524412    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 13:26:05.530057 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524417    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 13:26:05.530057 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524422    2566 flags.go:64] FLAG: --cgroup-root=""
Sep 09 13:26:05.530057 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524427    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 13:26:05.530057 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524431    2566 flags.go:64] FLAG: --client-ca-file=""
Sep 09 13:26:05.530057 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524437    2566 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 13:26:05.530057 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524442    2566 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 13:26:05.530057 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524449    2566 flags.go:64] FLAG: --cluster-domain=""
Sep 09 13:26:05.530057 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524454    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 13:26:05.530636 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524459    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 13:26:05.530636 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524464    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 13:26:05.530636 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524470    2566 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 13:26:05.530636 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524476    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 13:26:05.530636 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524482    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 13:26:05.530636 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524487    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 13:26:05.530636 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524493    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 13:26:05.530636 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524498    2566 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 13:26:05.530636 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524503    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 13:26:05.530636 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524508    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 13:26:05.530636 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524513    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 13:26:05.530636 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524518    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 13:26:05.530636 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524524    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 13:26:05.530636 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524528    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 13:26:05.530636 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524533    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 13:26:05.530636 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524537    2566 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 13:26:05.530636 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524542    2566 flags.go:64] FLAG: --enable-server="true"
Sep 09 13:26:05.530636 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524546    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 13:26:05.530636 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524555    2566 flags.go:64] FLAG: --event-burst="100"
Sep 09 13:26:05.530636 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524560    2566 flags.go:64] FLAG: --event-qps="50"
Sep 09 13:26:05.530636 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524565    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 13:26:05.530636 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524569    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 13:26:05.530636 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524576    2566 flags.go:64] FLAG: --eviction-hard=""
Sep 09 13:26:05.530636 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524582    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 13:26:05.531208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524587    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 13:26:05.531208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524592    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 13:26:05.531208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524598    2566 flags.go:64] FLAG: --eviction-soft=""
Sep 09 13:26:05.531208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524604    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 13:26:05.531208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524609    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 13:26:05.531208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524615    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 13:26:05.531208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524620    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 13:26:05.531208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524625    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 13:26:05.531208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524630    2566 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 13:26:05.531208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524634    2566 flags.go:64] FLAG: --feature-gates=""
Sep 09 13:26:05.531208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524641    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 13:26:05.531208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524646    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 13:26:05.531208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524652    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 13:26:05.531208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524657    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 13:26:05.531208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524662    2566 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 13:26:05.531208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524667    2566 flags.go:64] FLAG: --help="false"
Sep 09 13:26:05.531208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524671    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-129-84.ec2.internal"
Sep 09 13:26:05.531208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524678    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 13:26:05.531208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524682    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 13:26:05.531208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524687    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 13:26:05.531208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524693    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 13:26:05.531208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524699    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 13:26:05.531208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524704    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 13:26:05.531208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524708    2566 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 13:26:05.531846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524713    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 13:26:05.531846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524718    2566 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 13:26:05.531846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524722    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 13:26:05.531846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524727    2566 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 13:26:05.531846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524733    2566 flags.go:64] FLAG: --kube-reserved=""
Sep 09 13:26:05.531846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524737    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 13:26:05.531846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524744    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 13:26:05.531846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524749    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 13:26:05.531846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524753    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 13:26:05.531846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524758    2566 flags.go:64] FLAG: --lock-file=""
Sep 09 13:26:05.531846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524763    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 13:26:05.531846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524767    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 13:26:05.531846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524772    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 13:26:05.531846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524781    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 13:26:05.531846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524788    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 13:26:05.531846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524794    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 13:26:05.531846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524798    2566 flags.go:64] FLAG: --logging-format="text"
Sep 09 13:26:05.531846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524803    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 13:26:05.531846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524808    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 13:26:05.531846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524814    2566 flags.go:64] FLAG: --manifest-url=""
Sep 09 13:26:05.531846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524819    2566 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 13:26:05.531846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524826    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 13:26:05.531846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524831    2566 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 13:26:05.531846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524838    2566 flags.go:64] FLAG: --max-pods="110"
Sep 09 13:26:05.531846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524843    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 13:26:05.532509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524847    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 13:26:05.532509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524852    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 13:26:05.532509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524857    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 13:26:05.532509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524862    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 13:26:05.532509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524867    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 13:26:05.532509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524872    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 13:26:05.532509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524883    2566 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 13:26:05.532509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524888    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 13:26:05.532509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524892    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 13:26:05.532509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524897    2566 flags.go:64] FLAG: --pod-cidr=""
Sep 09 13:26:05.532509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524902    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 13:26:05.532509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524911    2566 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 13:26:05.532509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524915    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 13:26:05.532509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524922    2566 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 13:26:05.532509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524927    2566 flags.go:64] FLAG: --port="10250"
Sep 09 13:26:05.532509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524932    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 13:26:05.532509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524937    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f632430c0a8d5162"
Sep 09 13:26:05.532509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524942    2566 flags.go:64] FLAG: --qos-reserved=""
Sep 09 13:26:05.532509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524947    2566 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 13:26:05.532509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524952    2566 flags.go:64] FLAG: --register-node="true"
Sep 09 13:26:05.532509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524957    2566 flags.go:64] FLAG: --register-with-taints=""
Sep 09 13:26:05.532509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524963    2566 flags.go:64] FLAG: --registry-burst="10"
Sep 09 13:26:05.532509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524968    2566 flags.go:64] FLAG: --registry-qps="5"
Sep 09 13:26:05.532509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524973    2566 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 13:26:05.533105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524978    2566 flags.go:64] FLAG: --reserved-memory=""
Sep 09 13:26:05.533105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524984    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 13:26:05.533105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524989    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 13:26:05.533105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524994    2566 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 13:26:05.533105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.524999    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 13:26:05.533105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525005    2566 flags.go:64] FLAG: --runonce="false"
Sep 09 13:26:05.533105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525009    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 13:26:05.533105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525015    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 13:26:05.533105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525020    2566 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 13:26:05.533105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525025    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 13:26:05.533105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525029    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 13:26:05.533105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525034    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 13:26:05.533105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525039    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 13:26:05.533105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525044    2566 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 13:26:05.533105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525048    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 13:26:05.533105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525053    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 13:26:05.533105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525057    2566 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 13:26:05.533105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525062    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 13:26:05.533105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525067    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 13:26:05.533105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525072    2566 flags.go:64] FLAG: --system-cgroups=""
Sep 09 13:26:05.533105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525077    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 13:26:05.533105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525087    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 13:26:05.533105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525092    2566 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 13:26:05.533105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525097    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 13:26:05.533105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525103    2566 flags.go:64] FLAG: --tls-min-version=""
Sep 09 13:26:05.533784 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525107    2566 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 13:26:05.533784 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525112    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 13:26:05.533784 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525117    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 13:26:05.533784 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525121    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 13:26:05.533784 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525127    2566 flags.go:64] FLAG: --v="2"
Sep 09 13:26:05.533784 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525134    2566 flags.go:64] FLAG: --version="false"
Sep 09 13:26:05.533784 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525142    2566 flags.go:64] FLAG: --vmodule=""
Sep 09 13:26:05.533784 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525148    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 13:26:05.533784 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525153    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 13:26:05.533784 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525361    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 13:26:05.533784 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525368    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 13:26:05.533784 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525371    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 13:26:05.533784 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525374    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 13:26:05.533784 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525377    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 13:26:05.533784 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525380    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 13:26:05.533784 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525384    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 13:26:05.533784 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525386    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 13:26:05.533784 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525391    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 13:26:05.533784 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525395    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 13:26:05.533784 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525398    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 13:26:05.533784 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525401    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 13:26:05.533784 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525404    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 13:26:05.534342 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525407    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 13:26:05.534342 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525410    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 13:26:05.534342 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525413    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 13:26:05.534342 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525415    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 13:26:05.534342 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525418    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 13:26:05.534342 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525420    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 13:26:05.534342 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525425    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 13:26:05.534342 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525427    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 13:26:05.534342 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525430    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 13:26:05.534342 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525432    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 13:26:05.534342 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525435    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 13:26:05.534342 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525438    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 13:26:05.534342 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525442    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 13:26:05.534342 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525445    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 13:26:05.534342 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525448    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 13:26:05.534342 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525451    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 13:26:05.534342 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525453    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 13:26:05.534342 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525457    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 13:26:05.534342 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525460    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 13:26:05.534342 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525463    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 13:26:05.534847 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525465    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 13:26:05.534847 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525468    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 13:26:05.534847 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525473    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 13:26:05.534847 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525477    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 13:26:05.534847 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525480    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 13:26:05.534847 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525484    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 13:26:05.534847 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525487    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 13:26:05.534847 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525489    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 13:26:05.534847 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525492    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 13:26:05.534847 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525495    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 13:26:05.534847 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525497    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 13:26:05.534847 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525500    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 13:26:05.534847 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525503    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 13:26:05.534847 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525505    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 13:26:05.534847 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525508    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 13:26:05.534847 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525510    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 13:26:05.534847 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525513    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 13:26:05.534847 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525516    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 13:26:05.534847 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525520    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 13:26:05.535344 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525523    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 13:26:05.535344 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525527    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 13:26:05.535344 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525530    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 13:26:05.535344 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525533    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 13:26:05.535344 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525536    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 13:26:05.535344 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525540    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 13:26:05.535344 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525542    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 13:26:05.535344 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525545    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 13:26:05.535344 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525548    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 13:26:05.535344 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525551    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 13:26:05.535344 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525553    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 13:26:05.535344 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525558    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 13:26:05.535344 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525561    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 13:26:05.535344 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525564    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 13:26:05.535344 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525567    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 13:26:05.535344 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525570    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 13:26:05.535344 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525572    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 13:26:05.535344 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525575    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 13:26:05.535344 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525578    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 13:26:05.535827 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525581    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 13:26:05.535827 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525583    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 13:26:05.535827 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525586    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 13:26:05.535827 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525588    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 13:26:05.535827 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525591    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 13:26:05.535827 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525593    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 13:26:05.535827 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525596    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 13:26:05.535827 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525599    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 13:26:05.535827 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525601    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 13:26:05.535827 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525603    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 13:26:05.535827 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525606    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 13:26:05.535827 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525609    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 13:26:05.535827 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525613    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 13:26:05.535827 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525616    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 13:26:05.535827 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525618    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 13:26:05.535827 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525621    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 13:26:05.535827 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525624    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 13:26:05.535827 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525626    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 13:26:05.535827 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525630    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 13:26:05.535827 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525632    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 13:26:05.536335 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525635    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 13:26:05.536335 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525638    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 13:26:05.536335 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525640    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 13:26:05.536335 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525643    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 13:26:05.536335 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525645    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 13:26:05.536335 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525649    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 13:26:05.536335 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525652    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 13:26:05.536335 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.525655    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 13:26:05.536335 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.525660    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 13:26:05.536335 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.534748    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 13:26:05.536335 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.534765    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 13:26:05.536335 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.535695    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 13:26:05.536335 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.535703    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 13:26:05.536335 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535761    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 13:26:05.536335 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535765    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 13:26:05.536722 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535777    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 13:26:05.536722 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535781    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 13:26:05.536722 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535784    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 13:26:05.536722 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535787    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 13:26:05.536722 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535790    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 13:26:05.536722 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535792    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 13:26:05.536722 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535795    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 13:26:05.536722 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535798    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 13:26:05.536722 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535801    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 13:26:05.536722 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535804    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 13:26:05.536722 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535807    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 13:26:05.536722 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535810    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 13:26:05.536722 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535813    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 13:26:05.536722 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535815    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 13:26:05.536722 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535818    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 13:26:05.536722 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535821    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 13:26:05.536722 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535826    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 13:26:05.536722 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535831    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 13:26:05.536722 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535834    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 13:26:05.536722 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535837    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 13:26:05.537212 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535840    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 13:26:05.537212 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535844    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 13:26:05.537212 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535849    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 13:26:05.537212 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535852    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 13:26:05.537212 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535855    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 13:26:05.537212 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535858    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 13:26:05.537212 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535861    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 13:26:05.537212 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535864    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 13:26:05.537212 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535867    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 13:26:05.537212 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535870    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 13:26:05.537212 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535872    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 13:26:05.537212 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535876    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 13:26:05.537212 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535878    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 13:26:05.537212 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535881    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 13:26:05.537212 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535885    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 13:26:05.537212 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535889    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 13:26:05.537212 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535891    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 13:26:05.537212 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535894    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 13:26:05.537212 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535897    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 13:26:05.537212 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535899    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 13:26:05.537746 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535902    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 13:26:05.537746 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535905    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 13:26:05.537746 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535908    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 13:26:05.537746 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535911    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 13:26:05.537746 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535914    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 13:26:05.537746 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535917    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 13:26:05.537746 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535919    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 13:26:05.537746 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535922    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 13:26:05.537746 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535924    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 13:26:05.537746 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535927    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 13:26:05.537746 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535929    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 13:26:05.537746 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535932    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 13:26:05.537746 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535934    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 13:26:05.537746 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535937    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 13:26:05.537746 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535939    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 13:26:05.537746 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535943    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 13:26:05.537746 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535945    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 13:26:05.537746 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535948    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 13:26:05.537746 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535950    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 13:26:05.538214 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535953    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 13:26:05.538214 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535956    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 13:26:05.538214 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535959    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 13:26:05.538214 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535961    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 13:26:05.538214 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535964    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 13:26:05.538214 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535966    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 13:26:05.538214 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535969    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 13:26:05.538214 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535972    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 13:26:05.538214 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535974    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 13:26:05.538214 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535977    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 13:26:05.538214 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535980    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 13:26:05.538214 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535982    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 13:26:05.538214 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535985    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 13:26:05.538214 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535988    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 13:26:05.538214 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535991    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 13:26:05.538214 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535994    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 13:26:05.538214 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.535997    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 13:26:05.538214 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536000    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 13:26:05.538214 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536003    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 13:26:05.538701 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536005    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 13:26:05.538701 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536008    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 13:26:05.538701 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536011    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 13:26:05.538701 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536013    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 13:26:05.538701 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536016    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 13:26:05.538701 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536018    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 13:26:05.538701 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536021    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 13:26:05.538701 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536024    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 13:26:05.538701 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536026    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 13:26:05.538701 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536029    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 13:26:05.538701 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536032    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 13:26:05.538701 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536035    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 13:26:05.538701 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536037    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 13:26:05.538701 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536041    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 13:26:05.538701 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536045    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 13:26:05.538701 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536047    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 13:26:05.538701 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536050    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 13:26:05.538701 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536053    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 13:26:05.538701 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536056    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 13:26:05.539155 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.536061    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 13:26:05.539155 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536182    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 13:26:05.539155 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536186    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 13:26:05.539155 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536189    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 13:26:05.539155 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536192    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 13:26:05.539155 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536195    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 13:26:05.539155 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536198    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 13:26:05.539155 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536201    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 13:26:05.539155 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536204    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 13:26:05.539155 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536207    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 13:26:05.539155 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536211    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 13:26:05.539155 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536213    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 13:26:05.539155 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536216    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 13:26:05.539155 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536219    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 13:26:05.539155 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536222    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 13:26:05.539553 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536224    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 13:26:05.539553 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536227    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 13:26:05.539553 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536230    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 13:26:05.539553 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536233    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 13:26:05.539553 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536236    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 13:26:05.539553 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536239    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 13:26:05.539553 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536241    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 13:26:05.539553 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536244    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 13:26:05.539553 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536246    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 13:26:05.539553 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536251    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 13:26:05.539553 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536254    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 13:26:05.539553 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536258    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 13:26:05.539553 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536261    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 13:26:05.539553 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536264    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 13:26:05.539553 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536266    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 13:26:05.539553 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536269    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 13:26:05.539553 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536272    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 13:26:05.539553 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536275    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 13:26:05.539553 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536277    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 13:26:05.540030 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536280    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 13:26:05.540030 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536282    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 13:26:05.540030 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536285    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 13:26:05.540030 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536287    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 13:26:05.540030 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536290    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 13:26:05.540030 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536293    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 13:26:05.540030 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536296    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 13:26:05.540030 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536310    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 13:26:05.540030 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536315    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 13:26:05.540030 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536318    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 13:26:05.540030 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536321    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 13:26:05.540030 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536324    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 13:26:05.540030 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536326    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 13:26:05.540030 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536329    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 13:26:05.540030 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536332    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 13:26:05.540030 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536335    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 13:26:05.540030 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536338    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 13:26:05.540030 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536340    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 13:26:05.540030 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536343    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 13:26:05.540030 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536346    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 13:26:05.540545 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536348    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 13:26:05.540545 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536351    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 13:26:05.540545 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536353    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 13:26:05.540545 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536357    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 13:26:05.540545 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536359    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 13:26:05.540545 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536362    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 13:26:05.540545 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536364    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 13:26:05.540545 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536367    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 13:26:05.540545 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536369    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 13:26:05.540545 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536372    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 13:26:05.540545 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536374    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 13:26:05.540545 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536377    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 13:26:05.540545 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536379    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 13:26:05.540545 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536382    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 13:26:05.540545 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536384    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 13:26:05.540545 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536387    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 13:26:05.540545 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536390    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 13:26:05.540545 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536392    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 13:26:05.540545 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536397    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 13:26:05.540545 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536400    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 13:26:05.541025 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536403    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 13:26:05.541025 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536405    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 13:26:05.541025 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536408    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 13:26:05.541025 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536410    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 13:26:05.541025 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536413    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 13:26:05.541025 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536415    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 13:26:05.541025 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536418    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 13:26:05.541025 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536420    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 13:26:05.541025 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536423    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 13:26:05.541025 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536425    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 13:26:05.541025 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536428    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 13:26:05.541025 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536430    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 13:26:05.541025 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536433    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 13:26:05.541025 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536435    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 13:26:05.541025 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536438    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 13:26:05.541025 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536440    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 13:26:05.541025 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536443    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 13:26:05.541025 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536445    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 13:26:05.541025 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536448    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 13:26:05.541025 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536450    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 13:26:05.541678 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536454    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 13:26:05.541678 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536457    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 13:26:05.541678 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536460    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 13:26:05.541678 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536463    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 13:26:05.541678 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536465    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 13:26:05.541678 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:05.536468    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 13:26:05.541678 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.536473    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 13:26:05.541678 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.536617    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 13:26:05.541678 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.539115    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 13:26:05.541678 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.540268    2566 server.go:1019] "Starting client certificate rotation"
Sep 09 13:26:05.541678 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.540387    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 13:26:05.541678 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.540442    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 13:26:05.543214 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.543200    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 13:26:05.548494 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.548474    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 13:26:05.560006 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.559979    2566 log.go:25] "Validated CRI v1 runtime API"
Sep 09 13:26:05.563759 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.563739    2566 log.go:25] "Validated CRI v1 image API"
Sep 09 13:26:05.564631 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.564614    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 13:26:05.568200 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.568172    2566 fs.go:135] Filesystem UUIDs: map[168ecc34-2fb1-4eee-9a43-ae132e4e120b:/dev/nvme0n1p3 51781395-f60a-4047-85ba-93a470f03e61:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 09 13:26:05.568275 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.568199    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 13:26:05.573658 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.573546    2566 manager.go:216] Machine: {Timestamp:2026-09-09 13:26:05.572091861 +0000 UTC m=+0.386452039 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095104 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22bef8139d46e56e921893c2e94b6d SystemUUID:ec22bef8-139d-46e5-6e92-1893c2e94b6d BootID:2b49620e-e5cc-4359-9dba-9bb5b80ce94f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:56:08:28:8d:0d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:56:08:28:8d:0d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:e9:a1:81:79:fc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 13:26:05.573658 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.573654    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 13:26:05.573824 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.573812    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 13:26:05.577583 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.577550    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 13:26:05.577790 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.577589    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-84.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 13:26:05.578504 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.578491    2566 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 13:26:05.578542 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.578506    2566 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 13:26:05.578542 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.578514    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 13:26:05.579453 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.579440    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 13:26:05.581376 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.581363    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 13:26:05.585904 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.585890    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 13:26:05.586053 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.586043    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 13:26:05.586090 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.586076    2566 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 13:26:05.586090 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.586089    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 13:26:05.586145 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.586102    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 13:26:05.586145 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.586112    2566 kubelet.go:398] "Adding apiserver pod source"
Sep 09 13:26:05.586145 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.586123    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 13:26:05.586599 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.586584    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 13:26:05.586689 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.586606    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 13:26:05.589169 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.589152    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 13:26:05.591705 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.591675    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 13:26:05.592501 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.592488    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 13:26:05.592561 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.592518    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 13:26:05.593391 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.593375    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 13:26:05.593463 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.593400    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 13:26:05.593463 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.593408    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 13:26:05.593463 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.593415    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 13:26:05.593463 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.593425    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 13:26:05.593463 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.593430    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 13:26:05.593463 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.593444    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 13:26:05.593463 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.593452    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 13:26:05.593463 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.593458    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 13:26:05.593744 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.593483    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 13:26:05.593744 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.593498    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 13:26:05.593744 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.593504    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 13:26:05.595230 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.595216    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 13:26:05.595230 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.595227    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 13:26:05.599696 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.599679    2566 server.go:1268] "Started kubelet"
Sep 09 13:26:05.599836 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.599765    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 13:26:05.604773 ip-10-0-129-84 systemd[1]: Started Kubernetes Kubelet.
Sep 09 13:26:05.605150 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.605128    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-84.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 13:26:05.605607 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:05.605508    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-84.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 13:26:05.606095 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.599790    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 13:26:05.606095 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.606011    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 13:26:05.606564 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:05.606504    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 13:26:05.606788 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:05.606765    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 13:26:05.607164 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.607146    2566 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 13:26:05.609144 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.609124    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 13:26:05.614173 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.614154    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 13:26:05.614822 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:05.613629    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-84.ec2.internal.18d3a9f964cdcfa9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-84.ec2.internal,UID:ip-10-0-129-84.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-84.ec2.internal,},FirstTimestamp:2026-09-09 13:26:05.599379369 +0000 UTC m=+0.413739548,LastTimestamp:2026-09-09 13:26:05.599379369 +0000 UTC m=+0.413739548,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-84.ec2.internal,}"
Sep 09 13:26:05.614917 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.614811    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 13:26:05.614962 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:05.614912    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Sep 09 13:26:05.615736 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.615716    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 13:26:05.615736 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.615730    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 13:26:05.615736 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.615736    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 13:26:05.615917 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.615877    2566 factory.go:55] Registering systemd factory
Sep 09 13:26:05.615917 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.615894    2566 factory.go:223] Registration of the systemd container factory successfully
Sep 09 13:26:05.616010 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.615941    2566 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 13:26:05.616010 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.615951    2566 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 13:26:05.616145 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.616131    2566 factory.go:153] Registering CRI-O factory
Sep 09 13:26:05.616145 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.616145    2566 factory.go:223] Registration of the crio container factory successfully
Sep 09 13:26:05.616298 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.616196    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 13:26:05.616298 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.616215    2566 factory.go:103] Registering Raw factory
Sep 09 13:26:05.616298 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.616230    2566 manager.go:1196] Started watching for new ooms in manager
Sep 09 13:26:05.617427 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.617411    2566 manager.go:349] Starting recovery of all containers
Sep 09 13:26:05.622103 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:05.622066    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 13:26:05.622254 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:05.622200    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-84.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 13:26:05.629727 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.629705    2566 manager.go:354] Recovery completed
Sep 09 13:26:05.634554 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.634540    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 13:26:05.636635 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.636608    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 13:26:05.636724 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.636647    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 13:26:05.636724 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.636659    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientPID"
Sep 09 13:26:05.637268 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.637253    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 13:26:05.637268 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.637262    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 13:26:05.637438 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.637265    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 13:26:05.637438 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.637320    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 13:26:05.638095 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.638079    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 13:26:05.638156 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.638112    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 13:26:05.638156 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.638126    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientPID"
Sep 09 13:26:05.638979 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:05.638889    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-84.ec2.internal.18d3a9f96706360c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-84.ec2.internal,UID:ip-10-0-129-84.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-84.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-84.ec2.internal,},FirstTimestamp:2026-09-09 13:26:05.636630028 +0000 UTC m=+0.450990207,LastTimestamp:2026-09-09 13:26:05.636630028 +0000 UTC m=+0.450990207,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-84.ec2.internal,}"
Sep 09 13:26:05.639429 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.639416    2566 policy_none.go:49] "None policy: Start"
Sep 09 13:26:05.639429 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.639431    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 13:26:05.639543 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.639441    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 13:26:05.641379 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.641366    2566 policy_none.go:47] "Start"
Sep 09 13:26:05.654996 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:05.654907    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-84.ec2.internal.18d3a9f967068ab3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-84.ec2.internal,UID:ip-10-0-129-84.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-84.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-84.ec2.internal,},FirstTimestamp:2026-09-09 13:26:05.636651699 +0000 UTC m=+0.451011878,LastTimestamp:2026-09-09 13:26:05.636651699 +0000 UTC m=+0.451011878,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-84.ec2.internal,}"
Sep 09 13:26:05.674415 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:05.674276    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-84.ec2.internal.18d3a9f96706bf2d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-84.ec2.internal,UID:ip-10-0-129-84.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-84.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-84.ec2.internal,},FirstTimestamp:2026-09-09 13:26:05.636665133 +0000 UTC m=+0.451025314,LastTimestamp:2026-09-09 13:26:05.636665133 +0000 UTC m=+0.451025314,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-84.ec2.internal,}"
Sep 09 13:26:05.694995 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.674990    2566 manager.go:337] "Starting Device Plugin manager"
Sep 09 13:26:05.694995 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:05.675026    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 13:26:05.694995 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.675037    2566 server.go:87] "Starting device plugin registration server"
Sep 09 13:26:05.694995 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.675323    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 13:26:05.694995 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.675338    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 13:26:05.694995 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.675488    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 13:26:05.694995 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.675545    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 13:26:05.694995 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.675550    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 13:26:05.694995 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:05.676176    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 13:26:05.694995 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:05.676231    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-84.ec2.internal\" not found"
Sep 09 13:26:05.694995 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.684794    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cnhb7"
Sep 09 13:26:05.694995 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.687318    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 13:26:05.694995 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.688516    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 13:26:05.694995 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.688561    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 13:26:05.694995 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.688584    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 13:26:05.694995 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:05.688621    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 13:26:05.694995 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:05.692783    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 09 13:26:05.694995 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:05.692753    2566 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-84.ec2.internal.18d3a9f96706360c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-84.ec2.internal.18d3a9f96706360c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-84.ec2.internal,UID:ip-10-0-129-84.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-84.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-84.ec2.internal,},FirstTimestamp:2026-09-09 13:26:05.636630028 +0000 UTC m=+0.450990207,LastTimestamp:2026-09-09 13:26:05.638094506 +0000 UTC m=+0.452454686,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-84.ec2.internal,}"
Sep 09 13:26:05.697708 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.697689    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cnhb7"
Sep 09 13:26:05.701105 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:05.701038    2566 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-84.ec2.internal.18d3a9f967068ab3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-84.ec2.internal.18d3a9f967068ab3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-84.ec2.internal,UID:ip-10-0-129-84.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-84.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-84.ec2.internal,},FirstTimestamp:2026-09-09 13:26:05.636651699 +0000 UTC m=+0.451011878,LastTimestamp:2026-09-09 13:26:05.638119154 +0000 UTC m=+0.452479335,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-84.ec2.internal,}"
Sep 09 13:26:05.775979 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.775878    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 13:26:05.777131 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.777113    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 13:26:05.777241 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.777150    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 13:26:05.777241 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.777164    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientPID"
Sep 09 13:26:05.777241 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.777193    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-84.ec2.internal"
Sep 09 13:26:05.789023 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.788986    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal"]
Sep 09 13:26:05.789167 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.789074    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 13:26:05.790760 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.790742    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 13:26:05.790878 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.790772    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 13:26:05.790878 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.790787    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientPID"
Sep 09 13:26:05.791146 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.791133    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-84.ec2.internal"
Sep 09 13:26:05.791179 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:05.791153    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-84.ec2.internal\": node \"ip-10-0-129-84.ec2.internal\" not found"
Sep 09 13:26:05.793169 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.793157    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 13:26:05.793297 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.793280    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Sep 09 13:26:05.793428 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.793334    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 13:26:05.793994 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.793970    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 13:26:05.793994 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.793998    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 13:26:05.794127 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.794008    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientPID"
Sep 09 13:26:05.794127 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.794023    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 13:26:05.794127 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.794049    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 13:26:05.794127 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.794060    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientPID"
Sep 09 13:26:05.796443 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.796426    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal"
Sep 09 13:26:05.796513 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.796457    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 13:26:05.799970 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.799950    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 13:26:05.800067 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.799987    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 13:26:05.800067 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.800003    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientPID"
Sep 09 13:26:05.813015 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:05.812991    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-84.ec2.internal\" not found" node="ip-10-0-129-84.ec2.internal"
Sep 09 13:26:05.816950 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.816927    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/641a1c3349dfc0923275c7074bcf5214-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal\" (UID: \"641a1c3349dfc0923275c7074bcf5214\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Sep 09 13:26:05.817082 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.816954    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/641a1c3349dfc0923275c7074bcf5214-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal\" (UID: \"641a1c3349dfc0923275c7074bcf5214\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Sep 09 13:26:05.817082 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.816976    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bad615a00a972079824050bf66b78c89-config\") pod \"kube-apiserver-proxy-ip-10-0-129-84.ec2.internal\" (UID: \"bad615a00a972079824050bf66b78c89\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal"
Sep 09 13:26:05.819071 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:05.819055    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-84.ec2.internal\" not found" node="ip-10-0-129-84.ec2.internal"
Sep 09 13:26:05.826503 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:05.826485    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Sep 09 13:26:05.917739 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.917707    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/641a1c3349dfc0923275c7074bcf5214-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal\" (UID: \"641a1c3349dfc0923275c7074bcf5214\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Sep 09 13:26:05.917739 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.917737    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bad615a00a972079824050bf66b78c89-config\") pod \"kube-apiserver-proxy-ip-10-0-129-84.ec2.internal\" (UID: \"bad615a00a972079824050bf66b78c89\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal"
Sep 09 13:26:05.917932 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.917754    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/641a1c3349dfc0923275c7074bcf5214-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal\" (UID: \"641a1c3349dfc0923275c7074bcf5214\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Sep 09 13:26:05.917932 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.917835    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/641a1c3349dfc0923275c7074bcf5214-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal\" (UID: \"641a1c3349dfc0923275c7074bcf5214\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Sep 09 13:26:05.917932 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.917841    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bad615a00a972079824050bf66b78c89-config\") pod \"kube-apiserver-proxy-ip-10-0-129-84.ec2.internal\" (UID: \"bad615a00a972079824050bf66b78c89\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal"
Sep 09 13:26:05.917932 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:05.917879    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/641a1c3349dfc0923275c7074bcf5214-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal\" (UID: \"641a1c3349dfc0923275c7074bcf5214\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Sep 09 13:26:05.926833 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:05.926809    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Sep 09 13:26:06.027484 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:06.027421    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Sep 09 13:26:07.571699 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:07.571516    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Sep 09 13:26:07.578116 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:07.578097    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal"
Sep 09 13:26:07.582221 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:07.582197    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Sep 09 13:26:07.682460 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:07.682417    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Sep 09 13:26:07.782972 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:07.782890    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Sep 09 13:26:07.883536 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:07.883502    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Sep 09 13:26:07.953494 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:07.953468    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 13:26:07.983756 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:07.983730    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Sep 09 13:26:07.994953 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:07.994913    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 13:26:07.995173 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:07.995148    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 13:26:08.084219 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:08.084149    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Sep 09 13:26:08.118098 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:08.118055    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad615a00a972079824050bf66b78c89.slice/crio-daa89dced1dc7bf51674cf0882e10b75153d08a0d59aa8b544b4e06117524e23 WatchSource:0}: Error finding container daa89dced1dc7bf51674cf0882e10b75153d08a0d59aa8b544b4e06117524e23: Status 404 returned error can't find the container with id daa89dced1dc7bf51674cf0882e10b75153d08a0d59aa8b544b4e06117524e23
Sep 09 13:26:08.118310 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:08.118290    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641a1c3349dfc0923275c7074bcf5214.slice/crio-c304a02eb158aa9018d9d6ee4ae2718d22a0a8734b57dc2424932aada827df85 WatchSource:0}: Error finding container c304a02eb158aa9018d9d6ee4ae2718d22a0a8734b57dc2424932aada827df85: Status 404 returned error can't find the container with id c304a02eb158aa9018d9d6ee4ae2718d22a0a8734b57dc2424932aada827df85
Sep 09 13:26:08.122198 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:08.122179    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 13:26:08.141099 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:08.141070    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 13:26:08.146901 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:08.146844    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal" event={"ID":"bad615a00a972079824050bf66b78c89","Type":"ContainerStarted","Data":"daa89dced1dc7bf51674cf0882e10b75153d08a0d59aa8b544b4e06117524e23"}
Sep 09 13:26:08.148059 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:08.148032    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal" event={"ID":"641a1c3349dfc0923275c7074bcf5214","Type":"ContainerStarted","Data":"c304a02eb158aa9018d9d6ee4ae2718d22a0a8734b57dc2424932aada827df85"}
Sep 09 13:26:08.155240 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:08.155197    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 13:21:07 +0000 UTC" deadline="2028-04-01 05:16:01.379081193 +0000 UTC"
Sep 09 13:26:08.155240 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:08.155240    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13671h49m53.223844411s"
Sep 09 13:26:08.184717 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:08.184683    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Sep 09 13:26:08.274182 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:08.274149    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 13:26:08.275462 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:08.275445    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 13:26:08.275557 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:08.275476    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 13:26:08.275557 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:08.275487    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientPID"
Sep 09 13:26:08.285143 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:08.285113    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Sep 09 13:26:08.292044 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:08.292017    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 13:26:08.385478 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:08.385371    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Sep 09 13:26:08.485929 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:08.485876    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Sep 09 13:26:08.586692 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:08.586647    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Sep 09 13:26:08.622151 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:08.622123    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 13:26:08.670154 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:08.670075    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Sep 09 13:26:08.781429 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:08.781389    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 13:26:08.782651 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:08.782613    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal"
Sep 09 13:26:08.806100 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:08.806068    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 13:26:09.041657 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.041578    2566 apiserver.go:52] "Watching apiserver"
Sep 09 13:26:09.048187 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.048145    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 13:26:09.049520 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.049490    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7","openshift-dns/node-resolver-sw9qv","openshift-multus/multus-gwqft","openshift-multus/network-metrics-daemon-q5dxg","openshift-network-operator/iptables-alerter-bt7v9","kube-system/konnectivity-agent-q5njn","openshift-cluster-node-tuning-operator/tuned-nqs45","openshift-image-registry/node-ca-qrq28","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal","openshift-multus/multus-additional-cni-plugins-q6cft","openshift-network-diagnostics/network-check-target-zbqpk","openshift-ovn-kubernetes/ovnkube-node-m4s7g","kube-system/global-pull-secret-syncer-plsz4"]
Sep 09 13:26:09.051939 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.051916    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q5njn"
Sep 09 13:26:09.054340 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.054319    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 13:26:09.054540 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.054524    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 13:26:09.054667 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.054526    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5qtgs\""
Sep 09 13:26:09.054730 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.054654    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.057889 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.056985    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sw9qv"
Sep 09 13:26:09.057889 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.057482    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 13:26:09.057889 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.057503    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 13:26:09.057889 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.057607    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 13:26:09.057889 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.057665    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 13:26:09.058226 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.058201    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fcclt\""
Sep 09 13:26:09.058296 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.058236    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 13:26:09.058483 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.058466    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 13:26:09.058843 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.058733    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 13:26:09.059455 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.058974    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rg4zm\""
Sep 09 13:26:09.059455 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.059110    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.059455 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.059120    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 13:26:09.061341 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.061322    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:09.061437 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:09.061415    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5dxg" podUID="9ddda0bf-9853-4015-a006-c9e44d47acd5"
Sep 09 13:26:09.061886 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.061867    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-h4qgs\""
Sep 09 13:26:09.061886 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.061879    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 13:26:09.062036 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.061868    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 13:26:09.062036 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.061868    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 13:26:09.062036 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.061933    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 13:26:09.063486 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.063470    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bt7v9"
Sep 09 13:26:09.065642 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.065625    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 13:26:09.065913 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.065890    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 13:26:09.066142 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.066123    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gnd48\""
Sep 09 13:26:09.066524 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.066126    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 13:26:09.066524 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.066355    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qrq28"
Sep 09 13:26:09.068377 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.068353    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 13:26:09.068526 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.068506    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 13:26:09.069464 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.068671    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 13:26:09.069464 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.069077    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4bbxn\""
Sep 09 13:26:09.070521 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.070492    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 13:26:09.070685 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.070667    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:09.070769 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:09.070748    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plsz4" podUID="0ecc5e12-af64-4305-83ed-37552f51b76c"
Sep 09 13:26:09.073850 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.073830    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7"
Sep 09 13:26:09.076012 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.075994    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 13:26:09.076105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.076070    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2nspd\""
Sep 09 13:26:09.076312 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.076297    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.076839 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.076807    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 13:26:09.076931 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.076900    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 13:26:09.078727 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.078667    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q6cft"
Sep 09 13:26:09.078808 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.078762    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 13:26:09.079001 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.078668    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 13:26:09.079070 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.079030    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t4f89\""
Sep 09 13:26:09.081409 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.081206    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:26:09.081409 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:09.081264    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbqpk" podUID="49845a79-9f07-4b8c-9f9d-ad2a1035d184"
Sep 09 13:26:09.081551 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.081533    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-x7bq4\""
Sep 09 13:26:09.081949 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.081930    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 13:26:09.082168 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082148    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-host-run-k8s-cni-cncf-io\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.082244 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082184    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-etc-kubernetes\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.082244 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082207    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jmtl\" (UniqueName: \"kubernetes.io/projected/e61e15eb-3d0d-4a92-a91a-0987c9fd4eeb-kube-api-access-7jmtl\") pod \"node-ca-qrq28\" (UID: \"e61e15eb-3d0d-4a92-a91a-0987c9fd4eeb\") " pod="openshift-image-registry/node-ca-qrq28"
Sep 09 13:26:09.082244 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082231    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-host-slash\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.082414 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082255    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-host-cni-bin\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.082414 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082287    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ecc5e12-af64-4305-83ed-37552f51b76c-original-pull-secret\") pod \"global-pull-secret-syncer-plsz4\" (UID: \"0ecc5e12-af64-4305-83ed-37552f51b76c\") " pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:09.082414 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082301    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 13:26:09.082414 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082311    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/625682e0-c96b-4b33-b0b8-5b9fb93d4804-etc-selinux\") pod \"aws-ebs-csi-driver-node-flwf7\" (UID: \"625682e0-c96b-4b33-b0b8-5b9fb93d4804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7"
Sep 09 13:26:09.082414 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082343    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-log-socket\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.082414 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082367    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f06384ff-80d7-4e3a-a9bd-4925d55d6824-tmp-dir\") pod \"node-resolver-sw9qv\" (UID: \"f06384ff-80d7-4e3a-a9bd-4925d55d6824\") " pod="openshift-dns/node-resolver-sw9qv"
Sep 09 13:26:09.082684 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082415    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-multus-conf-dir\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.082684 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082439    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-host-run-multus-certs\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.082684 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082457    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-ovnkube-config\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.082684 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082475    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-ovn-node-metrics-cert\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.082684 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082500    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-host-var-lib-cni-multus\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.082684 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082529    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0ecc5e12-af64-4305-83ed-37552f51b76c-dbus\") pod \"global-pull-secret-syncer-plsz4\" (UID: \"0ecc5e12-af64-4305-83ed-37552f51b76c\") " pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:09.082684 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082561    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0ecc5e12-af64-4305-83ed-37552f51b76c-kubelet-config\") pod \"global-pull-secret-syncer-plsz4\" (UID: \"0ecc5e12-af64-4305-83ed-37552f51b76c\") " pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:09.082684 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082594    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e61e15eb-3d0d-4a92-a91a-0987c9fd4eeb-host\") pod \"node-ca-qrq28\" (UID: \"e61e15eb-3d0d-4a92-a91a-0987c9fd4eeb\") " pod="openshift-image-registry/node-ca-qrq28"
Sep 09 13:26:09.082684 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082626    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-host-run-netns\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.082684 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082650    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-run-systemd\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.082684 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082673    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-run-openvswitch\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.083261 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082696    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7f5s\" (UniqueName: \"kubernetes.io/projected/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-kube-api-access-v7f5s\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.083261 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082713    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-host-var-lib-kubelet\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.083261 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082732    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.083261 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082756    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-env-overrides\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.083261 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082778    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ddda0bf-9853-4015-a006-c9e44d47acd5-metrics-certs\") pod \"network-metrics-daemon-q5dxg\" (UID: \"9ddda0bf-9853-4015-a006-c9e44d47acd5\") " pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:09.083261 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082805    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-run-ovn\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.083261 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082846    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-host-run-ovn-kubernetes\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.083261 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082875    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e61e15eb-3d0d-4a92-a91a-0987c9fd4eeb-serviceca\") pod \"node-ca-qrq28\" (UID: \"e61e15eb-3d0d-4a92-a91a-0987c9fd4eeb\") " pod="openshift-image-registry/node-ca-qrq28"
Sep 09 13:26:09.083261 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082897    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f06384ff-80d7-4e3a-a9bd-4925d55d6824-hosts-file\") pod \"node-resolver-sw9qv\" (UID: \"f06384ff-80d7-4e3a-a9bd-4925d55d6824\") " pod="openshift-dns/node-resolver-sw9qv"
Sep 09 13:26:09.083261 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082913    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-host-var-lib-cni-bin\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.083261 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082936    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-multus-daemon-config\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.083261 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082970    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-host-kubelet\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.083261 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.082999    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng9pg\" (UniqueName: \"kubernetes.io/projected/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-kube-api-access-ng9pg\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.083261 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.083043    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ab687c9c-b837-4387-a696-7f24afd7711f-agent-certs\") pod \"konnectivity-agent-q5njn\" (UID: \"ab687c9c-b837-4387-a696-7f24afd7711f\") " pod="kube-system/konnectivity-agent-q5njn"
Sep 09 13:26:09.083261 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.083082    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/625682e0-c96b-4b33-b0b8-5b9fb93d4804-device-dir\") pod \"aws-ebs-csi-driver-node-flwf7\" (UID: \"625682e0-c96b-4b33-b0b8-5b9fb93d4804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7"
Sep 09 13:26:09.083261 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.083103    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtgxj\" (UniqueName: \"kubernetes.io/projected/2caadc2a-aabb-440f-8f3b-98ceb69052cd-kube-api-access-dtgxj\") pod \"iptables-alerter-bt7v9\" (UID: \"2caadc2a-aabb-440f-8f3b-98ceb69052cd\") " pod="openshift-network-operator/iptables-alerter-bt7v9"
Sep 09 13:26:09.084023 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.083132    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-etc-openvswitch\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.084023 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.083149    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-multus-cni-dir\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.084023 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.083163    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/625682e0-c96b-4b33-b0b8-5b9fb93d4804-kubelet-dir\") pod \"aws-ebs-csi-driver-node-flwf7\" (UID: \"625682e0-c96b-4b33-b0b8-5b9fb93d4804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7"
Sep 09 13:26:09.084023 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.083191    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/625682e0-c96b-4b33-b0b8-5b9fb93d4804-registration-dir\") pod \"aws-ebs-csi-driver-node-flwf7\" (UID: \"625682e0-c96b-4b33-b0b8-5b9fb93d4804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7"
Sep 09 13:26:09.084023 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.083219    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-cni-binary-copy\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.084023 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.083244    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-host-run-netns\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.084023 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.083266    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/625682e0-c96b-4b33-b0b8-5b9fb93d4804-socket-dir\") pod \"aws-ebs-csi-driver-node-flwf7\" (UID: \"625682e0-c96b-4b33-b0b8-5b9fb93d4804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7"
Sep 09 13:26:09.084023 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.083305    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxbt2\" (UniqueName: \"kubernetes.io/projected/625682e0-c96b-4b33-b0b8-5b9fb93d4804-kube-api-access-pxbt2\") pod \"aws-ebs-csi-driver-node-flwf7\" (UID: \"625682e0-c96b-4b33-b0b8-5b9fb93d4804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7"
Sep 09 13:26:09.084023 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.083336    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gvp5\" (UniqueName: \"kubernetes.io/projected/9ddda0bf-9853-4015-a006-c9e44d47acd5-kube-api-access-6gvp5\") pod \"network-metrics-daemon-q5dxg\" (UID: \"9ddda0bf-9853-4015-a006-c9e44d47acd5\") " pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:09.084023 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.083360    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-system-cni-dir\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.084023 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.083397    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-cnibin\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.084023 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.083419    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-node-log\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.084023 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.083438    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/625682e0-c96b-4b33-b0b8-5b9fb93d4804-sys-fs\") pod \"aws-ebs-csi-driver-node-flwf7\" (UID: \"625682e0-c96b-4b33-b0b8-5b9fb93d4804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7"
Sep 09 13:26:09.084023 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.083460    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2caadc2a-aabb-440f-8f3b-98ceb69052cd-iptables-alerter-script\") pod \"iptables-alerter-bt7v9\" (UID: \"2caadc2a-aabb-440f-8f3b-98ceb69052cd\") " pod="openshift-network-operator/iptables-alerter-bt7v9"
Sep 09 13:26:09.084023 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.083510    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-var-lib-openvswitch\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.084023 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.083536    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-host-cni-netd\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.084545 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.083580    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9t9q\" (UniqueName: \"kubernetes.io/projected/f06384ff-80d7-4e3a-a9bd-4925d55d6824-kube-api-access-z9t9q\") pod \"node-resolver-sw9qv\" (UID: \"f06384ff-80d7-4e3a-a9bd-4925d55d6824\") " pod="openshift-dns/node-resolver-sw9qv"
Sep 09 13:26:09.084545 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.083603    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-hostroot\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.084545 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.083624    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2caadc2a-aabb-440f-8f3b-98ceb69052cd-host-slash\") pod \"iptables-alerter-bt7v9\" (UID: \"2caadc2a-aabb-440f-8f3b-98ceb69052cd\") " pod="openshift-network-operator/iptables-alerter-bt7v9"
Sep 09 13:26:09.084545 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.083647    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-systemd-units\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.084545 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.083673    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-ovnkube-script-lib\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.084545 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.083712    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-multus-socket-dir-parent\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.084545 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.083737    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ab687c9c-b837-4387-a696-7f24afd7711f-konnectivity-ca\") pod \"konnectivity-agent-q5njn\" (UID: \"ab687c9c-b837-4387-a696-7f24afd7711f\") " pod="kube-system/konnectivity-agent-q5njn"
Sep 09 13:26:09.085364 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.085272    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 13:26:09.128059 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.128030    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-prsk6"
Sep 09 13:26:09.150609 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.150448    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-prsk6"
Sep 09 13:26:09.171476 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.171445    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 13:26:09.184749 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.184711    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-host-kubelet\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.184749 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.184750    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng9pg\" (UniqueName: \"kubernetes.io/projected/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-kube-api-access-ng9pg\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.185005 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.184779    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ab687c9c-b837-4387-a696-7f24afd7711f-agent-certs\") pod \"konnectivity-agent-q5njn\" (UID: \"ab687c9c-b837-4387-a696-7f24afd7711f\") " pod="kube-system/konnectivity-agent-q5njn"
Sep 09 13:26:09.185005 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.184802    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/625682e0-c96b-4b33-b0b8-5b9fb93d4804-device-dir\") pod \"aws-ebs-csi-driver-node-flwf7\" (UID: \"625682e0-c96b-4b33-b0b8-5b9fb93d4804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7"
Sep 09 13:26:09.185005 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.184839    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtgxj\" (UniqueName: \"kubernetes.io/projected/2caadc2a-aabb-440f-8f3b-98ceb69052cd-kube-api-access-dtgxj\") pod \"iptables-alerter-bt7v9\" (UID: \"2caadc2a-aabb-440f-8f3b-98ceb69052cd\") " pod="openshift-network-operator/iptables-alerter-bt7v9"
Sep 09 13:26:09.185005 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.184846    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-host-kubelet\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.185005 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.184862    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-etc-openvswitch\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.185005 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.184910    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-etc-openvswitch\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.185005 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.184933    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/625682e0-c96b-4b33-b0b8-5b9fb93d4804-device-dir\") pod \"aws-ebs-csi-driver-node-flwf7\" (UID: \"625682e0-c96b-4b33-b0b8-5b9fb93d4804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7"
Sep 09 13:26:09.185282 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185047    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-multus-cni-dir\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.185282 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185075    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/625682e0-c96b-4b33-b0b8-5b9fb93d4804-kubelet-dir\") pod \"aws-ebs-csi-driver-node-flwf7\" (UID: \"625682e0-c96b-4b33-b0b8-5b9fb93d4804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7"
Sep 09 13:26:09.185282 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185093    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/625682e0-c96b-4b33-b0b8-5b9fb93d4804-registration-dir\") pod \"aws-ebs-csi-driver-node-flwf7\" (UID: \"625682e0-c96b-4b33-b0b8-5b9fb93d4804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7"
Sep 09 13:26:09.185282 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185121    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6dde766b-3e64-4762-ab24-25b30b1b4465-system-cni-dir\") pod \"multus-additional-cni-plugins-q6cft\" (UID: \"6dde766b-3e64-4762-ab24-25b30b1b4465\") " pod="openshift-multus/multus-additional-cni-plugins-q6cft"
Sep 09 13:26:09.185282 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185140    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6dde766b-3e64-4762-ab24-25b30b1b4465-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q6cft\" (UID: \"6dde766b-3e64-4762-ab24-25b30b1b4465\") " pod="openshift-multus/multus-additional-cni-plugins-q6cft"
Sep 09 13:26:09.185282 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185156    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-cni-binary-copy\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.185282 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185171    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-host-run-netns\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.185282 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185170    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/625682e0-c96b-4b33-b0b8-5b9fb93d4804-registration-dir\") pod \"aws-ebs-csi-driver-node-flwf7\" (UID: \"625682e0-c96b-4b33-b0b8-5b9fb93d4804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7"
Sep 09 13:26:09.185282 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185212    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-host-run-netns\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.185673 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185274    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 13:26:09.185673 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185286    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/625682e0-c96b-4b33-b0b8-5b9fb93d4804-socket-dir\") pod \"aws-ebs-csi-driver-node-flwf7\" (UID: \"625682e0-c96b-4b33-b0b8-5b9fb93d4804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7"
Sep 09 13:26:09.185673 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185304    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/625682e0-c96b-4b33-b0b8-5b9fb93d4804-kubelet-dir\") pod \"aws-ebs-csi-driver-node-flwf7\" (UID: \"625682e0-c96b-4b33-b0b8-5b9fb93d4804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7"
Sep 09 13:26:09.185673 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185338    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxbt2\" (UniqueName: \"kubernetes.io/projected/625682e0-c96b-4b33-b0b8-5b9fb93d4804-kube-api-access-pxbt2\") pod \"aws-ebs-csi-driver-node-flwf7\" (UID: \"625682e0-c96b-4b33-b0b8-5b9fb93d4804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7"
Sep 09 13:26:09.185673 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185371    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gvp5\" (UniqueName: \"kubernetes.io/projected/9ddda0bf-9853-4015-a006-c9e44d47acd5-kube-api-access-6gvp5\") pod \"network-metrics-daemon-q5dxg\" (UID: \"9ddda0bf-9853-4015-a006-c9e44d47acd5\") " pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:09.185673 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185342    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-multus-cni-dir\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.185673 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185429    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6dde766b-3e64-4762-ab24-25b30b1b4465-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q6cft\" (UID: \"6dde766b-3e64-4762-ab24-25b30b1b4465\") " pod="openshift-multus/multus-additional-cni-plugins-q6cft"
Sep 09 13:26:09.185673 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185450    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/625682e0-c96b-4b33-b0b8-5b9fb93d4804-socket-dir\") pod \"aws-ebs-csi-driver-node-flwf7\" (UID: \"625682e0-c96b-4b33-b0b8-5b9fb93d4804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7"
Sep 09 13:26:09.185673 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185484    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-lib-modules\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.185673 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185518    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d65cbad0-4588-49d4-bac1-5303af169027-tmp\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.185673 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185564    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-system-cni-dir\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.185673 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185614    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-cnibin\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.185673 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185639    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-system-cni-dir\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.185673 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185644    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-etc-kubernetes\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.186320 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185682    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-552v9\" (UniqueName: \"kubernetes.io/projected/49845a79-9f07-4b8c-9f9d-ad2a1035d184-kube-api-access-552v9\") pod \"network-check-target-zbqpk\" (UID: \"49845a79-9f07-4b8c-9f9d-ad2a1035d184\") " pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:26:09.186320 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185698    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-cnibin\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.186320 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185717    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-node-log\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.186320 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185724    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-cni-binary-copy\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.186320 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185739    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/625682e0-c96b-4b33-b0b8-5b9fb93d4804-sys-fs\") pod \"aws-ebs-csi-driver-node-flwf7\" (UID: \"625682e0-c96b-4b33-b0b8-5b9fb93d4804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7"
Sep 09 13:26:09.186320 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185765    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6dde766b-3e64-4762-ab24-25b30b1b4465-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q6cft\" (UID: \"6dde766b-3e64-4762-ab24-25b30b1b4465\") " pod="openshift-multus/multus-additional-cni-plugins-q6cft"
Sep 09 13:26:09.186320 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185772    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-node-log\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.186320 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185785    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2caadc2a-aabb-440f-8f3b-98ceb69052cd-iptables-alerter-script\") pod \"iptables-alerter-bt7v9\" (UID: \"2caadc2a-aabb-440f-8f3b-98ceb69052cd\") " pod="openshift-network-operator/iptables-alerter-bt7v9"
Sep 09 13:26:09.186320 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185845    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/625682e0-c96b-4b33-b0b8-5b9fb93d4804-sys-fs\") pod \"aws-ebs-csi-driver-node-flwf7\" (UID: \"625682e0-c96b-4b33-b0b8-5b9fb93d4804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7"
Sep 09 13:26:09.186320 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185843    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-var-lib-openvswitch\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.186320 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185874    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-var-lib-openvswitch\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.186320 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185881    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-host-cni-netd\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.186320 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185907    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9t9q\" (UniqueName: \"kubernetes.io/projected/f06384ff-80d7-4e3a-a9bd-4925d55d6824-kube-api-access-z9t9q\") pod \"node-resolver-sw9qv\" (UID: \"f06384ff-80d7-4e3a-a9bd-4925d55d6824\") " pod="openshift-dns/node-resolver-sw9qv"
Sep 09 13:26:09.186320 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185929    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-hostroot\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.186320 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185956    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6dde766b-3e64-4762-ab24-25b30b1b4465-cnibin\") pod \"multus-additional-cni-plugins-q6cft\" (UID: \"6dde766b-3e64-4762-ab24-25b30b1b4465\") " pod="openshift-multus/multus-additional-cni-plugins-q6cft"
Sep 09 13:26:09.186320 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185977    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-host-cni-netd\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.186320 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.185984    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2caadc2a-aabb-440f-8f3b-98ceb69052cd-host-slash\") pod \"iptables-alerter-bt7v9\" (UID: \"2caadc2a-aabb-440f-8f3b-98ceb69052cd\") " pod="openshift-network-operator/iptables-alerter-bt7v9"
Sep 09 13:26:09.187146 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186008    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-systemd-units\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.187146 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186033    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-ovnkube-script-lib\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.187146 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186036    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2caadc2a-aabb-440f-8f3b-98ceb69052cd-host-slash\") pod \"iptables-alerter-bt7v9\" (UID: \"2caadc2a-aabb-440f-8f3b-98ceb69052cd\") " pod="openshift-network-operator/iptables-alerter-bt7v9"
Sep 09 13:26:09.187146 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186008    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-hostroot\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.187146 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186054    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-systemd-units\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.187146 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186060    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-multus-socket-dir-parent\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.187146 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186103    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ab687c9c-b837-4387-a696-7f24afd7711f-konnectivity-ca\") pod \"konnectivity-agent-q5njn\" (UID: \"ab687c9c-b837-4387-a696-7f24afd7711f\") " pod="kube-system/konnectivity-agent-q5njn"
Sep 09 13:26:09.187146 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186113    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-multus-socket-dir-parent\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.187146 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186127    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-etc-modprobe-d\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.187146 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186149    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d65cbad0-4588-49d4-bac1-5303af169027-etc-tuned\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.187146 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186174    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-host-run-k8s-cni-cncf-io\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.187146 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186202    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-etc-kubernetes\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.187146 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186227    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76w5k\" (UniqueName: \"kubernetes.io/projected/6dde766b-3e64-4762-ab24-25b30b1b4465-kube-api-access-76w5k\") pod \"multus-additional-cni-plugins-q6cft\" (UID: \"6dde766b-3e64-4762-ab24-25b30b1b4465\") " pod="openshift-multus/multus-additional-cni-plugins-q6cft"
Sep 09 13:26:09.187146 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186230    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-host-run-k8s-cni-cncf-io\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.187146 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186251    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jmtl\" (UniqueName: \"kubernetes.io/projected/e61e15eb-3d0d-4a92-a91a-0987c9fd4eeb-kube-api-access-7jmtl\") pod \"node-ca-qrq28\" (UID: \"e61e15eb-3d0d-4a92-a91a-0987c9fd4eeb\") " pod="openshift-image-registry/node-ca-qrq28"
Sep 09 13:26:09.187146 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186283    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-host-slash\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.187146 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186292    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-etc-kubernetes\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.187924 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186310    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-host-cni-bin\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.187924 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186335    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ecc5e12-af64-4305-83ed-37552f51b76c-original-pull-secret\") pod \"global-pull-secret-syncer-plsz4\" (UID: \"0ecc5e12-af64-4305-83ed-37552f51b76c\") " pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:09.187924 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186358    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-etc-systemd\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.187924 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186381    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/625682e0-c96b-4b33-b0b8-5b9fb93d4804-etc-selinux\") pod \"aws-ebs-csi-driver-node-flwf7\" (UID: \"625682e0-c96b-4b33-b0b8-5b9fb93d4804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7"
Sep 09 13:26:09.187924 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186405    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-log-socket\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.187924 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186403    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2caadc2a-aabb-440f-8f3b-98ceb69052cd-iptables-alerter-script\") pod \"iptables-alerter-bt7v9\" (UID: \"2caadc2a-aabb-440f-8f3b-98ceb69052cd\") " pod="openshift-network-operator/iptables-alerter-bt7v9"
Sep 09 13:26:09.187924 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186431    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f06384ff-80d7-4e3a-a9bd-4925d55d6824-tmp-dir\") pod \"node-resolver-sw9qv\" (UID: \"f06384ff-80d7-4e3a-a9bd-4925d55d6824\") " pod="openshift-dns/node-resolver-sw9qv"
Sep 09 13:26:09.187924 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186475    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-multus-conf-dir\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.187924 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186506    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-host-run-multus-certs\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.187924 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186533    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-etc-sysconfig\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.187924 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186557    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-var-lib-kubelet\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.187924 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:09.186569    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 13:26:09.187924 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186599    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-ovnkube-config\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.187924 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186624    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-ovn-node-metrics-cert\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.187924 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186637    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-ovnkube-script-lib\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.187924 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:09.186665    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ecc5e12-af64-4305-83ed-37552f51b76c-original-pull-secret podName:0ecc5e12-af64-4305-83ed-37552f51b76c nodeName:}" failed. No retries permitted until 2026-09-09 13:26:09.68662609 +0000 UTC m=+3.046475920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ecc5e12-af64-4305-83ed-37552f51b76c-original-pull-secret") pod "global-pull-secret-syncer-plsz4" (UID: "0ecc5e12-af64-4305-83ed-37552f51b76c") : object "kube-system"/"original-pull-secret" not registered
Sep 09 13:26:09.187924 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186698    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ab687c9c-b837-4387-a696-7f24afd7711f-konnectivity-ca\") pod \"konnectivity-agent-q5njn\" (UID: \"ab687c9c-b837-4387-a696-7f24afd7711f\") " pod="kube-system/konnectivity-agent-q5njn"
Sep 09 13:26:09.188552 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186705    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-host-slash\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.188552 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186711    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f06384ff-80d7-4e3a-a9bd-4925d55d6824-tmp-dir\") pod \"node-resolver-sw9qv\" (UID: \"f06384ff-80d7-4e3a-a9bd-4925d55d6824\") " pod="openshift-dns/node-resolver-sw9qv"
Sep 09 13:26:09.188552 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186747    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/625682e0-c96b-4b33-b0b8-5b9fb93d4804-etc-selinux\") pod \"aws-ebs-csi-driver-node-flwf7\" (UID: \"625682e0-c96b-4b33-b0b8-5b9fb93d4804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7"
Sep 09 13:26:09.188552 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186753    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-host-cni-bin\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.188552 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186794    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-host-var-lib-cni-multus\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.188552 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186802    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-log-socket\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.188552 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186836    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-multus-conf-dir\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.188552 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186847    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0ecc5e12-af64-4305-83ed-37552f51b76c-dbus\") pod \"global-pull-secret-syncer-plsz4\" (UID: \"0ecc5e12-af64-4305-83ed-37552f51b76c\") " pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:09.188552 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186869    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-host-var-lib-cni-multus\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.188552 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186873    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-host-run-multus-certs\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.188552 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186878    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-etc-sysctl-d\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.188552 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186927    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0ecc5e12-af64-4305-83ed-37552f51b76c-dbus\") pod \"global-pull-secret-syncer-plsz4\" (UID: \"0ecc5e12-af64-4305-83ed-37552f51b76c\") " pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:09.188552 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186927    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-host\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.188552 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186961    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8mbp\" (UniqueName: \"kubernetes.io/projected/d65cbad0-4588-49d4-bac1-5303af169027-kube-api-access-k8mbp\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.188552 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.186988    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0ecc5e12-af64-4305-83ed-37552f51b76c-kubelet-config\") pod \"global-pull-secret-syncer-plsz4\" (UID: \"0ecc5e12-af64-4305-83ed-37552f51b76c\") " pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:09.188552 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187012    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e61e15eb-3d0d-4a92-a91a-0987c9fd4eeb-host\") pod \"node-ca-qrq28\" (UID: \"e61e15eb-3d0d-4a92-a91a-0987c9fd4eeb\") " pod="openshift-image-registry/node-ca-qrq28"
Sep 09 13:26:09.188552 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187037    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-host-run-netns\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.188552 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187048    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0ecc5e12-af64-4305-83ed-37552f51b76c-kubelet-config\") pod \"global-pull-secret-syncer-plsz4\" (UID: \"0ecc5e12-af64-4305-83ed-37552f51b76c\") " pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:09.189220 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187066    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-run-systemd\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.189220 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187091    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-run-openvswitch\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.189220 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187094    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-host-run-netns\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.189220 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187065    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e61e15eb-3d0d-4a92-a91a-0987c9fd4eeb-host\") pod \"node-ca-qrq28\" (UID: \"e61e15eb-3d0d-4a92-a91a-0987c9fd4eeb\") " pod="openshift-image-registry/node-ca-qrq28"
Sep 09 13:26:09.189220 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187107    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-run-systemd\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.189220 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187221    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-run-openvswitch\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.189220 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187251    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7f5s\" (UniqueName: \"kubernetes.io/projected/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-kube-api-access-v7f5s\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.189220 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187281    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-host-var-lib-kubelet\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.189220 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187305    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-run\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.189220 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187311    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-ovnkube-config\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.189220 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187335    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.189220 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187362    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-env-overrides\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.189220 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187362    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-host-var-lib-kubelet\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.189220 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187393    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.189220 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187401    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ddda0bf-9853-4015-a006-c9e44d47acd5-metrics-certs\") pod \"network-metrics-daemon-q5dxg\" (UID: \"9ddda0bf-9853-4015-a006-c9e44d47acd5\") " pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:09.189220 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187432    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-run-ovn\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.189220 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187464    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-host-run-ovn-kubernetes\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.189869 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187491    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-sys\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.189869 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187514    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e61e15eb-3d0d-4a92-a91a-0987c9fd4eeb-serviceca\") pod \"node-ca-qrq28\" (UID: \"e61e15eb-3d0d-4a92-a91a-0987c9fd4eeb\") " pod="openshift-image-registry/node-ca-qrq28"
Sep 09 13:26:09.189869 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:09.187535    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 13:26:09.189869 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187537    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-run-ovn\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.189869 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187546    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f06384ff-80d7-4e3a-a9bd-4925d55d6824-hosts-file\") pod \"node-resolver-sw9qv\" (UID: \"f06384ff-80d7-4e3a-a9bd-4925d55d6824\") " pod="openshift-dns/node-resolver-sw9qv"
Sep 09 13:26:09.189869 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:09.187603    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ddda0bf-9853-4015-a006-c9e44d47acd5-metrics-certs podName:9ddda0bf-9853-4015-a006-c9e44d47acd5 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:09.687586836 +0000 UTC m=+3.047436655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ddda0bf-9853-4015-a006-c9e44d47acd5-metrics-certs") pod "network-metrics-daemon-q5dxg" (UID: "9ddda0bf-9853-4015-a006-c9e44d47acd5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 13:26:09.189869 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187657    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-host-var-lib-cni-bin\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.189869 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187662    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-host-run-ovn-kubernetes\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.189869 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187660    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f06384ff-80d7-4e3a-a9bd-4925d55d6824-hosts-file\") pod \"node-resolver-sw9qv\" (UID: \"f06384ff-80d7-4e3a-a9bd-4925d55d6824\") " pod="openshift-dns/node-resolver-sw9qv"
Sep 09 13:26:09.189869 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187769    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-multus-daemon-config\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.189869 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187795    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-host-var-lib-cni-bin\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.189869 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187801    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6dde766b-3e64-4762-ab24-25b30b1b4465-cni-binary-copy\") pod \"multus-additional-cni-plugins-q6cft\" (UID: \"6dde766b-3e64-4762-ab24-25b30b1b4465\") " pod="openshift-multus/multus-additional-cni-plugins-q6cft"
Sep 09 13:26:09.189869 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.187850    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-etc-sysctl-conf\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.189869 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.188068    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e61e15eb-3d0d-4a92-a91a-0987c9fd4eeb-serviceca\") pod \"node-ca-qrq28\" (UID: \"e61e15eb-3d0d-4a92-a91a-0987c9fd4eeb\") " pod="openshift-image-registry/node-ca-qrq28"
Sep 09 13:26:09.189869 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.188292    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-multus-daemon-config\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.189869 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.188356    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-env-overrides\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.189869 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.189199    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-ovn-node-metrics-cert\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.190458 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.189304    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ab687c9c-b837-4387-a696-7f24afd7711f-agent-certs\") pod \"konnectivity-agent-q5njn\" (UID: \"ab687c9c-b837-4387-a696-7f24afd7711f\") " pod="kube-system/konnectivity-agent-q5njn"
Sep 09 13:26:09.211240 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.211205    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtgxj\" (UniqueName: \"kubernetes.io/projected/2caadc2a-aabb-440f-8f3b-98ceb69052cd-kube-api-access-dtgxj\") pod \"iptables-alerter-bt7v9\" (UID: \"2caadc2a-aabb-440f-8f3b-98ceb69052cd\") " pod="openshift-network-operator/iptables-alerter-bt7v9"
Sep 09 13:26:09.228629 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.228592    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxbt2\" (UniqueName: \"kubernetes.io/projected/625682e0-c96b-4b33-b0b8-5b9fb93d4804-kube-api-access-pxbt2\") pod \"aws-ebs-csi-driver-node-flwf7\" (UID: \"625682e0-c96b-4b33-b0b8-5b9fb93d4804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7"
Sep 09 13:26:09.228924 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.228899    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9t9q\" (UniqueName: \"kubernetes.io/projected/f06384ff-80d7-4e3a-a9bd-4925d55d6824-kube-api-access-z9t9q\") pod \"node-resolver-sw9qv\" (UID: \"f06384ff-80d7-4e3a-a9bd-4925d55d6824\") " pod="openshift-dns/node-resolver-sw9qv"
Sep 09 13:26:09.229026 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.228900    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7f5s\" (UniqueName: \"kubernetes.io/projected/3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f-kube-api-access-v7f5s\") pod \"ovnkube-node-m4s7g\" (UID: \"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.229026 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.229001    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng9pg\" (UniqueName: \"kubernetes.io/projected/a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d-kube-api-access-ng9pg\") pod \"multus-gwqft\" (UID: \"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d\") " pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.231892 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.231870    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gvp5\" (UniqueName: \"kubernetes.io/projected/9ddda0bf-9853-4015-a006-c9e44d47acd5-kube-api-access-6gvp5\") pod \"network-metrics-daemon-q5dxg\" (UID: \"9ddda0bf-9853-4015-a006-c9e44d47acd5\") " pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:09.242829 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.242794    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jmtl\" (UniqueName: \"kubernetes.io/projected/e61e15eb-3d0d-4a92-a91a-0987c9fd4eeb-kube-api-access-7jmtl\") pod \"node-ca-qrq28\" (UID: \"e61e15eb-3d0d-4a92-a91a-0987c9fd4eeb\") " pod="openshift-image-registry/node-ca-qrq28"
Sep 09 13:26:09.288736 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.288696    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-sys\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.288736 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.288747    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6dde766b-3e64-4762-ab24-25b30b1b4465-cni-binary-copy\") pod \"multus-additional-cni-plugins-q6cft\" (UID: \"6dde766b-3e64-4762-ab24-25b30b1b4465\") " pod="openshift-multus/multus-additional-cni-plugins-q6cft"
Sep 09 13:26:09.288974 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.288854    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-sys\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.288974 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.288871    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-etc-sysctl-conf\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.288974 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.288920    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6dde766b-3e64-4762-ab24-25b30b1b4465-system-cni-dir\") pod \"multus-additional-cni-plugins-q6cft\" (UID: \"6dde766b-3e64-4762-ab24-25b30b1b4465\") " pod="openshift-multus/multus-additional-cni-plugins-q6cft"
Sep 09 13:26:09.288974 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.288952    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6dde766b-3e64-4762-ab24-25b30b1b4465-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q6cft\" (UID: \"6dde766b-3e64-4762-ab24-25b30b1b4465\") " pod="openshift-multus/multus-additional-cni-plugins-q6cft"
Sep 09 13:26:09.289171 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.288986    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6dde766b-3e64-4762-ab24-25b30b1b4465-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q6cft\" (UID: \"6dde766b-3e64-4762-ab24-25b30b1b4465\") " pod="openshift-multus/multus-additional-cni-plugins-q6cft"
Sep 09 13:26:09.289171 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289004    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6dde766b-3e64-4762-ab24-25b30b1b4465-system-cni-dir\") pod \"multus-additional-cni-plugins-q6cft\" (UID: \"6dde766b-3e64-4762-ab24-25b30b1b4465\") " pod="openshift-multus/multus-additional-cni-plugins-q6cft"
Sep 09 13:26:09.289171 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289020    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-etc-sysctl-conf\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.289171 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289011    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-lib-modules\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.289171 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289070    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d65cbad0-4588-49d4-bac1-5303af169027-tmp\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.289171 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289104    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-etc-kubernetes\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.289171 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289148    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-lib-modules\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.289171 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289155    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-552v9\" (UniqueName: \"kubernetes.io/projected/49845a79-9f07-4b8c-9f9d-ad2a1035d184-kube-api-access-552v9\") pod \"network-check-target-zbqpk\" (UID: \"49845a79-9f07-4b8c-9f9d-ad2a1035d184\") " pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:26:09.289560 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289184    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6dde766b-3e64-4762-ab24-25b30b1b4465-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q6cft\" (UID: \"6dde766b-3e64-4762-ab24-25b30b1b4465\") " pod="openshift-multus/multus-additional-cni-plugins-q6cft"
Sep 09 13:26:09.289560 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289193    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6dde766b-3e64-4762-ab24-25b30b1b4465-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q6cft\" (UID: \"6dde766b-3e64-4762-ab24-25b30b1b4465\") " pod="openshift-multus/multus-additional-cni-plugins-q6cft"
Sep 09 13:26:09.289560 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289228    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6dde766b-3e64-4762-ab24-25b30b1b4465-cnibin\") pod \"multus-additional-cni-plugins-q6cft\" (UID: \"6dde766b-3e64-4762-ab24-25b30b1b4465\") " pod="openshift-multus/multus-additional-cni-plugins-q6cft"
Sep 09 13:26:09.289560 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289260    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-etc-modprobe-d\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.289560 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289288    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d65cbad0-4588-49d4-bac1-5303af169027-etc-tuned\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.289560 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289315    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76w5k\" (UniqueName: \"kubernetes.io/projected/6dde766b-3e64-4762-ab24-25b30b1b4465-kube-api-access-76w5k\") pod \"multus-additional-cni-plugins-q6cft\" (UID: \"6dde766b-3e64-4762-ab24-25b30b1b4465\") " pod="openshift-multus/multus-additional-cni-plugins-q6cft"
Sep 09 13:26:09.289560 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289351    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6dde766b-3e64-4762-ab24-25b30b1b4465-cnibin\") pod \"multus-additional-cni-plugins-q6cft\" (UID: \"6dde766b-3e64-4762-ab24-25b30b1b4465\") " pod="openshift-multus/multus-additional-cni-plugins-q6cft"
Sep 09 13:26:09.289560 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289198    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-etc-kubernetes\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.289560 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289379    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-etc-systemd\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.289560 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289408    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-etc-sysconfig\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.289560 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289431    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-var-lib-kubelet\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.289560 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289457    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-etc-sysctl-d\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.289560 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289483    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-host\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.289560 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289497    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-etc-systemd\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.289560 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289510    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8mbp\" (UniqueName: \"kubernetes.io/projected/d65cbad0-4588-49d4-bac1-5303af169027-kube-api-access-k8mbp\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.289560 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289533    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6dde766b-3e64-4762-ab24-25b30b1b4465-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q6cft\" (UID: \"6dde766b-3e64-4762-ab24-25b30b1b4465\") " pod="openshift-multus/multus-additional-cni-plugins-q6cft"
Sep 09 13:26:09.289560 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289549    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-etc-sysconfig\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.290255 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289538    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-run\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.290255 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289323    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6dde766b-3e64-4762-ab24-25b30b1b4465-cni-binary-copy\") pod \"multus-additional-cni-plugins-q6cft\" (UID: \"6dde766b-3e64-4762-ab24-25b30b1b4465\") " pod="openshift-multus/multus-additional-cni-plugins-q6cft"
Sep 09 13:26:09.290255 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289552    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-var-lib-kubelet\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.290255 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289580    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-run\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.290255 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289501    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-etc-modprobe-d\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.290255 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289636    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-host\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.290255 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289664    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6dde766b-3e64-4762-ab24-25b30b1b4465-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q6cft\" (UID: \"6dde766b-3e64-4762-ab24-25b30b1b4465\") " pod="openshift-multus/multus-additional-cni-plugins-q6cft"
Sep 09 13:26:09.290255 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.289674    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d65cbad0-4588-49d4-bac1-5303af169027-etc-sysctl-d\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.292457 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.292400    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d65cbad0-4588-49d4-bac1-5303af169027-etc-tuned\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.292457 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.292426    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d65cbad0-4588-49d4-bac1-5303af169027-tmp\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.306381 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:09.306357    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 13:26:09.306381 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:09.306384    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 13:26:09.306574 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:09.306398    2566 projected.go:196] Error preparing data for projected volume kube-api-access-552v9 for pod openshift-network-diagnostics/network-check-target-zbqpk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 13:26:09.306574 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:09.306476    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49845a79-9f07-4b8c-9f9d-ad2a1035d184-kube-api-access-552v9 podName:49845a79-9f07-4b8c-9f9d-ad2a1035d184 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:09.806455638 +0000 UTC m=+3.166305446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-552v9" (UniqueName: "kubernetes.io/projected/49845a79-9f07-4b8c-9f9d-ad2a1035d184-kube-api-access-552v9") pod "network-check-target-zbqpk" (UID: "49845a79-9f07-4b8c-9f9d-ad2a1035d184") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 13:26:09.312417 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.312393    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76w5k\" (UniqueName: \"kubernetes.io/projected/6dde766b-3e64-4762-ab24-25b30b1b4465-kube-api-access-76w5k\") pod \"multus-additional-cni-plugins-q6cft\" (UID: \"6dde766b-3e64-4762-ab24-25b30b1b4465\") " pod="openshift-multus/multus-additional-cni-plugins-q6cft"
Sep 09 13:26:09.313220 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.313198    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8mbp\" (UniqueName: \"kubernetes.io/projected/d65cbad0-4588-49d4-bac1-5303af169027-kube-api-access-k8mbp\") pod \"tuned-nqs45\" (UID: \"d65cbad0-4588-49d4-bac1-5303af169027\") " pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.364178 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.364137    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q5njn"
Sep 09 13:26:09.374920 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.374893    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:09.384408 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.384382    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sw9qv"
Sep 09 13:26:09.389298 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.389263    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gwqft"
Sep 09 13:26:09.395063 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.395042    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bt7v9"
Sep 09 13:26:09.400284 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.400262    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qrq28"
Sep 09 13:26:09.406776 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.406743    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7"
Sep 09 13:26:09.412576 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.412553    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nqs45"
Sep 09 13:26:09.416351 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.416327    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q6cft"
Sep 09 13:26:09.692332 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.692234    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ecc5e12-af64-4305-83ed-37552f51b76c-original-pull-secret\") pod \"global-pull-secret-syncer-plsz4\" (UID: \"0ecc5e12-af64-4305-83ed-37552f51b76c\") " pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:09.692332 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.692293    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ddda0bf-9853-4015-a006-c9e44d47acd5-metrics-certs\") pod \"network-metrics-daemon-q5dxg\" (UID: \"9ddda0bf-9853-4015-a006-c9e44d47acd5\") " pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:09.692552 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:09.692429    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 13:26:09.692552 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:09.692493    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ddda0bf-9853-4015-a006-c9e44d47acd5-metrics-certs podName:9ddda0bf-9853-4015-a006-c9e44d47acd5 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:10.692474165 +0000 UTC m=+4.052323985 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ddda0bf-9853-4015-a006-c9e44d47acd5-metrics-certs") pod "network-metrics-daemon-q5dxg" (UID: "9ddda0bf-9853-4015-a006-c9e44d47acd5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 13:26:09.692791 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:09.692751    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 13:26:09.692957 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:09.692835    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ecc5e12-af64-4305-83ed-37552f51b76c-original-pull-secret podName:0ecc5e12-af64-4305-83ed-37552f51b76c nodeName:}" failed. No retries permitted until 2026-09-09 13:26:10.692802266 +0000 UTC m=+4.052652074 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ecc5e12-af64-4305-83ed-37552f51b76c-original-pull-secret") pod "global-pull-secret-syncer-plsz4" (UID: "0ecc5e12-af64-4305-83ed-37552f51b76c") : object "kube-system"/"original-pull-secret" not registered
Sep 09 13:26:09.774827 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:09.774768    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06384ff_80d7_4e3a_a9bd_4925d55d6824.slice/crio-a2ebc57eedd778e09b6740597ebc61203d1c504d63bdbca32dece4fb412cba94 WatchSource:0}: Error finding container a2ebc57eedd778e09b6740597ebc61203d1c504d63bdbca32dece4fb412cba94: Status 404 returned error can't find the container with id a2ebc57eedd778e09b6740597ebc61203d1c504d63bdbca32dece4fb412cba94
Sep 09 13:26:09.775885 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:09.775850    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2caadc2a_aabb_440f_8f3b_98ceb69052cd.slice/crio-7ce6a6c128d7fed8deed21ef3554f6536a085341b6ce800dd9904769f80b90d8 WatchSource:0}: Error finding container 7ce6a6c128d7fed8deed21ef3554f6536a085341b6ce800dd9904769f80b90d8: Status 404 returned error can't find the container with id 7ce6a6c128d7fed8deed21ef3554f6536a085341b6ce800dd9904769f80b90d8
Sep 09 13:26:09.779404 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:09.779380    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode61e15eb_3d0d_4a92_a91a_0987c9fd4eeb.slice/crio-69ae1efc4028746f1b23fda7efc7c3f96a378b0bbd595274459f0be0227bf283 WatchSource:0}: Error finding container 69ae1efc4028746f1b23fda7efc7c3f96a378b0bbd595274459f0be0227bf283: Status 404 returned error can't find the container with id 69ae1efc4028746f1b23fda7efc7c3f96a378b0bbd595274459f0be0227bf283
Sep 09 13:26:09.780971 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:09.780918    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab687c9c_b837_4387_a696_7f24afd7711f.slice/crio-8551b7b9045ea31bc28ec7d775888d4e631dccb0b43e67a6b484f29118f35d50 WatchSource:0}: Error finding container 8551b7b9045ea31bc28ec7d775888d4e631dccb0b43e67a6b484f29118f35d50: Status 404 returned error can't find the container with id 8551b7b9045ea31bc28ec7d775888d4e631dccb0b43e67a6b484f29118f35d50
Sep 09 13:26:09.781616 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:09.781527    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dde766b_3e64_4762_ab24_25b30b1b4465.slice/crio-c53c2d1783c57f6d51a6b1eaa7c8ece11766e59b78deac55379315c60d9d537f WatchSource:0}: Error finding container c53c2d1783c57f6d51a6b1eaa7c8ece11766e59b78deac55379315c60d9d537f: Status 404 returned error can't find the container with id c53c2d1783c57f6d51a6b1eaa7c8ece11766e59b78deac55379315c60d9d537f
Sep 09 13:26:09.783692 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:09.783662    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda455ac3b_27e3_42ce_b4c3_c0fe4b5f460d.slice/crio-3b3457413823a246d381d63d107294d44e55e652fde1da1c7a986b6fa3136f8e WatchSource:0}: Error finding container 3b3457413823a246d381d63d107294d44e55e652fde1da1c7a986b6fa3136f8e: Status 404 returned error can't find the container with id 3b3457413823a246d381d63d107294d44e55e652fde1da1c7a986b6fa3136f8e
Sep 09 13:26:09.784080 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:09.784062    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd65cbad0_4588_49d4_bac1_5303af169027.slice/crio-702233427c7904b94e07fb34fb6bfa66d607d51a7f2e0c3b4cfff33d5ed8835c WatchSource:0}: Error finding container 702233427c7904b94e07fb34fb6bfa66d607d51a7f2e0c3b4cfff33d5ed8835c: Status 404 returned error can't find the container with id 702233427c7904b94e07fb34fb6bfa66d607d51a7f2e0c3b4cfff33d5ed8835c
Sep 09 13:26:09.805541 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:09.805501    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625682e0_c96b_4b33_b0b8_5b9fb93d4804.slice/crio-b08d38231fd39c7fe819b244c68b82eaf6dd4ef0180f1198de83d863f7dbd9a9 WatchSource:0}: Error finding container b08d38231fd39c7fe819b244c68b82eaf6dd4ef0180f1198de83d863f7dbd9a9: Status 404 returned error can't find the container with id b08d38231fd39c7fe819b244c68b82eaf6dd4ef0180f1198de83d863f7dbd9a9
Sep 09 13:26:09.806218 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:09.806109    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3db565cf_bcf9_4a4d_b2f8_305fac6c4f3f.slice/crio-7f32fbf4d642f5aab3aceb5c2ac527d167e746cbadf965c785eccbc16e96fa07 WatchSource:0}: Error finding container 7f32fbf4d642f5aab3aceb5c2ac527d167e746cbadf965c785eccbc16e96fa07: Status 404 returned error can't find the container with id 7f32fbf4d642f5aab3aceb5c2ac527d167e746cbadf965c785eccbc16e96fa07
Sep 09 13:26:09.893626 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:09.893595    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-552v9\" (UniqueName: \"kubernetes.io/projected/49845a79-9f07-4b8c-9f9d-ad2a1035d184-kube-api-access-552v9\") pod \"network-check-target-zbqpk\" (UID: \"49845a79-9f07-4b8c-9f9d-ad2a1035d184\") " pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:26:09.893727 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:09.893708    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 13:26:09.893727 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:09.893723    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 13:26:09.893794 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:09.893732    2566 projected.go:196] Error preparing data for projected volume kube-api-access-552v9 for pod openshift-network-diagnostics/network-check-target-zbqpk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 13:26:09.893794 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:09.893779    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49845a79-9f07-4b8c-9f9d-ad2a1035d184-kube-api-access-552v9 podName:49845a79-9f07-4b8c-9f9d-ad2a1035d184 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:10.89376526 +0000 UTC m=+4.253615064 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-552v9" (UniqueName: "kubernetes.io/projected/49845a79-9f07-4b8c-9f9d-ad2a1035d184-kube-api-access-552v9") pod "network-check-target-zbqpk" (UID: "49845a79-9f07-4b8c-9f9d-ad2a1035d184") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 13:26:10.154114 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:10.154020    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 13:21:09 +0000 UTC" deadline="2028-05-27 08:17:03.801195416 +0000 UTC"
Sep 09 13:26:10.154114 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:10.154054    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15018h50m53.647145479s"
Sep 09 13:26:10.154905 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:10.154633    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q5njn" event={"ID":"ab687c9c-b837-4387-a696-7f24afd7711f","Type":"ContainerStarted","Data":"8551b7b9045ea31bc28ec7d775888d4e631dccb0b43e67a6b484f29118f35d50"}
Sep 09 13:26:10.155844 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:10.155808    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g" event={"ID":"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f","Type":"ContainerStarted","Data":"7f32fbf4d642f5aab3aceb5c2ac527d167e746cbadf965c785eccbc16e96fa07"}
Sep 09 13:26:10.156919 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:10.156898    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nqs45" event={"ID":"d65cbad0-4588-49d4-bac1-5303af169027","Type":"ContainerStarted","Data":"702233427c7904b94e07fb34fb6bfa66d607d51a7f2e0c3b4cfff33d5ed8835c"}
Sep 09 13:26:10.157960 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:10.157942    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gwqft" event={"ID":"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d","Type":"ContainerStarted","Data":"3b3457413823a246d381d63d107294d44e55e652fde1da1c7a986b6fa3136f8e"}
Sep 09 13:26:10.159075 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:10.159056    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q6cft" event={"ID":"6dde766b-3e64-4762-ab24-25b30b1b4465","Type":"ContainerStarted","Data":"c53c2d1783c57f6d51a6b1eaa7c8ece11766e59b78deac55379315c60d9d537f"}
Sep 09 13:26:10.160049 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:10.160033    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qrq28" event={"ID":"e61e15eb-3d0d-4a92-a91a-0987c9fd4eeb","Type":"ContainerStarted","Data":"69ae1efc4028746f1b23fda7efc7c3f96a378b0bbd595274459f0be0227bf283"}
Sep 09 13:26:10.161023 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:10.161004    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bt7v9" event={"ID":"2caadc2a-aabb-440f-8f3b-98ceb69052cd","Type":"ContainerStarted","Data":"7ce6a6c128d7fed8deed21ef3554f6536a085341b6ce800dd9904769f80b90d8"}
Sep 09 13:26:10.162017 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:10.162001    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sw9qv" event={"ID":"f06384ff-80d7-4e3a-a9bd-4925d55d6824","Type":"ContainerStarted","Data":"a2ebc57eedd778e09b6740597ebc61203d1c504d63bdbca32dece4fb412cba94"}
Sep 09 13:26:10.163628 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:10.163609    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal" event={"ID":"bad615a00a972079824050bf66b78c89","Type":"ContainerStarted","Data":"842e57b26c55bd91831028f65590bba2863b8be1157d5d1cd3a6be2c3ed1ed47"}
Sep 09 13:26:10.164754 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:10.164737    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7" event={"ID":"625682e0-c96b-4b33-b0b8-5b9fb93d4804","Type":"ContainerStarted","Data":"b08d38231fd39c7fe819b244c68b82eaf6dd4ef0180f1198de83d863f7dbd9a9"}
Sep 09 13:26:10.179443 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:10.179376    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal" podStartSLOduration=2.179357037 podStartE2EDuration="2.179357037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 13:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:26:10.179120357 +0000 UTC m=+3.538970182" watchObservedRunningTime="2026-09-09 13:26:10.179357037 +0000 UTC m=+3.539206864"
Sep 09 13:26:10.488284 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:10.487989    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 13:26:10.700518 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:10.699705    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ecc5e12-af64-4305-83ed-37552f51b76c-original-pull-secret\") pod \"global-pull-secret-syncer-plsz4\" (UID: \"0ecc5e12-af64-4305-83ed-37552f51b76c\") " pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:10.700518 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:10.699773    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ddda0bf-9853-4015-a006-c9e44d47acd5-metrics-certs\") pod \"network-metrics-daemon-q5dxg\" (UID: \"9ddda0bf-9853-4015-a006-c9e44d47acd5\") " pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:10.700518 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:10.699978    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 13:26:10.700518 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:10.700044    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ddda0bf-9853-4015-a006-c9e44d47acd5-metrics-certs podName:9ddda0bf-9853-4015-a006-c9e44d47acd5 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:12.700025681 +0000 UTC m=+6.059875509 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ddda0bf-9853-4015-a006-c9e44d47acd5-metrics-certs") pod "network-metrics-daemon-q5dxg" (UID: "9ddda0bf-9853-4015-a006-c9e44d47acd5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 13:26:10.700518 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:10.700428    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 13:26:10.700518 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:10.700480    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ecc5e12-af64-4305-83ed-37552f51b76c-original-pull-secret podName:0ecc5e12-af64-4305-83ed-37552f51b76c nodeName:}" failed. No retries permitted until 2026-09-09 13:26:12.7004648 +0000 UTC m=+6.060314620 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ecc5e12-af64-4305-83ed-37552f51b76c-original-pull-secret") pod "global-pull-secret-syncer-plsz4" (UID: "0ecc5e12-af64-4305-83ed-37552f51b76c") : object "kube-system"/"original-pull-secret" not registered
Sep 09 13:26:10.901671 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:10.901579    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-552v9\" (UniqueName: \"kubernetes.io/projected/49845a79-9f07-4b8c-9f9d-ad2a1035d184-kube-api-access-552v9\") pod \"network-check-target-zbqpk\" (UID: \"49845a79-9f07-4b8c-9f9d-ad2a1035d184\") " pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:26:10.901867 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:10.901774    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 13:26:10.901867 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:10.901798    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 13:26:10.901867 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:10.901823    2566 projected.go:196] Error preparing data for projected volume kube-api-access-552v9 for pod openshift-network-diagnostics/network-check-target-zbqpk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 13:26:10.902041 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:10.901883    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49845a79-9f07-4b8c-9f9d-ad2a1035d184-kube-api-access-552v9 podName:49845a79-9f07-4b8c-9f9d-ad2a1035d184 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:12.90186435 +0000 UTC m=+6.261714157 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-552v9" (UniqueName: "kubernetes.io/projected/49845a79-9f07-4b8c-9f9d-ad2a1035d184-kube-api-access-552v9") pod "network-check-target-zbqpk" (UID: "49845a79-9f07-4b8c-9f9d-ad2a1035d184") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 13:26:11.145743 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:11.144913    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:11.145743 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:11.145051    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plsz4" podUID="0ecc5e12-af64-4305-83ed-37552f51b76c"
Sep 09 13:26:11.145743 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:11.145425    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:26:11.145743 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:11.145512    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbqpk" podUID="49845a79-9f07-4b8c-9f9d-ad2a1035d184"
Sep 09 13:26:11.145743 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:11.145585    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:11.145743 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:11.145694    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5dxg" podUID="9ddda0bf-9853-4015-a006-c9e44d47acd5"
Sep 09 13:26:11.154796 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:11.154695    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 13:21:09 +0000 UTC" deadline="2028-03-21 12:14:13.060159188 +0000 UTC"
Sep 09 13:26:11.154796 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:11.154731    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13414h48m1.905432391s"
Sep 09 13:26:12.719728 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:12.718919    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ecc5e12-af64-4305-83ed-37552f51b76c-original-pull-secret\") pod \"global-pull-secret-syncer-plsz4\" (UID: \"0ecc5e12-af64-4305-83ed-37552f51b76c\") " pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:12.719728 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:12.718987    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ddda0bf-9853-4015-a006-c9e44d47acd5-metrics-certs\") pod \"network-metrics-daemon-q5dxg\" (UID: \"9ddda0bf-9853-4015-a006-c9e44d47acd5\") " pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:12.719728 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:12.719132    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 13:26:12.719728 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:12.719199    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ddda0bf-9853-4015-a006-c9e44d47acd5-metrics-certs podName:9ddda0bf-9853-4015-a006-c9e44d47acd5 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:16.719179411 +0000 UTC m=+10.079029217 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ddda0bf-9853-4015-a006-c9e44d47acd5-metrics-certs") pod "network-metrics-daemon-q5dxg" (UID: "9ddda0bf-9853-4015-a006-c9e44d47acd5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 13:26:12.719728 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:12.719625    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 13:26:12.719728 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:12.719683    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ecc5e12-af64-4305-83ed-37552f51b76c-original-pull-secret podName:0ecc5e12-af64-4305-83ed-37552f51b76c nodeName:}" failed. No retries permitted until 2026-09-09 13:26:16.719666527 +0000 UTC m=+10.079516332 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ecc5e12-af64-4305-83ed-37552f51b76c-original-pull-secret") pod "global-pull-secret-syncer-plsz4" (UID: "0ecc5e12-af64-4305-83ed-37552f51b76c") : object "kube-system"/"original-pull-secret" not registered
Sep 09 13:26:12.920163 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:12.920120    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-552v9\" (UniqueName: \"kubernetes.io/projected/49845a79-9f07-4b8c-9f9d-ad2a1035d184-kube-api-access-552v9\") pod \"network-check-target-zbqpk\" (UID: \"49845a79-9f07-4b8c-9f9d-ad2a1035d184\") " pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:26:12.920369 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:12.920322    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 13:26:12.920369 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:12.920344    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 13:26:12.920369 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:12.920357    2566 projected.go:196] Error preparing data for projected volume kube-api-access-552v9 for pod openshift-network-diagnostics/network-check-target-zbqpk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 13:26:12.920537 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:12.920420    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49845a79-9f07-4b8c-9f9d-ad2a1035d184-kube-api-access-552v9 podName:49845a79-9f07-4b8c-9f9d-ad2a1035d184 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:16.920400297 +0000 UTC m=+10.280250124 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-552v9" (UniqueName: "kubernetes.io/projected/49845a79-9f07-4b8c-9f9d-ad2a1035d184-kube-api-access-552v9") pod "network-check-target-zbqpk" (UID: "49845a79-9f07-4b8c-9f9d-ad2a1035d184") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 13:26:13.143956 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:13.143874    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:13.143956 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:13.143911    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:26:13.144175 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:13.143874    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:13.144175 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:13.144011    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plsz4" podUID="0ecc5e12-af64-4305-83ed-37552f51b76c"
Sep 09 13:26:13.144175 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:13.144107    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5dxg" podUID="9ddda0bf-9853-4015-a006-c9e44d47acd5"
Sep 09 13:26:13.144310 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:13.144187    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbqpk" podUID="49845a79-9f07-4b8c-9f9d-ad2a1035d184"
Sep 09 13:26:15.143442 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:15.143373    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:15.143873 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:15.143519    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plsz4" podUID="0ecc5e12-af64-4305-83ed-37552f51b76c"
Sep 09 13:26:15.143873 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:15.143604    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:26:15.143873 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:15.143725    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbqpk" podUID="49845a79-9f07-4b8c-9f9d-ad2a1035d184"
Sep 09 13:26:15.143873 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:15.143761    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:15.144033 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:15.143872    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5dxg" podUID="9ddda0bf-9853-4015-a006-c9e44d47acd5"
Sep 09 13:26:16.751352 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:16.751110    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ecc5e12-af64-4305-83ed-37552f51b76c-original-pull-secret\") pod \"global-pull-secret-syncer-plsz4\" (UID: \"0ecc5e12-af64-4305-83ed-37552f51b76c\") " pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:16.751794 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:16.751389    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 13:26:16.751794 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:16.751407    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ddda0bf-9853-4015-a006-c9e44d47acd5-metrics-certs\") pod \"network-metrics-daemon-q5dxg\" (UID: \"9ddda0bf-9853-4015-a006-c9e44d47acd5\") " pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:16.751794 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:16.751461    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ecc5e12-af64-4305-83ed-37552f51b76c-original-pull-secret podName:0ecc5e12-af64-4305-83ed-37552f51b76c nodeName:}" failed. No retries permitted until 2026-09-09 13:26:24.751440724 +0000 UTC m=+18.111290545 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ecc5e12-af64-4305-83ed-37552f51b76c-original-pull-secret") pod "global-pull-secret-syncer-plsz4" (UID: "0ecc5e12-af64-4305-83ed-37552f51b76c") : object "kube-system"/"original-pull-secret" not registered
Sep 09 13:26:16.751794 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:16.751511    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 13:26:16.751794 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:16.751556    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ddda0bf-9853-4015-a006-c9e44d47acd5-metrics-certs podName:9ddda0bf-9853-4015-a006-c9e44d47acd5 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:24.751543452 +0000 UTC m=+18.111393260 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ddda0bf-9853-4015-a006-c9e44d47acd5-metrics-certs") pod "network-metrics-daemon-q5dxg" (UID: "9ddda0bf-9853-4015-a006-c9e44d47acd5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 13:26:16.952775 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:16.952499    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-552v9\" (UniqueName: \"kubernetes.io/projected/49845a79-9f07-4b8c-9f9d-ad2a1035d184-kube-api-access-552v9\") pod \"network-check-target-zbqpk\" (UID: \"49845a79-9f07-4b8c-9f9d-ad2a1035d184\") " pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:26:16.952775 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:16.952672    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 13:26:16.952775 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:16.952690    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 13:26:16.952775 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:16.952702    2566 projected.go:196] Error preparing data for projected volume kube-api-access-552v9 for pod openshift-network-diagnostics/network-check-target-zbqpk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 13:26:16.952775 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:16.952747    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49845a79-9f07-4b8c-9f9d-ad2a1035d184-kube-api-access-552v9 podName:49845a79-9f07-4b8c-9f9d-ad2a1035d184 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:24.952735043 +0000 UTC m=+18.312584860 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-552v9" (UniqueName: "kubernetes.io/projected/49845a79-9f07-4b8c-9f9d-ad2a1035d184-kube-api-access-552v9") pod "network-check-target-zbqpk" (UID: "49845a79-9f07-4b8c-9f9d-ad2a1035d184") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 13:26:17.144948 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:17.144832    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:17.145106 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:17.144973    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plsz4" podUID="0ecc5e12-af64-4305-83ed-37552f51b76c"
Sep 09 13:26:17.145439 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:17.145412    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:17.145599 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:17.145576    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5dxg" podUID="9ddda0bf-9853-4015-a006-c9e44d47acd5"
Sep 09 13:26:17.146376 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:17.146354    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:26:17.146489 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:17.146468    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbqpk" podUID="49845a79-9f07-4b8c-9f9d-ad2a1035d184"
Sep 09 13:26:19.143934 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:19.143890    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:19.144393 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:19.144035    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plsz4" podUID="0ecc5e12-af64-4305-83ed-37552f51b76c"
Sep 09 13:26:19.144393 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:19.144063    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:26:19.144393 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:19.143904    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:19.144393 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:19.144143    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbqpk" podUID="49845a79-9f07-4b8c-9f9d-ad2a1035d184"
Sep 09 13:26:19.144393 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:19.144283    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5dxg" podUID="9ddda0bf-9853-4015-a006-c9e44d47acd5"
Sep 09 13:26:21.144351 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:21.144314    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:21.144351 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:21.144356    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:26:21.144869 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:21.144321    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:21.144869 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:21.144449    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plsz4" podUID="0ecc5e12-af64-4305-83ed-37552f51b76c"
Sep 09 13:26:21.144869 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:21.144542    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5dxg" podUID="9ddda0bf-9853-4015-a006-c9e44d47acd5"
Sep 09 13:26:21.144869 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:21.144625    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbqpk" podUID="49845a79-9f07-4b8c-9f9d-ad2a1035d184"
Sep 09 13:26:23.143674 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:23.143629    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:23.143674 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:23.143655    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:23.144201 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:23.143779    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plsz4" podUID="0ecc5e12-af64-4305-83ed-37552f51b76c"
Sep 09 13:26:23.144201 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:23.143839    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:26:23.144201 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:23.143941    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5dxg" podUID="9ddda0bf-9853-4015-a006-c9e44d47acd5"
Sep 09 13:26:23.144201 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:23.144031    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbqpk" podUID="49845a79-9f07-4b8c-9f9d-ad2a1035d184"
Sep 09 13:26:24.814628 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:24.814582    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ecc5e12-af64-4305-83ed-37552f51b76c-original-pull-secret\") pod \"global-pull-secret-syncer-plsz4\" (UID: \"0ecc5e12-af64-4305-83ed-37552f51b76c\") " pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:24.815238 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:24.814652    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ddda0bf-9853-4015-a006-c9e44d47acd5-metrics-certs\") pod \"network-metrics-daemon-q5dxg\" (UID: \"9ddda0bf-9853-4015-a006-c9e44d47acd5\") " pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:24.815238 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:24.814758    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 13:26:24.815238 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:24.814766    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 13:26:24.815238 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:24.814843    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ecc5e12-af64-4305-83ed-37552f51b76c-original-pull-secret podName:0ecc5e12-af64-4305-83ed-37552f51b76c nodeName:}" failed. No retries permitted until 2026-09-09 13:26:40.81480874 +0000 UTC m=+34.174658559 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ecc5e12-af64-4305-83ed-37552f51b76c-original-pull-secret") pod "global-pull-secret-syncer-plsz4" (UID: "0ecc5e12-af64-4305-83ed-37552f51b76c") : object "kube-system"/"original-pull-secret" not registered
Sep 09 13:26:24.815238 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:24.814899    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ddda0bf-9853-4015-a006-c9e44d47acd5-metrics-certs podName:9ddda0bf-9853-4015-a006-c9e44d47acd5 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:40.814870017 +0000 UTC m=+34.174719835 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ddda0bf-9853-4015-a006-c9e44d47acd5-metrics-certs") pod "network-metrics-daemon-q5dxg" (UID: "9ddda0bf-9853-4015-a006-c9e44d47acd5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 13:26:25.016618 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:25.016582    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-552v9\" (UniqueName: \"kubernetes.io/projected/49845a79-9f07-4b8c-9f9d-ad2a1035d184-kube-api-access-552v9\") pod \"network-check-target-zbqpk\" (UID: \"49845a79-9f07-4b8c-9f9d-ad2a1035d184\") " pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:26:25.016840 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:25.016758    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 13:26:25.016840 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:25.016786    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 13:26:25.016840 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:25.016801    2566 projected.go:196] Error preparing data for projected volume kube-api-access-552v9 for pod openshift-network-diagnostics/network-check-target-zbqpk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 13:26:25.017005 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:25.016887    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49845a79-9f07-4b8c-9f9d-ad2a1035d184-kube-api-access-552v9 podName:49845a79-9f07-4b8c-9f9d-ad2a1035d184 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:41.016866232 +0000 UTC m=+34.376716056 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-552v9" (UniqueName: "kubernetes.io/projected/49845a79-9f07-4b8c-9f9d-ad2a1035d184-kube-api-access-552v9") pod "network-check-target-zbqpk" (UID: "49845a79-9f07-4b8c-9f9d-ad2a1035d184") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 13:26:25.143447 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:25.143361    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:25.143610 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:25.143506    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5dxg" podUID="9ddda0bf-9853-4015-a006-c9e44d47acd5"
Sep 09 13:26:25.143610 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:25.143526    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:25.143610 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:25.143540    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:26:25.143732 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:25.143623    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plsz4" podUID="0ecc5e12-af64-4305-83ed-37552f51b76c"
Sep 09 13:26:25.143732 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:25.143692    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbqpk" podUID="49845a79-9f07-4b8c-9f9d-ad2a1035d184"
Sep 09 13:26:27.144919 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:27.144889    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:26:27.145384 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:27.145064    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:27.145384 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:27.145097    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:27.145384 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:27.145059    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbqpk" podUID="49845a79-9f07-4b8c-9f9d-ad2a1035d184"
Sep 09 13:26:27.145384 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:27.145195    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plsz4" podUID="0ecc5e12-af64-4305-83ed-37552f51b76c"
Sep 09 13:26:27.145384 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:27.145281    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5dxg" podUID="9ddda0bf-9853-4015-a006-c9e44d47acd5"
Sep 09 13:26:28.217366 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:28.217322    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g" event={"ID":"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f","Type":"ContainerStarted","Data":"64c0bcad8d74ace6eebb260a4c2baad43c90cda63225432dd7ff12978d06fc36"}
Sep 09 13:26:28.222791 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:28.222729    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nqs45" event={"ID":"d65cbad0-4588-49d4-bac1-5303af169027","Type":"ContainerStarted","Data":"b392613e8aff19c8bd8fba666928d71876cceec9fba418e6106b380f8aac89ee"}
Sep 09 13:26:28.228336 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:28.227587    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gwqft" event={"ID":"a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d","Type":"ContainerStarted","Data":"a85b89bffce287c73b1de208a1071e473964b7cfb19cb31ce262cf86f999512d"}
Sep 09 13:26:28.249887 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:28.249831    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nqs45" podStartSLOduration=3.068556982 podStartE2EDuration="21.249794921s" totalImagesPullingTime="18.181237939s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 13:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:26:28.249605832 +0000 UTC m=+21.609455671" watchObservedRunningTime="2026-09-09 13:26:28.249794921 +0000 UTC m=+21.609644737"
Sep 09 13:26:29.143832 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:29.143789    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:29.144041 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:29.143789    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:26:29.144041 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:29.143946    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5dxg" podUID="9ddda0bf-9853-4015-a006-c9e44d47acd5"
Sep 09 13:26:29.144041 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:29.143789    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:29.144041 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:29.143967    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbqpk" podUID="49845a79-9f07-4b8c-9f9d-ad2a1035d184"
Sep 09 13:26:29.144204 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:29.144064    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plsz4" podUID="0ecc5e12-af64-4305-83ed-37552f51b76c"
Sep 09 13:26:29.230741 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:29.230710    2566 generic.go:356] "Generic (PLEG): container finished" podID="641a1c3349dfc0923275c7074bcf5214" containerID="5b938bff6872846c1ca205053d7e0459fcd60c50198ff2971b08d4a717fdc483" exitCode=0
Sep 09 13:26:29.231738 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:29.230795    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal" event={"ID":"641a1c3349dfc0923275c7074bcf5214","Type":"ContainerDied","Data":"5b938bff6872846c1ca205053d7e0459fcd60c50198ff2971b08d4a717fdc483"}
Sep 09 13:26:29.234261 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:29.234223    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g" event={"ID":"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f","Type":"ContainerStarted","Data":"bbc60b64fbe9b7d5f24c0d86f17e6eef84beeaacd2eb8910ae6aca91ebcdecf2"}
Sep 09 13:26:29.234382 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:29.234303    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g" event={"ID":"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f","Type":"ContainerStarted","Data":"73186d3251957bc4b195fef512e9d8d508fd4b31fa6ce9ef3058047f4b801825"}
Sep 09 13:26:29.234382 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:29.234337    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g" event={"ID":"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f","Type":"ContainerStarted","Data":"e2a471e5c9e4639d567b37161ec4ae98dcdc44517cf2176663e5159451c6fb04"}
Sep 09 13:26:29.234382 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:29.234372    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g" event={"ID":"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f","Type":"ContainerStarted","Data":"fe4d24ae00b634fdde0488e674b7133df0c17fa72420b2c1b9d8ffd122a51d5f"}
Sep 09 13:26:29.234496 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:29.234417    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g" event={"ID":"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f","Type":"ContainerStarted","Data":"f9094ed1c0b7210997de701b4f8cfe73cdbfd112b8781d152b010a4d765262f1"}
Sep 09 13:26:29.235715 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:29.235692    2566 generic.go:356] "Generic (PLEG): container finished" podID="6dde766b-3e64-4762-ab24-25b30b1b4465" containerID="d726624094d87ad818356939e7f01361357ea1296fbc8caf6045dcb5ad995fc4" exitCode=0
Sep 09 13:26:29.235823 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:29.235772    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q6cft" event={"ID":"6dde766b-3e64-4762-ab24-25b30b1b4465","Type":"ContainerDied","Data":"d726624094d87ad818356939e7f01361357ea1296fbc8caf6045dcb5ad995fc4"}
Sep 09 13:26:29.237271 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:29.237248    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qrq28" event={"ID":"e61e15eb-3d0d-4a92-a91a-0987c9fd4eeb","Type":"ContainerStarted","Data":"523b87897d797c56c5d53993f64145012d20ac5e9b31d39e035793ad93603077"}
Sep 09 13:26:29.238668 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:29.238639    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sw9qv" event={"ID":"f06384ff-80d7-4e3a-a9bd-4925d55d6824","Type":"ContainerStarted","Data":"71a8e031878974664bc39cde8c77496f54d0b0fce12dab3bac3360664a4ec3a3"}
Sep 09 13:26:29.240093 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:29.240073    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7" event={"ID":"625682e0-c96b-4b33-b0b8-5b9fb93d4804","Type":"ContainerStarted","Data":"ed5f21993aeaf558762372ed1690137fd3bd7cad0902ce2db7032a80da8ae6d6"}
Sep 09 13:26:29.241454 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:29.241435    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q5njn" event={"ID":"ab687c9c-b837-4387-a696-7f24afd7711f","Type":"ContainerStarted","Data":"58c5f0b90803c46df0da7e22779767c612d890cfec4b1e8aab8b03f60c1045f2"}
Sep 09 13:26:29.249488 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:29.249445    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gwqft" podStartSLOduration=3.981019007 podStartE2EDuration="22.249430735s" totalImagesPullingTime="18.268411728s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 13:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:26:28.271872488 +0000 UTC m=+21.631722306" watchObservedRunningTime="2026-09-09 13:26:29.249430735 +0000 UTC m=+22.609280563"
Sep 09 13:26:29.305451 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:29.305094    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qrq28" podStartSLOduration=4.119247354 podStartE2EDuration="22.305071446s" totalImagesPullingTime="18.185824092s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 13:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:26:29.304917817 +0000 UTC m=+22.664767645" watchObservedRunningTime="2026-09-09 13:26:29.305071446 +0000 UTC m=+22.664921274"
Sep 09 13:26:29.306074 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:29.306021    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-sw9qv" podStartSLOduration=4.099055181 podStartE2EDuration="22.306008442s" totalImagesPullingTime="18.206953261s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 13:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:26:29.27358666 +0000 UTC m=+22.633436488" watchObservedRunningTime="2026-09-09 13:26:29.306008442 +0000 UTC m=+22.665858272"
Sep 09 13:26:29.365453 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:29.364303    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-q5njn" podStartSLOduration=4.163166764 podStartE2EDuration="22.36428177s" totalImagesPullingTime="18.201115006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:26:29.363336813 +0000 UTC m=+22.723186639" watchObservedRunningTime="2026-09-09 13:26:29.36428177 +0000 UTC m=+22.724131596"
Sep 09 13:26:29.725223 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:29.725202    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 13:26:30.141929 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:30.141805    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T13:26:29.725219452Z","UUID":"57eb5bcd-6db4-4f56-b825-9bac8493886f","Handler":null,"Name":"","Endpoint":""}
Sep 09 13:26:30.143342 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:30.143321    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 13:26:30.143473 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:30.143351    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 13:26:30.246688 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:30.246641    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal" event={"ID":"641a1c3349dfc0923275c7074bcf5214","Type":"ContainerStarted","Data":"291f5297051252f40b5cc3b1727790ad83f7b41c754cfaa7d29190d0eb36f361"}
Sep 09 13:26:30.248557 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:30.248522    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bt7v9" event={"ID":"2caadc2a-aabb-440f-8f3b-98ceb69052cd","Type":"ContainerStarted","Data":"c9e374acb08abd482e7dbdf331ecebd07a2bec8ff947617414865b160af136a8"}
Sep 09 13:26:30.251381 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:30.251353    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7" event={"ID":"625682e0-c96b-4b33-b0b8-5b9fb93d4804","Type":"ContainerStarted","Data":"3bd2af7dce4299fec6249866a3a3b35f26de0b9cd92154fcd0601ef065783b8f"}
Sep 09 13:26:30.269372 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:30.269318    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal" podStartSLOduration=22.269300092 podStartE2EDuration="22.269300092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 13:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:26:30.268894206 +0000 UTC m=+23.628744062" watchObservedRunningTime="2026-09-09 13:26:30.269300092 +0000 UTC m=+23.629149919"
Sep 09 13:26:31.143769 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:31.143624    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:31.143769 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:31.143624    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:26:31.143769 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:31.143625    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:31.144038 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:31.143796    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5dxg" podUID="9ddda0bf-9853-4015-a006-c9e44d47acd5"
Sep 09 13:26:31.144038 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:31.143892    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbqpk" podUID="49845a79-9f07-4b8c-9f9d-ad2a1035d184"
Sep 09 13:26:31.144038 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:31.143950    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plsz4" podUID="0ecc5e12-af64-4305-83ed-37552f51b76c"
Sep 09 13:26:31.257402 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:31.257362    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g" event={"ID":"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f","Type":"ContainerStarted","Data":"f16b8e44e146968011832b5e20035ba7c6de82d43c334330262e7c8ab1a53749"}
Sep 09 13:26:31.260141 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:31.260100    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7" event={"ID":"625682e0-c96b-4b33-b0b8-5b9fb93d4804","Type":"ContainerStarted","Data":"e73f35366b99d682704c2d2439fe8cd8918c0f858f3bd61e17a4ab2b679a0da0"}
Sep 09 13:26:31.287562 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:31.287508    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-flwf7" podStartSLOduration=4.064659563 podStartE2EDuration="24.28748009s" totalImagesPullingTime="20.222820527s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 13:26:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:26:31.286932106 +0000 UTC m=+24.646781932" watchObservedRunningTime="2026-09-09 13:26:31.28748009 +0000 UTC m=+24.647329915"
Sep 09 13:26:31.287759 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:31.287734    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bt7v9" podStartSLOduration=6.082056419 podStartE2EDuration="24.287727859s" totalImagesPullingTime="18.20567144s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 13:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:26:30.288768425 +0000 UTC m=+23.648618253" watchObservedRunningTime="2026-09-09 13:26:31.287727859 +0000 UTC m=+24.647577695"
Sep 09 13:26:31.804152 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:31.804104    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-q5njn"
Sep 09 13:26:31.804905 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:31.804884    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-q5njn"
Sep 09 13:26:32.262783 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:32.262696    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-q5njn"
Sep 09 13:26:32.263459 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:32.262985    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-q5njn"
Sep 09 13:26:33.146908 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:33.146880    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:26:33.147085 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:33.146913    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:33.147085 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:33.146913    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:33.147085 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:33.146991    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbqpk" podUID="49845a79-9f07-4b8c-9f9d-ad2a1035d184"
Sep 09 13:26:33.147211 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:33.147080    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plsz4" podUID="0ecc5e12-af64-4305-83ed-37552f51b76c"
Sep 09 13:26:33.147211 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:33.147183    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5dxg" podUID="9ddda0bf-9853-4015-a006-c9e44d47acd5"
Sep 09 13:26:35.147331 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:35.147096    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:26:35.147930 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:35.147145    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:35.147930 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:35.147356    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbqpk" podUID="49845a79-9f07-4b8c-9f9d-ad2a1035d184"
Sep 09 13:26:35.147930 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:35.147158    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:35.147930 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:35.147419    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plsz4" podUID="0ecc5e12-af64-4305-83ed-37552f51b76c"
Sep 09 13:26:35.147930 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:35.147519    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5dxg" podUID="9ddda0bf-9853-4015-a006-c9e44d47acd5"
Sep 09 13:26:35.272944 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:35.272906    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g" event={"ID":"3db565cf-bcf9-4a4d-b2f8-305fac6c4f3f","Type":"ContainerStarted","Data":"bebd8766009b65db37164967c93541e31b7098cb188aa2755685af5d79e01740"}
Sep 09 13:26:35.273181 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:35.273159    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:35.273259 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:35.273234    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:35.274904 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:35.274867    2566 generic.go:356] "Generic (PLEG): container finished" podID="6dde766b-3e64-4762-ab24-25b30b1b4465" containerID="9cae1052b64fe71ec027e2713e6c459e02a71250cb2eb1a09b0cb77fb968aafb" exitCode=0
Sep 09 13:26:35.275020 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:35.274919    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q6cft" event={"ID":"6dde766b-3e64-4762-ab24-25b30b1b4465","Type":"ContainerDied","Data":"9cae1052b64fe71ec027e2713e6c459e02a71250cb2eb1a09b0cb77fb968aafb"}
Sep 09 13:26:35.289678 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:35.289649    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:35.370603 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:35.370547    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g" podStartSLOduration=10.177902096 podStartE2EDuration="28.370528509s" totalImagesPullingTime="18.192626413s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 13:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:26:35.337671618 +0000 UTC m=+28.697521445" watchObservedRunningTime="2026-09-09 13:26:35.370528509 +0000 UTC m=+28.730378329"
Sep 09 13:26:36.282837 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:36.282573    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q6cft" event={"ID":"6dde766b-3e64-4762-ab24-25b30b1b4465","Type":"ContainerStarted","Data":"f89ef3490498c5470c4698f222f0269f5097b7c14b33a3bd4141dda04d23f0e5"}
Sep 09 13:26:36.283251 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:36.283158    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:36.297552 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:36.297520    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:26:36.494825 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:36.494684    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q5dxg"]
Sep 09 13:26:36.494963 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:36.494834    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:36.495009 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:36.494972    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5dxg" podUID="9ddda0bf-9853-4015-a006-c9e44d47acd5"
Sep 09 13:26:36.495288 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:36.495268    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-plsz4"]
Sep 09 13:26:36.495363 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:36.495353    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:36.495445 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:36.495427    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plsz4" podUID="0ecc5e12-af64-4305-83ed-37552f51b76c"
Sep 09 13:26:36.496051 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:36.496030    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zbqpk"]
Sep 09 13:26:36.496227 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:36.496208    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:26:36.496324 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:36.496306    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbqpk" podUID="49845a79-9f07-4b8c-9f9d-ad2a1035d184"
Sep 09 13:26:37.287012 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:37.286916    2566 generic.go:356] "Generic (PLEG): container finished" podID="6dde766b-3e64-4762-ab24-25b30b1b4465" containerID="f89ef3490498c5470c4698f222f0269f5097b7c14b33a3bd4141dda04d23f0e5" exitCode=0
Sep 09 13:26:37.287012 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:37.286996    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q6cft" event={"ID":"6dde766b-3e64-4762-ab24-25b30b1b4465","Type":"ContainerDied","Data":"f89ef3490498c5470c4698f222f0269f5097b7c14b33a3bd4141dda04d23f0e5"}
Sep 09 13:26:38.144122 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:38.144070    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:26:38.144122 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:38.144105    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:38.144334 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:38.144105    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:38.144334 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:38.144180    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbqpk" podUID="49845a79-9f07-4b8c-9f9d-ad2a1035d184"
Sep 09 13:26:38.144334 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:38.144256    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plsz4" podUID="0ecc5e12-af64-4305-83ed-37552f51b76c"
Sep 09 13:26:38.144334 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:38.144323    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5dxg" podUID="9ddda0bf-9853-4015-a006-c9e44d47acd5"
Sep 09 13:26:38.292489 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:38.292406    2566 generic.go:356] "Generic (PLEG): container finished" podID="6dde766b-3e64-4762-ab24-25b30b1b4465" containerID="326e0b0bba7280895cac45908385799244371e60202c150309b2bb4dfd549bc3" exitCode=0
Sep 09 13:26:38.292489 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:38.292478    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q6cft" event={"ID":"6dde766b-3e64-4762-ab24-25b30b1b4465","Type":"ContainerDied","Data":"326e0b0bba7280895cac45908385799244371e60202c150309b2bb4dfd549bc3"}
Sep 09 13:26:40.143735 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:40.143698    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:40.144334 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:40.143698    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:26:40.144334 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:40.143824    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plsz4" podUID="0ecc5e12-af64-4305-83ed-37552f51b76c"
Sep 09 13:26:40.144334 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:40.143698    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:40.144334 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:40.143915    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbqpk" podUID="49845a79-9f07-4b8c-9f9d-ad2a1035d184"
Sep 09 13:26:40.144334 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:40.143986    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q5dxg" podUID="9ddda0bf-9853-4015-a006-c9e44d47acd5"
Sep 09 13:26:40.831471 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:40.831372    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ecc5e12-af64-4305-83ed-37552f51b76c-original-pull-secret\") pod \"global-pull-secret-syncer-plsz4\" (UID: \"0ecc5e12-af64-4305-83ed-37552f51b76c\") " pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:40.831471 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:40.831427    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ddda0bf-9853-4015-a006-c9e44d47acd5-metrics-certs\") pod \"network-metrics-daemon-q5dxg\" (UID: \"9ddda0bf-9853-4015-a006-c9e44d47acd5\") " pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:40.831702 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:40.831526    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 13:26:40.831702 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:40.831556    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 13:26:40.831702 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:40.831592    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ddda0bf-9853-4015-a006-c9e44d47acd5-metrics-certs podName:9ddda0bf-9853-4015-a006-c9e44d47acd5 nodeName:}" failed. No retries permitted until 2026-09-09 13:27:12.831578469 +0000 UTC m=+66.191428272 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ddda0bf-9853-4015-a006-c9e44d47acd5-metrics-certs") pod "network-metrics-daemon-q5dxg" (UID: "9ddda0bf-9853-4015-a006-c9e44d47acd5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 13:26:40.831702 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:40.831616    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ecc5e12-af64-4305-83ed-37552f51b76c-original-pull-secret podName:0ecc5e12-af64-4305-83ed-37552f51b76c nodeName:}" failed. No retries permitted until 2026-09-09 13:27:12.831599372 +0000 UTC m=+66.191449188 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ecc5e12-af64-4305-83ed-37552f51b76c-original-pull-secret") pod "global-pull-secret-syncer-plsz4" (UID: "0ecc5e12-af64-4305-83ed-37552f51b76c") : object "kube-system"/"original-pull-secret" not registered
Sep 09 13:26:40.987857 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:40.987807    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeReady"
Sep 09 13:26:40.988024 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:40.987961    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 13:26:41.034011 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.033978    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-552v9\" (UniqueName: \"kubernetes.io/projected/49845a79-9f07-4b8c-9f9d-ad2a1035d184-kube-api-access-552v9\") pod \"network-check-target-zbqpk\" (UID: \"49845a79-9f07-4b8c-9f9d-ad2a1035d184\") " pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:26:41.034210 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:41.034181    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 13:26:41.034275 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:41.034212    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 13:26:41.034275 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:41.034223    2566 projected.go:196] Error preparing data for projected volume kube-api-access-552v9 for pod openshift-network-diagnostics/network-check-target-zbqpk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 13:26:41.034369 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:41.034305    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49845a79-9f07-4b8c-9f9d-ad2a1035d184-kube-api-access-552v9 podName:49845a79-9f07-4b8c-9f9d-ad2a1035d184 nodeName:}" failed. No retries permitted until 2026-09-09 13:27:13.034268901 +0000 UTC m=+66.394118709 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-552v9" (UniqueName: "kubernetes.io/projected/49845a79-9f07-4b8c-9f9d-ad2a1035d184-kube-api-access-552v9") pod "network-check-target-zbqpk" (UID: "49845a79-9f07-4b8c-9f9d-ad2a1035d184") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 13:26:41.091857 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.091759    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hsrwj"]
Sep 09 13:26:41.112853 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.112797    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wdm7n"]
Sep 09 13:26:41.113036 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.112934    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hsrwj"
Sep 09 13:26:41.119606 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.119579    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 09 13:26:41.119870 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.119852    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-q2hz7\""
Sep 09 13:26:41.123961 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.123938    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-sdx78"]
Sep 09 13:26:41.124071 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.124055    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wdm7n"
Sep 09 13:26:41.132518 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.132501    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 13:26:41.133116 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.133101    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 13:26:41.135730 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.135708    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 09 13:26:41.135833 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.135726    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-shkt9\""
Sep 09 13:26:41.136009 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.135995    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 09 13:26:41.140890 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.140869    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-5hgw6"]
Sep 09 13:26:41.141007 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.140992    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-sdx78"
Sep 09 13:26:41.142731 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.142707    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 09 13:26:41.155440 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.153726    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 09 13:26:41.155440 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.153926    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 09 13:26:41.155440 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.153935    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-jrkz6\""
Sep 09 13:26:41.155440 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.154031    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 09 13:26:41.155440 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.154174    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 09 13:26:41.155440 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.154286    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 09 13:26:41.169685 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.169652    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-z7s72"]
Sep 09 13:26:41.170026 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.169992    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-5hgw6"
Sep 09 13:26:41.182504 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.182118    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-z7s72"
Sep 09 13:26:41.182504 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.182217    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 09 13:26:41.182504 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.182298    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 09 13:26:41.186487 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.186462    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5fb8d9b79d-nxqx6"]
Sep 09 13:26:41.188308 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.188280    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 09 13:26:41.188436 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.188282    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 09 13:26:41.198136 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.198108    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-ftm96"]
Sep 09 13:26:41.198310 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.198261    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:26:41.209772 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.209314    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 09 13:26:41.209772 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.209564    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 09 13:26:41.209772 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.209592    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 13:26:41.209772 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.209567    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 13:26:41.210835 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.210797    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 09 13:26:41.211557 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.210874    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xmzdt\""
Sep 09 13:26:41.211557 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.210895    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 09 13:26:41.211724 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.210929    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 09 13:26:41.211724 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.211006    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-4rd9r\""
Sep 09 13:26:41.211724 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.211042    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 13:26:41.211724 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.211051    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 13:26:41.211724 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.211126    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 09 13:26:41.217319 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.217292    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nm782\""
Sep 09 13:26:41.221012 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.220988    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65cf7ccc98-jg9dz"]
Sep 09 13:26:41.221163 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.221118    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-ftm96"
Sep 09 13:26:41.225999 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.225976    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 09 13:26:41.226327 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.226032    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 09 13:26:41.226467 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.226448    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 09 13:26:41.226723 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.226704    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-d4tdb\""
Sep 09 13:26:41.228781 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.228758    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 09 13:26:41.229352 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.229335    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 09 13:26:41.234589 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.234566    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/717fe8eb-5931-4e1b-abce-d011c8db18dd-config\") pod \"service-ca-operator-859f94c4b-sdx78\" (UID: \"717fe8eb-5931-4e1b-abce-d011c8db18dd\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-sdx78"
Sep 09 13:26:41.234706 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.234595    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/613994fe-2f81-47d4-81f3-1b6a63682f5b-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-5hgw6\" (UID: \"613994fe-2f81-47d4-81f3-1b6a63682f5b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-5hgw6"
Sep 09 13:26:41.234706 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.234616    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2crz5\" (UniqueName: \"kubernetes.io/projected/2cd7847c-15f0-4e5a-aef5-69342b3c6499-kube-api-access-2crz5\") pod \"cluster-samples-operator-7cdcfcf76c-hsrwj\" (UID: \"2cd7847c-15f0-4e5a-aef5-69342b3c6499\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hsrwj"
Sep 09 13:26:41.234838 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.234740    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bbe27bd0-1d07-43fe-8bdc-11439590bbab-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-wdm7n\" (UID: \"bbe27bd0-1d07-43fe-8bdc-11439590bbab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wdm7n"
Sep 09 13:26:41.234838 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.234768    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv8m4\" (UniqueName: \"kubernetes.io/projected/bbe27bd0-1d07-43fe-8bdc-11439590bbab-kube-api-access-mv8m4\") pod \"cluster-monitoring-operator-6cdcc96965-wdm7n\" (UID: \"bbe27bd0-1d07-43fe-8bdc-11439590bbab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wdm7n"
Sep 09 13:26:41.234838 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.234784    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rt8s\" (UniqueName: \"kubernetes.io/projected/717fe8eb-5931-4e1b-abce-d011c8db18dd-kube-api-access-2rt8s\") pod \"service-ca-operator-859f94c4b-sdx78\" (UID: \"717fe8eb-5931-4e1b-abce-d011c8db18dd\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-sdx78"
Sep 09 13:26:41.235001 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.234847    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/717fe8eb-5931-4e1b-abce-d011c8db18dd-serving-cert\") pod \"service-ca-operator-859f94c4b-sdx78\" (UID: \"717fe8eb-5931-4e1b-abce-d011c8db18dd\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-sdx78"
Sep 09 13:26:41.235001 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.234868    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2cd7847c-15f0-4e5a-aef5-69342b3c6499-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hsrwj\" (UID: \"2cd7847c-15f0-4e5a-aef5-69342b3c6499\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hsrwj"
Sep 09 13:26:41.235001 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.234894    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/613994fe-2f81-47d4-81f3-1b6a63682f5b-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-5hgw6\" (UID: \"613994fe-2f81-47d4-81f3-1b6a63682f5b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-5hgw6"
Sep 09 13:26:41.235001 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.234916    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbs4l\" (UniqueName: \"kubernetes.io/projected/613994fe-2f81-47d4-81f3-1b6a63682f5b-kube-api-access-kbs4l\") pod \"kube-storage-version-migrator-operator-8d7dd4675-5hgw6\" (UID: \"613994fe-2f81-47d4-81f3-1b6a63682f5b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-5hgw6"
Sep 09 13:26:41.235001 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.234945    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbe27bd0-1d07-43fe-8bdc-11439590bbab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wdm7n\" (UID: \"bbe27bd0-1d07-43fe-8bdc-11439590bbab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wdm7n"
Sep 09 13:26:41.252995 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.252955    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-d6qbv"]
Sep 09 13:26:41.253722 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.253304    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 09 13:26:41.253722 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.253334    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:41.258046 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.258020    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 13:26:41.258322 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.258299    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 13:26:41.258458 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.258031    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 13:26:41.258624 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.258605    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4wzf7\""
Sep 09 13:26:41.264196 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.264174    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 13:26:41.274468 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.274433    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wdm7n"]
Sep 09 13:26:41.274637 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.274485    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-sdx78"]
Sep 09 13:26:41.274637 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.274597    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-d6qbv"
Sep 09 13:26:41.274726 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.274604    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-pvnqg"]
Sep 09 13:26:41.278716 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.278688    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-kvspm\""
Sep 09 13:26:41.278895 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.278701    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 09 13:26:41.278895 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.278756    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 09 13:26:41.292136 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.292095    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ffff64488-lk689"]
Sep 09 13:26:41.292837 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.292392    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-pvnqg"
Sep 09 13:26:41.297326 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.296421    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hsrwj"]
Sep 09 13:26:41.297326 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.296463    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-d6qbv"]
Sep 09 13:26:41.297326 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.296479    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d4cfdb4-2wxv2"]
Sep 09 13:26:41.297326 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.296549    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ffff64488-lk689"
Sep 09 13:26:41.297543 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.297380    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 13:26:41.297699 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.297649    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 13:26:41.298050 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.298032    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-8gkkw\""
Sep 09 13:26:41.299489 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.299469    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl"]
Sep 09 13:26:41.299657 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.299642    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d4cfdb4-2wxv2"
Sep 09 13:26:41.300141 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.300107    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 13:26:41.301806 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.301787    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 13:26:41.301919 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.301892    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 13:26:41.301975 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.301963    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 09 13:26:41.302064 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.302046    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-56b8c\""
Sep 09 13:26:41.302562 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.302544    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 09 13:26:41.304094 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.304076    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65cf7ccc98-jg9dz"]
Sep 09 13:26:41.304190 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.304111    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-z7s72"]
Sep 09 13:26:41.304190 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.304127    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-shzt9"]
Sep 09 13:26:41.304295 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.304194    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl"
Sep 09 13:26:41.308134 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.308111    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-5hgw6"]
Sep 09 13:26:41.308229 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.308144    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5fb8d9b79d-nxqx6"]
Sep 09 13:26:41.308229 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.308164    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d4cfdb4-2wxv2"]
Sep 09 13:26:41.308229 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.308173    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-pvnqg"]
Sep 09 13:26:41.308229 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.308185    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-shzt9"]
Sep 09 13:26:41.308229 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.308196    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl"]
Sep 09 13:26:41.308229 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.308209    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-shzt9"
Sep 09 13:26:41.308561 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.308215    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ffff64488-lk689"]
Sep 09 13:26:41.308561 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.308507    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-thhvt"]
Sep 09 13:26:41.312357 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.312336    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4dnj2"]
Sep 09 13:26:41.312496 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.312480    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-thhvt"
Sep 09 13:26:41.317594 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.317576    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 09 13:26:41.317724 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.317707    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 09 13:26:41.317872 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.317799    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 09 13:26:41.318001 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.317981    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 13:26:41.318490 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.318469    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 09 13:26:41.318700 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.318680    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-fxsgl\""
Sep 09 13:26:41.318777 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.318760    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 13:26:41.318842 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.318784    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kmkp6\""
Sep 09 13:26:41.319347 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.319331    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 09 13:26:41.322431 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.322407    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 09 13:26:41.322686 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.322668    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 13:26:41.323434 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.323417    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-ftm96"]
Sep 09 13:26:41.323491 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.323458    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-thhvt"]
Sep 09 13:26:41.323491 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.323486    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4dnj2"]
Sep 09 13:26:41.323606 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.323595    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4dnj2"
Sep 09 13:26:41.329614 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.329586    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 13:26:41.329614 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.329587    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ckz5q\""
Sep 09 13:26:41.329792 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.329689    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 13:26:41.335559 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.335525    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2fed7f4-5c70-4dc5-a61d-880e4c65492f-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-z7s72\" (UID: \"e2fed7f4-5c70-4dc5-a61d-880e4c65492f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z7s72"
Sep 09 13:26:41.335690 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.335575    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2crz5\" (UniqueName: \"kubernetes.io/projected/2cd7847c-15f0-4e5a-aef5-69342b3c6499-kube-api-access-2crz5\") pod \"cluster-samples-operator-7cdcfcf76c-hsrwj\" (UID: \"2cd7847c-15f0-4e5a-aef5-69342b3c6499\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hsrwj"
Sep 09 13:26:41.335690 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.335624    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/717fe8eb-5931-4e1b-abce-d011c8db18dd-config\") pod \"service-ca-operator-859f94c4b-sdx78\" (UID: \"717fe8eb-5931-4e1b-abce-d011c8db18dd\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-sdx78"
Sep 09 13:26:41.335690 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.335649    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6210e253-1f3a-480f-84de-2cc61e20c535-stats-auth\") pod \"router-default-5fb8d9b79d-nxqx6\" (UID: \"6210e253-1f3a-480f-84de-2cc61e20c535\") " pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:26:41.335690 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.335673    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-ca-trust-extracted\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:41.335929 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.335698    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/613994fe-2f81-47d4-81f3-1b6a63682f5b-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-5hgw6\" (UID: \"613994fe-2f81-47d4-81f3-1b6a63682f5b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-5hgw6"
Sep 09 13:26:41.335929 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.335720    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6210e253-1f3a-480f-84de-2cc61e20c535-metrics-certs\") pod \"router-default-5fb8d9b79d-nxqx6\" (UID: \"6210e253-1f3a-480f-84de-2cc61e20c535\") " pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:26:41.335929 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.335768    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/af902e8b-d693-4341-9979-03cf69548cfb-trusted-ca\") pod \"console-operator-85b454874b-ftm96\" (UID: \"af902e8b-d693-4341-9979-03cf69548cfb\") " pod="openshift-console-operator/console-operator-85b454874b-ftm96"
Sep 09 13:26:41.335929 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.335796    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrvkx\" (UniqueName: \"kubernetes.io/projected/97bffee4-6cb2-4047-8a04-e92e83309608-kube-api-access-wrvkx\") pod \"volume-data-source-validator-695c895bd6-d6qbv\" (UID: \"97bffee4-6cb2-4047-8a04-e92e83309608\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-d6qbv"
Sep 09 13:26:41.335929 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.335834    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgs4m\" (UniqueName: \"kubernetes.io/projected/6210e253-1f3a-480f-84de-2cc61e20c535-kube-api-access-xgs4m\") pod \"router-default-5fb8d9b79d-nxqx6\" (UID: \"6210e253-1f3a-480f-84de-2cc61e20c535\") " pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:26:41.335929 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.335856    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2fed7f4-5c70-4dc5-a61d-880e4c65492f-tmp\") pod \"insights-operator-76bcd6ccf5-z7s72\" (UID: \"e2fed7f4-5c70-4dc5-a61d-880e4c65492f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z7s72"
Sep 09 13:26:41.335929 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.335883    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-trusted-ca\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:41.335929 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.335916    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-tls\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:41.336268 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.335938    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-installation-pull-secrets\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:41.336268 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.335965    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rt8s\" (UniqueName: \"kubernetes.io/projected/717fe8eb-5931-4e1b-abce-d011c8db18dd-kube-api-access-2rt8s\") pod \"service-ca-operator-859f94c4b-sdx78\" (UID: \"717fe8eb-5931-4e1b-abce-d011c8db18dd\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-sdx78"
Sep 09 13:26:41.336268 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.335989    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e2fed7f4-5c70-4dc5-a61d-880e4c65492f-snapshots\") pod \"insights-operator-76bcd6ccf5-z7s72\" (UID: \"e2fed7f4-5c70-4dc5-a61d-880e4c65492f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z7s72"
Sep 09 13:26:41.336268 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.336022    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/717fe8eb-5931-4e1b-abce-d011c8db18dd-serving-cert\") pod \"service-ca-operator-859f94c4b-sdx78\" (UID: \"717fe8eb-5931-4e1b-abce-d011c8db18dd\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-sdx78"
Sep 09 13:26:41.336268 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.336051    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2cd7847c-15f0-4e5a-aef5-69342b3c6499-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hsrwj\" (UID: \"2cd7847c-15f0-4e5a-aef5-69342b3c6499\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hsrwj"
Sep 09 13:26:41.336268 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.336078    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbe27bd0-1d07-43fe-8bdc-11439590bbab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wdm7n\" (UID: \"bbe27bd0-1d07-43fe-8bdc-11439590bbab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wdm7n"
Sep 09 13:26:41.336268 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.336103    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2fed7f4-5c70-4dc5-a61d-880e4c65492f-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-z7s72\" (UID: \"e2fed7f4-5c70-4dc5-a61d-880e4c65492f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z7s72"
Sep 09 13:26:41.336268 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.336126    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6210e253-1f3a-480f-84de-2cc61e20c535-default-certificate\") pod \"router-default-5fb8d9b79d-nxqx6\" (UID: \"6210e253-1f3a-480f-84de-2cc61e20c535\") " pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:26:41.336268 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.336158    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2fed7f4-5c70-4dc5-a61d-880e4c65492f-serving-cert\") pod \"insights-operator-76bcd6ccf5-z7s72\" (UID: \"e2fed7f4-5c70-4dc5-a61d-880e4c65492f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z7s72"
Sep 09 13:26:41.336268 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.336186    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/af902e8b-d693-4341-9979-03cf69548cfb-config\") pod \"console-operator-85b454874b-ftm96\" (UID: \"af902e8b-d693-4341-9979-03cf69548cfb\") " pod="openshift-console-operator/console-operator-85b454874b-ftm96"
Sep 09 13:26:41.336268 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.336211    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-bound-sa-token\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:41.336268 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.336234    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qv4b\" (UniqueName: \"kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-kube-api-access-7qv4b\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:41.336268 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:41.336258    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 13:26:41.336741 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.336279    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-image-registry-private-configuration\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:41.336741 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:41.336321    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cd7847c-15f0-4e5a-aef5-69342b3c6499-samples-operator-tls podName:2cd7847c-15f0-4e5a-aef5-69342b3c6499 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:41.836303303 +0000 UTC m=+35.196153120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2cd7847c-15f0-4e5a-aef5-69342b3c6499-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-hsrwj" (UID: "2cd7847c-15f0-4e5a-aef5-69342b3c6499") : secret "samples-operator-tls" not found
Sep 09 13:26:41.336741 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.336340    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6210e253-1f3a-480f-84de-2cc61e20c535-service-ca-bundle\") pod \"router-default-5fb8d9b79d-nxqx6\" (UID: \"6210e253-1f3a-480f-84de-2cc61e20c535\") " pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:26:41.336741 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.336377    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bbe27bd0-1d07-43fe-8bdc-11439590bbab-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-wdm7n\" (UID: \"bbe27bd0-1d07-43fe-8bdc-11439590bbab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wdm7n"
Sep 09 13:26:41.336741 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:41.336397    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 13:26:41.336741 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.336404    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7l72\" (UniqueName: \"kubernetes.io/projected/af902e8b-d693-4341-9979-03cf69548cfb-kube-api-access-m7l72\") pod \"console-operator-85b454874b-ftm96\" (UID: \"af902e8b-d693-4341-9979-03cf69548cfb\") " pod="openshift-console-operator/console-operator-85b454874b-ftm96"
Sep 09 13:26:41.336741 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:41.336429    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbe27bd0-1d07-43fe-8bdc-11439590bbab-cluster-monitoring-operator-tls podName:bbe27bd0-1d07-43fe-8bdc-11439590bbab nodeName:}" failed. No retries permitted until 2026-09-09 13:26:41.836419263 +0000 UTC m=+35.196269080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bbe27bd0-1d07-43fe-8bdc-11439590bbab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-wdm7n" (UID: "bbe27bd0-1d07-43fe-8bdc-11439590bbab") : secret "cluster-monitoring-operator-tls" not found
Sep 09 13:26:41.336741 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.336445    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/af902e8b-d693-4341-9979-03cf69548cfb-serving-cert\") pod \"console-operator-85b454874b-ftm96\" (UID: \"af902e8b-d693-4341-9979-03cf69548cfb\") " pod="openshift-console-operator/console-operator-85b454874b-ftm96"
Sep 09 13:26:41.336741 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.336474    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv8m4\" (UniqueName: \"kubernetes.io/projected/bbe27bd0-1d07-43fe-8bdc-11439590bbab-kube-api-access-mv8m4\") pod \"cluster-monitoring-operator-6cdcc96965-wdm7n\" (UID: \"bbe27bd0-1d07-43fe-8bdc-11439590bbab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wdm7n"
Sep 09 13:26:41.337322 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.336756    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-certificates\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:41.337322 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.336790    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nchw\" (UniqueName: \"kubernetes.io/projected/e2fed7f4-5c70-4dc5-a61d-880e4c65492f-kube-api-access-9nchw\") pod \"insights-operator-76bcd6ccf5-z7s72\" (UID: \"e2fed7f4-5c70-4dc5-a61d-880e4c65492f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z7s72"
Sep 09 13:26:41.337322 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.336838    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/613994fe-2f81-47d4-81f3-1b6a63682f5b-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-5hgw6\" (UID: \"613994fe-2f81-47d4-81f3-1b6a63682f5b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-5hgw6"
Sep 09 13:26:41.337322 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.336861    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbs4l\" (UniqueName: \"kubernetes.io/projected/613994fe-2f81-47d4-81f3-1b6a63682f5b-kube-api-access-kbs4l\") pod \"kube-storage-version-migrator-operator-8d7dd4675-5hgw6\" (UID: \"613994fe-2f81-47d4-81f3-1b6a63682f5b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-5hgw6"
Sep 09 13:26:41.337322 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.337230    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/717fe8eb-5931-4e1b-abce-d011c8db18dd-config\") pod \"service-ca-operator-859f94c4b-sdx78\" (UID: \"717fe8eb-5931-4e1b-abce-d011c8db18dd\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-sdx78"
Sep 09 13:26:41.337514 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.337440    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bbe27bd0-1d07-43fe-8bdc-11439590bbab-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-wdm7n\" (UID: \"bbe27bd0-1d07-43fe-8bdc-11439590bbab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wdm7n"
Sep 09 13:26:41.337631 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.337608    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/613994fe-2f81-47d4-81f3-1b6a63682f5b-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-5hgw6\" (UID: \"613994fe-2f81-47d4-81f3-1b6a63682f5b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-5hgw6"
Sep 09 13:26:41.341515 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.341392    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/717fe8eb-5931-4e1b-abce-d011c8db18dd-serving-cert\") pod \"service-ca-operator-859f94c4b-sdx78\" (UID: \"717fe8eb-5931-4e1b-abce-d011c8db18dd\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-sdx78"
Sep 09 13:26:41.341611 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.341392    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/613994fe-2f81-47d4-81f3-1b6a63682f5b-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-5hgw6\" (UID: \"613994fe-2f81-47d4-81f3-1b6a63682f5b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-5hgw6"
Sep 09 13:26:41.364211 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.364146    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rt8s\" (UniqueName: \"kubernetes.io/projected/717fe8eb-5931-4e1b-abce-d011c8db18dd-kube-api-access-2rt8s\") pod \"service-ca-operator-859f94c4b-sdx78\" (UID: \"717fe8eb-5931-4e1b-abce-d011c8db18dd\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-sdx78"
Sep 09 13:26:41.364650 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.364625    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2crz5\" (UniqueName: \"kubernetes.io/projected/2cd7847c-15f0-4e5a-aef5-69342b3c6499-kube-api-access-2crz5\") pod \"cluster-samples-operator-7cdcfcf76c-hsrwj\" (UID: \"2cd7847c-15f0-4e5a-aef5-69342b3c6499\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hsrwj"
Sep 09 13:26:41.369020 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.368995    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv8m4\" (UniqueName: \"kubernetes.io/projected/bbe27bd0-1d07-43fe-8bdc-11439590bbab-kube-api-access-mv8m4\") pod \"cluster-monitoring-operator-6cdcc96965-wdm7n\" (UID: \"bbe27bd0-1d07-43fe-8bdc-11439590bbab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wdm7n"
Sep 09 13:26:41.371345 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.371311    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbs4l\" (UniqueName: \"kubernetes.io/projected/613994fe-2f81-47d4-81f3-1b6a63682f5b-kube-api-access-kbs4l\") pod \"kube-storage-version-migrator-operator-8d7dd4675-5hgw6\" (UID: \"613994fe-2f81-47d4-81f3-1b6a63682f5b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-5hgw6"
Sep 09 13:26:41.437598 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.437558    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be3c3076-7d51-4348-ba53-1d350b9f69b6-tmp-dir\") pod \"dns-default-4dnj2\" (UID: \"be3c3076-7d51-4348-ba53-1d350b9f69b6\") " pod="openshift-dns/dns-default-4dnj2"
Sep 09 13:26:41.437598 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.437595    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edc23e7d-c615-47bf-8614-0ed3cd38c69b-tmp\") pod \"klusterlet-addon-workmgr-6ffff64488-lk689\" (UID: \"edc23e7d-c615-47bf-8614-0ed3cd38c69b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ffff64488-lk689"
Sep 09 13:26:41.437837 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.437628    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpcth\" (UniqueName: \"kubernetes.io/projected/fee30cf9-8a3a-466d-8460-8ec8b363ff6d-kube-api-access-cpcth\") pod \"ingress-canary-thhvt\" (UID: \"fee30cf9-8a3a-466d-8460-8ec8b363ff6d\") " pod="openshift-ingress-canary/ingress-canary-thhvt"
Sep 09 13:26:41.437837 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.437682    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2fed7f4-5c70-4dc5-a61d-880e4c65492f-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-z7s72\" (UID: \"e2fed7f4-5c70-4dc5-a61d-880e4c65492f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z7s72"
Sep 09 13:26:41.437837 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.437798    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6br92\" (UniqueName: \"kubernetes.io/projected/c838ec52-950d-4e56-a558-657a595ef5ac-kube-api-access-6br92\") pod \"network-check-source-685456655f-pvnqg\" (UID: \"c838ec52-950d-4e56-a558-657a595ef5ac\") " pod="openshift-network-diagnostics/network-check-source-685456655f-pvnqg"
Sep 09 13:26:41.438001 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.437873    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6210e253-1f3a-480f-84de-2cc61e20c535-default-certificate\") pod \"router-default-5fb8d9b79d-nxqx6\" (UID: \"6210e253-1f3a-480f-84de-2cc61e20c535\") " pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:26:41.438001 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.437899    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e858dafd-7a28-4576-9897-802ab13a56c4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d7d4cfdb4-2wxv2\" (UID: \"e858dafd-7a28-4576-9897-802ab13a56c4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d4cfdb4-2wxv2"
Sep 09 13:26:41.438001 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.437924    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c-nginx-conf\") pod \"networking-console-plugin-79977f9844-shzt9\" (UID: \"6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c\") " pod="openshift-network-console/networking-console-plugin-79977f9844-shzt9"
Sep 09 13:26:41.438001 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.437945    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6e9b0e9f-87db-45fd-abd3-bd3956bc9805-ca\") pod \"cluster-proxy-proxy-agent-7d5f6575bc-cfqzl\" (UID: \"6e9b0e9f-87db-45fd-abd3-bd3956bc9805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl"
Sep 09 13:26:41.438001 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.437973    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2fed7f4-5c70-4dc5-a61d-880e4c65492f-serving-cert\") pod \"insights-operator-76bcd6ccf5-z7s72\" (UID: \"e2fed7f4-5c70-4dc5-a61d-880e4c65492f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z7s72"
Sep 09 13:26:41.438199 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.438069    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/af902e8b-d693-4341-9979-03cf69548cfb-config\") pod \"console-operator-85b454874b-ftm96\" (UID: \"af902e8b-d693-4341-9979-03cf69548cfb\") " pod="openshift-console-operator/console-operator-85b454874b-ftm96"
Sep 09 13:26:41.438199 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.438104    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjggv\" (UniqueName: \"kubernetes.io/projected/6e9b0e9f-87db-45fd-abd3-bd3956bc9805-kube-api-access-cjggv\") pod \"cluster-proxy-proxy-agent-7d5f6575bc-cfqzl\" (UID: \"6e9b0e9f-87db-45fd-abd3-bd3956bc9805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl"
Sep 09 13:26:41.438199 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.438134    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-bound-sa-token\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:41.438199 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.438159    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qv4b\" (UniqueName: \"kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-kube-api-access-7qv4b\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:41.438339 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.438204    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-image-registry-private-configuration\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:41.438339 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.438233    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6210e253-1f3a-480f-84de-2cc61e20c535-service-ca-bundle\") pod \"router-default-5fb8d9b79d-nxqx6\" (UID: \"6210e253-1f3a-480f-84de-2cc61e20c535\") " pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:26:41.438339 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.438268    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6e9b0e9f-87db-45fd-abd3-bd3956bc9805-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d5f6575bc-cfqzl\" (UID: \"6e9b0e9f-87db-45fd-abd3-bd3956bc9805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl"
Sep 09 13:26:41.438339 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.438272    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2fed7f4-5c70-4dc5-a61d-880e4c65492f-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-z7s72\" (UID: \"e2fed7f4-5c70-4dc5-a61d-880e4c65492f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z7s72"
Sep 09 13:26:41.438339 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.438320    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7l72\" (UniqueName: \"kubernetes.io/projected/af902e8b-d693-4341-9979-03cf69548cfb-kube-api-access-m7l72\") pod \"console-operator-85b454874b-ftm96\" (UID: \"af902e8b-d693-4341-9979-03cf69548cfb\") " pod="openshift-console-operator/console-operator-85b454874b-ftm96"
Sep 09 13:26:41.438526 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.438347    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-shzt9\" (UID: \"6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c\") " pod="openshift-network-console/networking-console-plugin-79977f9844-shzt9"
Sep 09 13:26:41.438526 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.438375    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/af902e8b-d693-4341-9979-03cf69548cfb-serving-cert\") pod \"console-operator-85b454874b-ftm96\" (UID: \"af902e8b-d693-4341-9979-03cf69548cfb\") " pod="openshift-console-operator/console-operator-85b454874b-ftm96"
Sep 09 13:26:41.438526 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.438403    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-certificates\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:41.438526 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.438433    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nchw\" (UniqueName: \"kubernetes.io/projected/e2fed7f4-5c70-4dc5-a61d-880e4c65492f-kube-api-access-9nchw\") pod \"insights-operator-76bcd6ccf5-z7s72\" (UID: \"e2fed7f4-5c70-4dc5-a61d-880e4c65492f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z7s72"
Sep 09 13:26:41.438526 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.438458    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/edc23e7d-c615-47bf-8614-0ed3cd38c69b-klusterlet-config\") pod \"klusterlet-addon-workmgr-6ffff64488-lk689\" (UID: \"edc23e7d-c615-47bf-8614-0ed3cd38c69b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ffff64488-lk689"
Sep 09 13:26:41.438526 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.438489    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2fed7f4-5c70-4dc5-a61d-880e4c65492f-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-z7s72\" (UID: \"e2fed7f4-5c70-4dc5-a61d-880e4c65492f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z7s72"
Sep 09 13:26:41.438526 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.438514    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fee30cf9-8a3a-466d-8460-8ec8b363ff6d-cert\") pod \"ingress-canary-thhvt\" (UID: \"fee30cf9-8a3a-466d-8460-8ec8b363ff6d\") " pod="openshift-ingress-canary/ingress-canary-thhvt"
Sep 09 13:26:41.438861 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.438562    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6210e253-1f3a-480f-84de-2cc61e20c535-stats-auth\") pod \"router-default-5fb8d9b79d-nxqx6\" (UID: \"6210e253-1f3a-480f-84de-2cc61e20c535\") " pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:26:41.438861 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.438588    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be3c3076-7d51-4348-ba53-1d350b9f69b6-metrics-tls\") pod \"dns-default-4dnj2\" (UID: \"be3c3076-7d51-4348-ba53-1d350b9f69b6\") " pod="openshift-dns/dns-default-4dnj2"
Sep 09 13:26:41.438861 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.438612    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e9b0e9f-87db-45fd-abd3-bd3956bc9805-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d5f6575bc-cfqzl\" (UID: \"6e9b0e9f-87db-45fd-abd3-bd3956bc9805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl"
Sep 09 13:26:41.438861 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.438637    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-ca-trust-extracted\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:41.438861 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.438668    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6210e253-1f3a-480f-84de-2cc61e20c535-metrics-certs\") pod \"router-default-5fb8d9b79d-nxqx6\" (UID: \"6210e253-1f3a-480f-84de-2cc61e20c535\") " pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:26:41.438861 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.438698    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bns2l\" (UniqueName: \"kubernetes.io/projected/edc23e7d-c615-47bf-8614-0ed3cd38c69b-kube-api-access-bns2l\") pod \"klusterlet-addon-workmgr-6ffff64488-lk689\" (UID: \"edc23e7d-c615-47bf-8614-0ed3cd38c69b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ffff64488-lk689"
Sep 09 13:26:41.438861 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.438752    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/af902e8b-d693-4341-9979-03cf69548cfb-trusted-ca\") pod \"console-operator-85b454874b-ftm96\" (UID: \"af902e8b-d693-4341-9979-03cf69548cfb\") " pod="openshift-console-operator/console-operator-85b454874b-ftm96"
Sep 09 13:26:41.438861 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.438848    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/af902e8b-d693-4341-9979-03cf69548cfb-config\") pod \"console-operator-85b454874b-ftm96\" (UID: \"af902e8b-d693-4341-9979-03cf69548cfb\") " pod="openshift-console-operator/console-operator-85b454874b-ftm96"
Sep 09 13:26:41.439280 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:41.439184    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 13:26:41.439280 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:41.439278    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6210e253-1f3a-480f-84de-2cc61e20c535-metrics-certs podName:6210e253-1f3a-480f-84de-2cc61e20c535 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:41.939257624 +0000 UTC m=+35.299107433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6210e253-1f3a-480f-84de-2cc61e20c535-metrics-certs") pod "router-default-5fb8d9b79d-nxqx6" (UID: "6210e253-1f3a-480f-84de-2cc61e20c535") : secret "router-metrics-certs-default" not found
Sep 09 13:26:41.439557 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.439535    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e9b0e9f-87db-45fd-abd3-bd3956bc9805-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d5f6575bc-cfqzl\" (UID: \"6e9b0e9f-87db-45fd-abd3-bd3956bc9805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl"
Sep 09 13:26:41.439557 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.439551    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2fed7f4-5c70-4dc5-a61d-880e4c65492f-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-z7s72\" (UID: \"e2fed7f4-5c70-4dc5-a61d-880e4c65492f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z7s72"
Sep 09 13:26:41.439681 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.439588    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrvkx\" (UniqueName: \"kubernetes.io/projected/97bffee4-6cb2-4047-8a04-e92e83309608-kube-api-access-wrvkx\") pod \"volume-data-source-validator-695c895bd6-d6qbv\" (UID: \"97bffee4-6cb2-4047-8a04-e92e83309608\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-d6qbv"
Sep 09 13:26:41.439681 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.439623    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nnks\" (UniqueName: \"kubernetes.io/projected/e858dafd-7a28-4576-9897-802ab13a56c4-kube-api-access-9nnks\") pod \"managed-serviceaccount-addon-agent-d7d4cfdb4-2wxv2\" (UID: \"e858dafd-7a28-4576-9897-802ab13a56c4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d4cfdb4-2wxv2"
Sep 09 13:26:41.439681 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.439651    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6e9b0e9f-87db-45fd-abd3-bd3956bc9805-hub\") pod \"cluster-proxy-proxy-agent-7d5f6575bc-cfqzl\" (UID: \"6e9b0e9f-87db-45fd-abd3-bd3956bc9805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl"
Sep 09 13:26:41.439681 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.439680    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgs4m\" (UniqueName: \"kubernetes.io/projected/6210e253-1f3a-480f-84de-2cc61e20c535-kube-api-access-xgs4m\") pod \"router-default-5fb8d9b79d-nxqx6\" (UID: \"6210e253-1f3a-480f-84de-2cc61e20c535\") " pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:26:41.439906 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.439706    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2fed7f4-5c70-4dc5-a61d-880e4c65492f-tmp\") pod \"insights-operator-76bcd6ccf5-z7s72\" (UID: \"e2fed7f4-5c70-4dc5-a61d-880e4c65492f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z7s72"
Sep 09 13:26:41.439906 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.439789    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-trusted-ca\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:41.439906 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.439833    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kchf\" (UniqueName: \"kubernetes.io/projected/be3c3076-7d51-4348-ba53-1d350b9f69b6-kube-api-access-2kchf\") pod \"dns-default-4dnj2\" (UID: \"be3c3076-7d51-4348-ba53-1d350b9f69b6\") " pod="openshift-dns/dns-default-4dnj2"
Sep 09 13:26:41.439906 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.439877    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-tls\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:41.439906 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.439904    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-installation-pull-secrets\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:41.440141 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.439935    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e2fed7f4-5c70-4dc5-a61d-880e4c65492f-snapshots\") pod \"insights-operator-76bcd6ccf5-z7s72\" (UID: \"e2fed7f4-5c70-4dc5-a61d-880e4c65492f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z7s72"
Sep 09 13:26:41.440141 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.440031    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/be3c3076-7d51-4348-ba53-1d350b9f69b6-config-volume\") pod \"dns-default-4dnj2\" (UID: \"be3c3076-7d51-4348-ba53-1d350b9f69b6\") " pod="openshift-dns/dns-default-4dnj2"
Sep 09 13:26:41.440736 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.440325    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/af902e8b-d693-4341-9979-03cf69548cfb-trusted-ca\") pod \"console-operator-85b454874b-ftm96\" (UID: \"af902e8b-d693-4341-9979-03cf69548cfb\") " pod="openshift-console-operator/console-operator-85b454874b-ftm96"
Sep 09 13:26:41.440736 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.440591    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-ca-trust-extracted\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:41.440736 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.440686    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e2fed7f4-5c70-4dc5-a61d-880e4c65492f-snapshots\") pod \"insights-operator-76bcd6ccf5-z7s72\" (UID: \"e2fed7f4-5c70-4dc5-a61d-880e4c65492f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z7s72"
Sep 09 13:26:41.440952 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.440932    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2fed7f4-5c70-4dc5-a61d-880e4c65492f-serving-cert\") pod \"insights-operator-76bcd6ccf5-z7s72\" (UID: \"e2fed7f4-5c70-4dc5-a61d-880e4c65492f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z7s72"
Sep 09 13:26:41.440997 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.440956    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6210e253-1f3a-480f-84de-2cc61e20c535-default-certificate\") pod \"router-default-5fb8d9b79d-nxqx6\" (UID: \"6210e253-1f3a-480f-84de-2cc61e20c535\") " pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:26:41.441048 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:41.441040    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 13:26:41.441102 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:41.441052    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65cf7ccc98-jg9dz: secret "image-registry-tls" not found
Sep 09 13:26:41.441102 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.441054    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2fed7f4-5c70-4dc5-a61d-880e4c65492f-tmp\") pod \"insights-operator-76bcd6ccf5-z7s72\" (UID: \"e2fed7f4-5c70-4dc5-a61d-880e4c65492f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z7s72"
Sep 09 13:26:41.441364 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:41.441113    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-tls podName:2adb3e0a-ed98-4c24-8f7e-7209c7a4324d nodeName:}" failed. No retries permitted until 2026-09-09 13:26:41.941095948 +0000 UTC m=+35.300945755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-tls") pod "image-registry-65cf7ccc98-jg9dz" (UID: "2adb3e0a-ed98-4c24-8f7e-7209c7a4324d") : secret "image-registry-tls" not found
Sep 09 13:26:41.441443 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.441419    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6210e253-1f3a-480f-84de-2cc61e20c535-stats-auth\") pod \"router-default-5fb8d9b79d-nxqx6\" (UID: \"6210e253-1f3a-480f-84de-2cc61e20c535\") " pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:26:41.441500 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.441449    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/af902e8b-d693-4341-9979-03cf69548cfb-serving-cert\") pod \"console-operator-85b454874b-ftm96\" (UID: \"af902e8b-d693-4341-9979-03cf69548cfb\") " pod="openshift-console-operator/console-operator-85b454874b-ftm96"
Sep 09 13:26:41.441549 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.441521    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-trusted-ca\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:41.441589 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:41.441550    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6210e253-1f3a-480f-84de-2cc61e20c535-service-ca-bundle podName:6210e253-1f3a-480f-84de-2cc61e20c535 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:41.941533095 +0000 UTC m=+35.301382917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6210e253-1f3a-480f-84de-2cc61e20c535-service-ca-bundle") pod "router-default-5fb8d9b79d-nxqx6" (UID: "6210e253-1f3a-480f-84de-2cc61e20c535") : configmap references non-existent config key: service-ca.crt
Sep 09 13:26:41.441670 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.441644    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-certificates\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:41.443287 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.443267    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-image-registry-private-configuration\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:41.443608 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.443587    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-installation-pull-secrets\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:41.452912 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.452884    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-sdx78"
Sep 09 13:26:41.463439 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.463404    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qv4b\" (UniqueName: \"kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-kube-api-access-7qv4b\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:41.474876 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.474846    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgs4m\" (UniqueName: \"kubernetes.io/projected/6210e253-1f3a-480f-84de-2cc61e20c535-kube-api-access-xgs4m\") pod \"router-default-5fb8d9b79d-nxqx6\" (UID: \"6210e253-1f3a-480f-84de-2cc61e20c535\") " pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:26:41.479256 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.479222    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-5hgw6"
Sep 09 13:26:41.479476 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.479442    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrvkx\" (UniqueName: \"kubernetes.io/projected/97bffee4-6cb2-4047-8a04-e92e83309608-kube-api-access-wrvkx\") pod \"volume-data-source-validator-695c895bd6-d6qbv\" (UID: \"97bffee4-6cb2-4047-8a04-e92e83309608\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-d6qbv"
Sep 09 13:26:41.479587 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.479444    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7l72\" (UniqueName: \"kubernetes.io/projected/af902e8b-d693-4341-9979-03cf69548cfb-kube-api-access-m7l72\") pod \"console-operator-85b454874b-ftm96\" (UID: \"af902e8b-d693-4341-9979-03cf69548cfb\") " pod="openshift-console-operator/console-operator-85b454874b-ftm96"
Sep 09 13:26:41.479587 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.479511    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nchw\" (UniqueName: \"kubernetes.io/projected/e2fed7f4-5c70-4dc5-a61d-880e4c65492f-kube-api-access-9nchw\") pod \"insights-operator-76bcd6ccf5-z7s72\" (UID: \"e2fed7f4-5c70-4dc5-a61d-880e4c65492f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z7s72"
Sep 09 13:26:41.493788 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.493753    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-z7s72"
Sep 09 13:26:41.497912 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.497881    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-bound-sa-token\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:41.532063 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.532020    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-ftm96"
Sep 09 13:26:41.541459 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.541419    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/be3c3076-7d51-4348-ba53-1d350b9f69b6-config-volume\") pod \"dns-default-4dnj2\" (UID: \"be3c3076-7d51-4348-ba53-1d350b9f69b6\") " pod="openshift-dns/dns-default-4dnj2"
Sep 09 13:26:41.541651 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.541471    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be3c3076-7d51-4348-ba53-1d350b9f69b6-tmp-dir\") pod \"dns-default-4dnj2\" (UID: \"be3c3076-7d51-4348-ba53-1d350b9f69b6\") " pod="openshift-dns/dns-default-4dnj2"
Sep 09 13:26:41.541651 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.541502    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edc23e7d-c615-47bf-8614-0ed3cd38c69b-tmp\") pod \"klusterlet-addon-workmgr-6ffff64488-lk689\" (UID: \"edc23e7d-c615-47bf-8614-0ed3cd38c69b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ffff64488-lk689"
Sep 09 13:26:41.541651 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.541547    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpcth\" (UniqueName: \"kubernetes.io/projected/fee30cf9-8a3a-466d-8460-8ec8b363ff6d-kube-api-access-cpcth\") pod \"ingress-canary-thhvt\" (UID: \"fee30cf9-8a3a-466d-8460-8ec8b363ff6d\") " pod="openshift-ingress-canary/ingress-canary-thhvt"
Sep 09 13:26:41.541651 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.541588    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6br92\" (UniqueName: \"kubernetes.io/projected/c838ec52-950d-4e56-a558-657a595ef5ac-kube-api-access-6br92\") pod \"network-check-source-685456655f-pvnqg\" (UID: \"c838ec52-950d-4e56-a558-657a595ef5ac\") " pod="openshift-network-diagnostics/network-check-source-685456655f-pvnqg"
Sep 09 13:26:41.541651 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.541619    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e858dafd-7a28-4576-9897-802ab13a56c4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d7d4cfdb4-2wxv2\" (UID: \"e858dafd-7a28-4576-9897-802ab13a56c4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d4cfdb4-2wxv2"
Sep 09 13:26:41.541931 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.541668    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c-nginx-conf\") pod \"networking-console-plugin-79977f9844-shzt9\" (UID: \"6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c\") " pod="openshift-network-console/networking-console-plugin-79977f9844-shzt9"
Sep 09 13:26:41.541931 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.541696    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6e9b0e9f-87db-45fd-abd3-bd3956bc9805-ca\") pod \"cluster-proxy-proxy-agent-7d5f6575bc-cfqzl\" (UID: \"6e9b0e9f-87db-45fd-abd3-bd3956bc9805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl"
Sep 09 13:26:41.541931 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.541737    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjggv\" (UniqueName: \"kubernetes.io/projected/6e9b0e9f-87db-45fd-abd3-bd3956bc9805-kube-api-access-cjggv\") pod \"cluster-proxy-proxy-agent-7d5f6575bc-cfqzl\" (UID: \"6e9b0e9f-87db-45fd-abd3-bd3956bc9805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl"
Sep 09 13:26:41.541931 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.541791    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6e9b0e9f-87db-45fd-abd3-bd3956bc9805-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d5f6575bc-cfqzl\" (UID: \"6e9b0e9f-87db-45fd-abd3-bd3956bc9805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl"
Sep 09 13:26:41.541931 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.541847    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-shzt9\" (UID: \"6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c\") " pod="openshift-network-console/networking-console-plugin-79977f9844-shzt9"
Sep 09 13:26:41.541931 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.541885    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/edc23e7d-c615-47bf-8614-0ed3cd38c69b-klusterlet-config\") pod \"klusterlet-addon-workmgr-6ffff64488-lk689\" (UID: \"edc23e7d-c615-47bf-8614-0ed3cd38c69b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ffff64488-lk689"
Sep 09 13:26:41.541931 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.541916    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fee30cf9-8a3a-466d-8460-8ec8b363ff6d-cert\") pod \"ingress-canary-thhvt\" (UID: \"fee30cf9-8a3a-466d-8460-8ec8b363ff6d\") " pod="openshift-ingress-canary/ingress-canary-thhvt"
Sep 09 13:26:41.542262 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.541962    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be3c3076-7d51-4348-ba53-1d350b9f69b6-metrics-tls\") pod \"dns-default-4dnj2\" (UID: \"be3c3076-7d51-4348-ba53-1d350b9f69b6\") " pod="openshift-dns/dns-default-4dnj2"
Sep 09 13:26:41.542262 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.541987    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e9b0e9f-87db-45fd-abd3-bd3956bc9805-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d5f6575bc-cfqzl\" (UID: \"6e9b0e9f-87db-45fd-abd3-bd3956bc9805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl"
Sep 09 13:26:41.542262 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.542024    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bns2l\" (UniqueName: \"kubernetes.io/projected/edc23e7d-c615-47bf-8614-0ed3cd38c69b-kube-api-access-bns2l\") pod \"klusterlet-addon-workmgr-6ffff64488-lk689\" (UID: \"edc23e7d-c615-47bf-8614-0ed3cd38c69b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ffff64488-lk689"
Sep 09 13:26:41.542262 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.542107    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e9b0e9f-87db-45fd-abd3-bd3956bc9805-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d5f6575bc-cfqzl\" (UID: \"6e9b0e9f-87db-45fd-abd3-bd3956bc9805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl"
Sep 09 13:26:41.542262 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.542140    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nnks\" (UniqueName: \"kubernetes.io/projected/e858dafd-7a28-4576-9897-802ab13a56c4-kube-api-access-9nnks\") pod \"managed-serviceaccount-addon-agent-d7d4cfdb4-2wxv2\" (UID: \"e858dafd-7a28-4576-9897-802ab13a56c4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d4cfdb4-2wxv2"
Sep 09 13:26:41.542262 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.542143    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/be3c3076-7d51-4348-ba53-1d350b9f69b6-config-volume\") pod \"dns-default-4dnj2\" (UID: \"be3c3076-7d51-4348-ba53-1d350b9f69b6\") " pod="openshift-dns/dns-default-4dnj2"
Sep 09 13:26:41.542262 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.542164    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6e9b0e9f-87db-45fd-abd3-bd3956bc9805-hub\") pod \"cluster-proxy-proxy-agent-7d5f6575bc-cfqzl\" (UID: \"6e9b0e9f-87db-45fd-abd3-bd3956bc9805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl"
Sep 09 13:26:41.542262 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.542208    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kchf\" (UniqueName: \"kubernetes.io/projected/be3c3076-7d51-4348-ba53-1d350b9f69b6-kube-api-access-2kchf\") pod \"dns-default-4dnj2\" (UID: \"be3c3076-7d51-4348-ba53-1d350b9f69b6\") " pod="openshift-dns/dns-default-4dnj2"
Sep 09 13:26:41.542759 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:41.542628    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 13:26:41.542759 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:41.542704    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c-networking-console-plugin-cert podName:6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c nodeName:}" failed. No retries permitted until 2026-09-09 13:26:42.042683364 +0000 UTC m=+35.402533171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-shzt9" (UID: "6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c") : secret "networking-console-plugin-cert" not found
Sep 09 13:26:41.542973 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:41.542946    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 13:26:41.543137 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.542988    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c-nginx-conf\") pod \"networking-console-plugin-79977f9844-shzt9\" (UID: \"6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c\") " pod="openshift-network-console/networking-console-plugin-79977f9844-shzt9"
Sep 09 13:26:41.543137 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:41.542996    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be3c3076-7d51-4348-ba53-1d350b9f69b6-metrics-tls podName:be3c3076-7d51-4348-ba53-1d350b9f69b6 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:42.042979525 +0000 UTC m=+35.402829334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/be3c3076-7d51-4348-ba53-1d350b9f69b6-metrics-tls") pod "dns-default-4dnj2" (UID: "be3c3076-7d51-4348-ba53-1d350b9f69b6") : secret "dns-default-metrics-tls" not found
Sep 09 13:26:41.543245 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.543217    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be3c3076-7d51-4348-ba53-1d350b9f69b6-tmp-dir\") pod \"dns-default-4dnj2\" (UID: \"be3c3076-7d51-4348-ba53-1d350b9f69b6\") " pod="openshift-dns/dns-default-4dnj2"
Sep 09 13:26:41.544071 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.543637    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e9b0e9f-87db-45fd-abd3-bd3956bc9805-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d5f6575bc-cfqzl\" (UID: \"6e9b0e9f-87db-45fd-abd3-bd3956bc9805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl"
Sep 09 13:26:41.544071 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.543985    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edc23e7d-c615-47bf-8614-0ed3cd38c69b-tmp\") pod \"klusterlet-addon-workmgr-6ffff64488-lk689\" (UID: \"edc23e7d-c615-47bf-8614-0ed3cd38c69b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ffff64488-lk689"
Sep 09 13:26:41.546030 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.545524    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fee30cf9-8a3a-466d-8460-8ec8b363ff6d-cert\") pod \"ingress-canary-thhvt\" (UID: \"fee30cf9-8a3a-466d-8460-8ec8b363ff6d\") " pod="openshift-ingress-canary/ingress-canary-thhvt"
Sep 09 13:26:41.546030 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.545988    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6e9b0e9f-87db-45fd-abd3-bd3956bc9805-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d5f6575bc-cfqzl\" (UID: \"6e9b0e9f-87db-45fd-abd3-bd3956bc9805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl"
Sep 09 13:26:41.546201 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.546091    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6e9b0e9f-87db-45fd-abd3-bd3956bc9805-hub\") pod \"cluster-proxy-proxy-agent-7d5f6575bc-cfqzl\" (UID: \"6e9b0e9f-87db-45fd-abd3-bd3956bc9805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl"
Sep 09 13:26:41.546201 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.546130    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/edc23e7d-c615-47bf-8614-0ed3cd38c69b-klusterlet-config\") pod \"klusterlet-addon-workmgr-6ffff64488-lk689\" (UID: \"edc23e7d-c615-47bf-8614-0ed3cd38c69b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ffff64488-lk689"
Sep 09 13:26:41.546793 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.546755    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6e9b0e9f-87db-45fd-abd3-bd3956bc9805-ca\") pod \"cluster-proxy-proxy-agent-7d5f6575bc-cfqzl\" (UID: \"6e9b0e9f-87db-45fd-abd3-bd3956bc9805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl"
Sep 09 13:26:41.547235 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.547210    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e858dafd-7a28-4576-9897-802ab13a56c4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d7d4cfdb4-2wxv2\" (UID: \"e858dafd-7a28-4576-9897-802ab13a56c4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d4cfdb4-2wxv2"
Sep 09 13:26:41.547421 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.547403    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e9b0e9f-87db-45fd-abd3-bd3956bc9805-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d5f6575bc-cfqzl\" (UID: \"6e9b0e9f-87db-45fd-abd3-bd3956bc9805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl"
Sep 09 13:26:41.559704 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.559671    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6br92\" (UniqueName: \"kubernetes.io/projected/c838ec52-950d-4e56-a558-657a595ef5ac-kube-api-access-6br92\") pod \"network-check-source-685456655f-pvnqg\" (UID: \"c838ec52-950d-4e56-a558-657a595ef5ac\") " pod="openshift-network-diagnostics/network-check-source-685456655f-pvnqg"
Sep 09 13:26:41.580072 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.580045    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bns2l\" (UniqueName: \"kubernetes.io/projected/edc23e7d-c615-47bf-8614-0ed3cd38c69b-kube-api-access-bns2l\") pod \"klusterlet-addon-workmgr-6ffff64488-lk689\" (UID: \"edc23e7d-c615-47bf-8614-0ed3cd38c69b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ffff64488-lk689"
Sep 09 13:26:41.580213 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.580101    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpcth\" (UniqueName: \"kubernetes.io/projected/fee30cf9-8a3a-466d-8460-8ec8b363ff6d-kube-api-access-cpcth\") pod \"ingress-canary-thhvt\" (UID: \"fee30cf9-8a3a-466d-8460-8ec8b363ff6d\") " pod="openshift-ingress-canary/ingress-canary-thhvt"
Sep 09 13:26:41.581721 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.581694    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjggv\" (UniqueName: \"kubernetes.io/projected/6e9b0e9f-87db-45fd-abd3-bd3956bc9805-kube-api-access-cjggv\") pod \"cluster-proxy-proxy-agent-7d5f6575bc-cfqzl\" (UID: \"6e9b0e9f-87db-45fd-abd3-bd3956bc9805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl"
Sep 09 13:26:41.582726 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.582697    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kchf\" (UniqueName: \"kubernetes.io/projected/be3c3076-7d51-4348-ba53-1d350b9f69b6-kube-api-access-2kchf\") pod \"dns-default-4dnj2\" (UID: \"be3c3076-7d51-4348-ba53-1d350b9f69b6\") " pod="openshift-dns/dns-default-4dnj2"
Sep 09 13:26:41.584581 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.584557    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-d6qbv"
Sep 09 13:26:41.593303 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.593272    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nnks\" (UniqueName: \"kubernetes.io/projected/e858dafd-7a28-4576-9897-802ab13a56c4-kube-api-access-9nnks\") pod \"managed-serviceaccount-addon-agent-d7d4cfdb4-2wxv2\" (UID: \"e858dafd-7a28-4576-9897-802ab13a56c4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d4cfdb4-2wxv2"
Sep 09 13:26:41.596711 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.596668    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q5njn_ab687c9c-b837-4387-a696-7f24afd7711f/konnectivity-agent/0.log" limit=1024
Sep 09 13:26:41.605410 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.605378    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-pvnqg"
Sep 09 13:26:41.612216 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.612192    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ffff64488-lk689"
Sep 09 13:26:41.633835 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.633783    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d4cfdb4-2wxv2"
Sep 09 13:26:41.642950 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.642913    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl"
Sep 09 13:26:41.689843 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.689795    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-thhvt"
Sep 09 13:26:41.819334 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.819289    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q5njn_ab687c9c-b837-4387-a696-7f24afd7711f/konnectivity-agent/0.log" limit=1024
Sep 09 13:26:41.844669 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.844634    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2cd7847c-15f0-4e5a-aef5-69342b3c6499-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hsrwj\" (UID: \"2cd7847c-15f0-4e5a-aef5-69342b3c6499\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hsrwj"
Sep 09 13:26:41.844861 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.844685    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbe27bd0-1d07-43fe-8bdc-11439590bbab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wdm7n\" (UID: \"bbe27bd0-1d07-43fe-8bdc-11439590bbab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wdm7n"
Sep 09 13:26:41.844861 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:41.844802    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 13:26:41.844967 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:41.844886    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cd7847c-15f0-4e5a-aef5-69342b3c6499-samples-operator-tls podName:2cd7847c-15f0-4e5a-aef5-69342b3c6499 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:42.84486625 +0000 UTC m=+36.204716055 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2cd7847c-15f0-4e5a-aef5-69342b3c6499-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-hsrwj" (UID: "2cd7847c-15f0-4e5a-aef5-69342b3c6499") : secret "samples-operator-tls" not found
Sep 09 13:26:41.844967 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:41.844805    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 13:26:41.845059 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:41.845001    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbe27bd0-1d07-43fe-8bdc-11439590bbab-cluster-monitoring-operator-tls podName:bbe27bd0-1d07-43fe-8bdc-11439590bbab nodeName:}" failed. No retries permitted until 2026-09-09 13:26:42.844984139 +0000 UTC m=+36.204833961 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bbe27bd0-1d07-43fe-8bdc-11439590bbab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-wdm7n" (UID: "bbe27bd0-1d07-43fe-8bdc-11439590bbab") : secret "cluster-monitoring-operator-tls" not found
Sep 09 13:26:41.945422 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.945335    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6210e253-1f3a-480f-84de-2cc61e20c535-service-ca-bundle\") pod \"router-default-5fb8d9b79d-nxqx6\" (UID: \"6210e253-1f3a-480f-84de-2cc61e20c535\") " pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:26:41.945586 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.945444    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6210e253-1f3a-480f-84de-2cc61e20c535-metrics-certs\") pod \"router-default-5fb8d9b79d-nxqx6\" (UID: \"6210e253-1f3a-480f-84de-2cc61e20c535\") " pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:26:41.945586 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:41.945497    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-tls\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:41.945586 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:41.945557    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6210e253-1f3a-480f-84de-2cc61e20c535-service-ca-bundle podName:6210e253-1f3a-480f-84de-2cc61e20c535 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:42.945538849 +0000 UTC m=+36.305388658 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6210e253-1f3a-480f-84de-2cc61e20c535-service-ca-bundle") pod "router-default-5fb8d9b79d-nxqx6" (UID: "6210e253-1f3a-480f-84de-2cc61e20c535") : configmap references non-existent config key: service-ca.crt
Sep 09 13:26:41.945754 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:41.945610    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 13:26:41.945754 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:41.945637    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 13:26:41.945754 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:41.945657    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65cf7ccc98-jg9dz: secret "image-registry-tls" not found
Sep 09 13:26:41.945754 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:41.945666    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6210e253-1f3a-480f-84de-2cc61e20c535-metrics-certs podName:6210e253-1f3a-480f-84de-2cc61e20c535 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:42.945652385 +0000 UTC m=+36.305502211 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6210e253-1f3a-480f-84de-2cc61e20c535-metrics-certs") pod "router-default-5fb8d9b79d-nxqx6" (UID: "6210e253-1f3a-480f-84de-2cc61e20c535") : secret "router-metrics-certs-default" not found
Sep 09 13:26:41.945754 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:41.945696    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-tls podName:2adb3e0a-ed98-4c24-8f7e-7209c7a4324d nodeName:}" failed. No retries permitted until 2026-09-09 13:26:42.945682382 +0000 UTC m=+36.305532203 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-tls") pod "image-registry-65cf7ccc98-jg9dz" (UID: "2adb3e0a-ed98-4c24-8f7e-7209c7a4324d") : secret "image-registry-tls" not found
Sep 09 13:26:42.046798 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:42.046751    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-shzt9\" (UID: \"6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c\") " pod="openshift-network-console/networking-console-plugin-79977f9844-shzt9"
Sep 09 13:26:42.047003 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:42.046868    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be3c3076-7d51-4348-ba53-1d350b9f69b6-metrics-tls\") pod \"dns-default-4dnj2\" (UID: \"be3c3076-7d51-4348-ba53-1d350b9f69b6\") " pod="openshift-dns/dns-default-4dnj2"
Sep 09 13:26:42.047003 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:42.046944    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 13:26:42.047111 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:42.047019    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 13:26:42.047111 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:42.047042    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c-networking-console-plugin-cert podName:6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c nodeName:}" failed. No retries permitted until 2026-09-09 13:26:43.047018708 +0000 UTC m=+36.406868535 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-shzt9" (UID: "6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c") : secret "networking-console-plugin-cert" not found
Sep 09 13:26:42.047111 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:42.047079    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be3c3076-7d51-4348-ba53-1d350b9f69b6-metrics-tls podName:be3c3076-7d51-4348-ba53-1d350b9f69b6 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:43.047066869 +0000 UTC m=+36.406916697 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/be3c3076-7d51-4348-ba53-1d350b9f69b6-metrics-tls") pod "dns-default-4dnj2" (UID: "be3c3076-7d51-4348-ba53-1d350b9f69b6") : secret "dns-default-metrics-tls" not found
Sep 09 13:26:42.144098 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:42.144061    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:26:42.144298 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:42.144081    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:26:42.144298 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:42.144081    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:26:42.147562 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:42.147290    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ps8jz\""
Sep 09 13:26:42.147562 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:42.147339    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zpj6j\""
Sep 09 13:26:42.147562 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:42.147359    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 13:26:42.147562 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:42.147521    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 13:26:42.855327 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:42.854410    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2cd7847c-15f0-4e5a-aef5-69342b3c6499-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hsrwj\" (UID: \"2cd7847c-15f0-4e5a-aef5-69342b3c6499\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hsrwj"
Sep 09 13:26:42.855327 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:42.854496    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbe27bd0-1d07-43fe-8bdc-11439590bbab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wdm7n\" (UID: \"bbe27bd0-1d07-43fe-8bdc-11439590bbab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wdm7n"
Sep 09 13:26:42.855327 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:42.854653    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 13:26:42.855327 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:42.854736    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbe27bd0-1d07-43fe-8bdc-11439590bbab-cluster-monitoring-operator-tls podName:bbe27bd0-1d07-43fe-8bdc-11439590bbab nodeName:}" failed. No retries permitted until 2026-09-09 13:26:44.854715745 +0000 UTC m=+38.214565565 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bbe27bd0-1d07-43fe-8bdc-11439590bbab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-wdm7n" (UID: "bbe27bd0-1d07-43fe-8bdc-11439590bbab") : secret "cluster-monitoring-operator-tls" not found
Sep 09 13:26:42.855327 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:42.855226    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 13:26:42.855327 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:42.855280    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cd7847c-15f0-4e5a-aef5-69342b3c6499-samples-operator-tls podName:2cd7847c-15f0-4e5a-aef5-69342b3c6499 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:44.855262474 +0000 UTC m=+38.215112291 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2cd7847c-15f0-4e5a-aef5-69342b3c6499-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-hsrwj" (UID: "2cd7847c-15f0-4e5a-aef5-69342b3c6499") : secret "samples-operator-tls" not found
Sep 09 13:26:42.955918 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:42.955880    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6210e253-1f3a-480f-84de-2cc61e20c535-metrics-certs\") pod \"router-default-5fb8d9b79d-nxqx6\" (UID: \"6210e253-1f3a-480f-84de-2cc61e20c535\") " pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:26:42.956069 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:42.955935    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-tls\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:42.956069 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:42.956003    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6210e253-1f3a-480f-84de-2cc61e20c535-service-ca-bundle\") pod \"router-default-5fb8d9b79d-nxqx6\" (UID: \"6210e253-1f3a-480f-84de-2cc61e20c535\") " pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:26:42.956238 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:42.956181    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6210e253-1f3a-480f-84de-2cc61e20c535-service-ca-bundle podName:6210e253-1f3a-480f-84de-2cc61e20c535 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:44.956163878 +0000 UTC m=+38.316013686 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6210e253-1f3a-480f-84de-2cc61e20c535-service-ca-bundle") pod "router-default-5fb8d9b79d-nxqx6" (UID: "6210e253-1f3a-480f-84de-2cc61e20c535") : configmap references non-existent config key: service-ca.crt
Sep 09 13:26:42.956333 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:42.956283    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 13:26:42.956333 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:42.956302    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65cf7ccc98-jg9dz: secret "image-registry-tls" not found
Sep 09 13:26:42.956421 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:42.956344    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-tls podName:2adb3e0a-ed98-4c24-8f7e-7209c7a4324d nodeName:}" failed. No retries permitted until 2026-09-09 13:26:44.956331727 +0000 UTC m=+38.316181546 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-tls") pod "image-registry-65cf7ccc98-jg9dz" (UID: "2adb3e0a-ed98-4c24-8f7e-7209c7a4324d") : secret "image-registry-tls" not found
Sep 09 13:26:42.956421 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:42.956347    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 13:26:42.956421 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:42.956404    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6210e253-1f3a-480f-84de-2cc61e20c535-metrics-certs podName:6210e253-1f3a-480f-84de-2cc61e20c535 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:44.956391192 +0000 UTC m=+38.316241019 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6210e253-1f3a-480f-84de-2cc61e20c535-metrics-certs") pod "router-default-5fb8d9b79d-nxqx6" (UID: "6210e253-1f3a-480f-84de-2cc61e20c535") : secret "router-metrics-certs-default" not found
Sep 09 13:26:43.056994 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:43.056953    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-shzt9\" (UID: \"6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c\") " pod="openshift-network-console/networking-console-plugin-79977f9844-shzt9"
Sep 09 13:26:43.057158 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:43.057099    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be3c3076-7d51-4348-ba53-1d350b9f69b6-metrics-tls\") pod \"dns-default-4dnj2\" (UID: \"be3c3076-7d51-4348-ba53-1d350b9f69b6\") " pod="openshift-dns/dns-default-4dnj2"
Sep 09 13:26:43.057158 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:43.057109    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 13:26:43.057280 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:43.057176    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c-networking-console-plugin-cert podName:6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c nodeName:}" failed. No retries permitted until 2026-09-09 13:26:45.057160579 +0000 UTC m=+38.417010386 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-shzt9" (UID: "6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c") : secret "networking-console-plugin-cert" not found
Sep 09 13:26:43.057280 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:43.057198    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 13:26:43.057280 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:43.057237    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be3c3076-7d51-4348-ba53-1d350b9f69b6-metrics-tls podName:be3c3076-7d51-4348-ba53-1d350b9f69b6 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:45.05722583 +0000 UTC m=+38.417075650 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/be3c3076-7d51-4348-ba53-1d350b9f69b6-metrics-tls") pod "dns-default-4dnj2" (UID: "be3c3076-7d51-4348-ba53-1d350b9f69b6") : secret "dns-default-metrics-tls" not found
Sep 09 13:26:43.135725 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:43.135542    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-d6qbv"]
Sep 09 13:26:43.138267 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:43.138230    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97bffee4_6cb2_4047_8a04_e92e83309608.slice/crio-4d02e77a0a2a6568e976924394a7752cacc9b5d608329c3a17d56e9f312bb2c1 WatchSource:0}: Error finding container 4d02e77a0a2a6568e976924394a7752cacc9b5d608329c3a17d56e9f312bb2c1: Status 404 returned error can't find the container with id 4d02e77a0a2a6568e976924394a7752cacc9b5d608329c3a17d56e9f312bb2c1
Sep 09 13:26:43.149170 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:43.149125    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-thhvt"]
Sep 09 13:26:43.149297 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:43.149183    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-sdx78"]
Sep 09 13:26:43.149455 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:43.149433    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717fe8eb_5931_4e1b_abce_d011c8db18dd.slice/crio-2fef297772cb1da47d9d95f5838115b0e2c856e53129548670f610540914428f WatchSource:0}: Error finding container 2fef297772cb1da47d9d95f5838115b0e2c856e53129548670f610540914428f: Status 404 returned error can't find the container with id 2fef297772cb1da47d9d95f5838115b0e2c856e53129548670f610540914428f
Sep 09 13:26:43.150082 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:43.150063    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee30cf9_8a3a_466d_8460_8ec8b363ff6d.slice/crio-f0a2b0afc03952f6b5a7b377e760290cf849a89661cb5097c33cda9463b187c2 WatchSource:0}: Error finding container f0a2b0afc03952f6b5a7b377e760290cf849a89661cb5097c33cda9463b187c2: Status 404 returned error can't find the container with id f0a2b0afc03952f6b5a7b377e760290cf849a89661cb5097c33cda9463b187c2
Sep 09 13:26:43.160769 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:43.160659    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-ftm96"]
Sep 09 13:26:43.168033 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:43.168013    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl"]
Sep 09 13:26:43.181075 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:43.180990    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-5hgw6"]
Sep 09 13:26:43.182156 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:43.182031    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ffff64488-lk689"]
Sep 09 13:26:43.183157 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:43.183127    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-z7s72"]
Sep 09 13:26:43.190699 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:43.190667    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-pvnqg"]
Sep 09 13:26:43.216700 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:43.216670    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d4cfdb4-2wxv2"]
Sep 09 13:26:43.220023 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:43.219956    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc838ec52_950d_4e56_a558_657a595ef5ac.slice/crio-cbd94d52b084d1fc2b9bc9be2b5ec257f3c1a0320e3480afdf05155ded67cfd9 WatchSource:0}: Error finding container cbd94d52b084d1fc2b9bc9be2b5ec257f3c1a0320e3480afdf05155ded67cfd9: Status 404 returned error can't find the container with id cbd94d52b084d1fc2b9bc9be2b5ec257f3c1a0320e3480afdf05155ded67cfd9
Sep 09 13:26:43.220352 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:43.220269    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode858dafd_7a28_4576_9897_802ab13a56c4.slice/crio-2cf7fe2c029f4b97c5d43757bb4eb7ad0b7fbc048d2f275974b21bd798d523fb WatchSource:0}: Error finding container 2cf7fe2c029f4b97c5d43757bb4eb7ad0b7fbc048d2f275974b21bd798d523fb: Status 404 returned error can't find the container with id 2cf7fe2c029f4b97c5d43757bb4eb7ad0b7fbc048d2f275974b21bd798d523fb
Sep 09 13:26:43.221164 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:43.220940    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf902e8b_d693_4341_9979_03cf69548cfb.slice/crio-611776fbb56fd31a02ac562e2b2f2dbe63622fc286b78ff7dd0bed050847bae6 WatchSource:0}: Error finding container 611776fbb56fd31a02ac562e2b2f2dbe63622fc286b78ff7dd0bed050847bae6: Status 404 returned error can't find the container with id 611776fbb56fd31a02ac562e2b2f2dbe63622fc286b78ff7dd0bed050847bae6
Sep 09 13:26:43.222169 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:43.221909    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e9b0e9f_87db_45fd_abd3_bd3956bc9805.slice/crio-68d88e07b2912e2ee7db39665b1cfcef353cc1c4e0e9f3bd62845bc9142bbae4 WatchSource:0}: Error finding container 68d88e07b2912e2ee7db39665b1cfcef353cc1c4e0e9f3bd62845bc9142bbae4: Status 404 returned error can't find the container with id 68d88e07b2912e2ee7db39665b1cfcef353cc1c4e0e9f3bd62845bc9142bbae4
Sep 09 13:26:43.224656 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:43.223134    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc23e7d_c615_47bf_8614_0ed3cd38c69b.slice/crio-28e3dcd5de7120622ed8158fe4a61286366217fb4202f44c43db67d077dd3df3 WatchSource:0}: Error finding container 28e3dcd5de7120622ed8158fe4a61286366217fb4202f44c43db67d077dd3df3: Status 404 returned error can't find the container with id 28e3dcd5de7120622ed8158fe4a61286366217fb4202f44c43db67d077dd3df3
Sep 09 13:26:43.224656 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:43.224154    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613994fe_2f81_47d4_81f3_1b6a63682f5b.slice/crio-a08dfeb835c3cc1e218b332d380cab17ed4db9600e7467b5240a192d645e751e WatchSource:0}: Error finding container a08dfeb835c3cc1e218b332d380cab17ed4db9600e7467b5240a192d645e751e: Status 404 returned error can't find the container with id a08dfeb835c3cc1e218b332d380cab17ed4db9600e7467b5240a192d645e751e
Sep 09 13:26:43.225610 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:26:43.225388    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2fed7f4_5c70_4dc5_a61d_880e4c65492f.slice/crio-ac05c0bab36db6e12b8ad75aa0809482c3f416453ab1a9a1ca764bd776a0a9ee WatchSource:0}: Error finding container ac05c0bab36db6e12b8ad75aa0809482c3f416453ab1a9a1ca764bd776a0a9ee: Status 404 returned error can't find the container with id ac05c0bab36db6e12b8ad75aa0809482c3f416453ab1a9a1ca764bd776a0a9ee
Sep 09 13:26:43.246376 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:43.246330    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q5njn_ab687c9c-b837-4387-a696-7f24afd7711f/konnectivity-agent/0.log" limit=1024
Sep 09 13:26:43.306438 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:43.306322    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-pvnqg" event={"ID":"c838ec52-950d-4e56-a558-657a595ef5ac","Type":"ContainerStarted","Data":"cbd94d52b084d1fc2b9bc9be2b5ec257f3c1a0320e3480afdf05155ded67cfd9"}
Sep 09 13:26:43.307388 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:43.307368    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-d6qbv" event={"ID":"97bffee4-6cb2-4047-8a04-e92e83309608","Type":"ContainerStarted","Data":"4d02e77a0a2a6568e976924394a7752cacc9b5d608329c3a17d56e9f312bb2c1"}
Sep 09 13:26:43.308428 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:43.308403    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-thhvt" event={"ID":"fee30cf9-8a3a-466d-8460-8ec8b363ff6d","Type":"ContainerStarted","Data":"f0a2b0afc03952f6b5a7b377e760290cf849a89661cb5097c33cda9463b187c2"}
Sep 09 13:26:43.310932 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:43.310894    2566 generic.go:356] "Generic (PLEG): container finished" podID="6dde766b-3e64-4762-ab24-25b30b1b4465" containerID="79eec5975329fd5df6f7dab36bcbd73a19f2ca35f3410886a3fadf7991fa1601" exitCode=0
Sep 09 13:26:43.311015 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:43.310965    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q6cft" event={"ID":"6dde766b-3e64-4762-ab24-25b30b1b4465","Type":"ContainerDied","Data":"79eec5975329fd5df6f7dab36bcbd73a19f2ca35f3410886a3fadf7991fa1601"}
Sep 09 13:26:43.312121 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:43.312104    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-5hgw6" event={"ID":"613994fe-2f81-47d4-81f3-1b6a63682f5b","Type":"ContainerStarted","Data":"a08dfeb835c3cc1e218b332d380cab17ed4db9600e7467b5240a192d645e751e"}
Sep 09 13:26:43.313196 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:43.313177    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl" event={"ID":"6e9b0e9f-87db-45fd-abd3-bd3956bc9805","Type":"ContainerStarted","Data":"68d88e07b2912e2ee7db39665b1cfcef353cc1c4e0e9f3bd62845bc9142bbae4"}
Sep 09 13:26:43.314257 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:43.314232    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-ftm96" event={"ID":"af902e8b-d693-4341-9979-03cf69548cfb","Type":"ContainerStarted","Data":"611776fbb56fd31a02ac562e2b2f2dbe63622fc286b78ff7dd0bed050847bae6"}
Sep 09 13:26:43.315335 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:43.315316    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-z7s72" event={"ID":"e2fed7f4-5c70-4dc5-a61d-880e4c65492f","Type":"ContainerStarted","Data":"ac05c0bab36db6e12b8ad75aa0809482c3f416453ab1a9a1ca764bd776a0a9ee"}
Sep 09 13:26:43.316385 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:43.316369    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ffff64488-lk689" event={"ID":"edc23e7d-c615-47bf-8614-0ed3cd38c69b","Type":"ContainerStarted","Data":"28e3dcd5de7120622ed8158fe4a61286366217fb4202f44c43db67d077dd3df3"}
Sep 09 13:26:43.317482 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:43.317465    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d4cfdb4-2wxv2" event={"ID":"e858dafd-7a28-4576-9897-802ab13a56c4","Type":"ContainerStarted","Data":"2cf7fe2c029f4b97c5d43757bb4eb7ad0b7fbc048d2f275974b21bd798d523fb"}
Sep 09 13:26:43.318615 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:43.318591    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-sdx78" event={"ID":"717fe8eb-5931-4e1b-abce-d011c8db18dd","Type":"ContainerStarted","Data":"2fef297772cb1da47d9d95f5838115b0e2c856e53129548670f610540914428f"}
Sep 09 13:26:43.430343 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:43.430307    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q5njn_ab687c9c-b837-4387-a696-7f24afd7711f/konnectivity-agent/0.log" limit=1024
Sep 09 13:26:43.991037 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:43.990979    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q5njn_ab687c9c-b837-4387-a696-7f24afd7711f/konnectivity-agent/0.log" limit=1024
Sep 09 13:26:44.109154 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:44.109098    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q5njn_ab687c9c-b837-4387-a696-7f24afd7711f/konnectivity-agent/0.log" limit=1024
Sep 09 13:26:44.344076 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:44.342964    2566 generic.go:356] "Generic (PLEG): container finished" podID="6dde766b-3e64-4762-ab24-25b30b1b4465" containerID="572ce37776c4dbec803bef9fd480333bd6d4685c13774f51162a1452274d6a75" exitCode=0
Sep 09 13:26:44.344076 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:44.343027    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q6cft" event={"ID":"6dde766b-3e64-4762-ab24-25b30b1b4465","Type":"ContainerDied","Data":"572ce37776c4dbec803bef9fd480333bd6d4685c13774f51162a1452274d6a75"}
Sep 09 13:26:44.878951 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:44.878908    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbe27bd0-1d07-43fe-8bdc-11439590bbab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wdm7n\" (UID: \"bbe27bd0-1d07-43fe-8bdc-11439590bbab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wdm7n"
Sep 09 13:26:44.879186 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:44.879163    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2cd7847c-15f0-4e5a-aef5-69342b3c6499-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hsrwj\" (UID: \"2cd7847c-15f0-4e5a-aef5-69342b3c6499\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hsrwj"
Sep 09 13:26:44.879370 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:44.879315    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 13:26:44.879434 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:44.879386    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cd7847c-15f0-4e5a-aef5-69342b3c6499-samples-operator-tls podName:2cd7847c-15f0-4e5a-aef5-69342b3c6499 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:48.879366569 +0000 UTC m=+42.239216388 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2cd7847c-15f0-4e5a-aef5-69342b3c6499-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-hsrwj" (UID: "2cd7847c-15f0-4e5a-aef5-69342b3c6499") : secret "samples-operator-tls" not found
Sep 09 13:26:44.879520 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:44.879455    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 13:26:44.879520 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:44.879488    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbe27bd0-1d07-43fe-8bdc-11439590bbab-cluster-monitoring-operator-tls podName:bbe27bd0-1d07-43fe-8bdc-11439590bbab nodeName:}" failed. No retries permitted until 2026-09-09 13:26:48.879476471 +0000 UTC m=+42.239326279 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bbe27bd0-1d07-43fe-8bdc-11439590bbab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-wdm7n" (UID: "bbe27bd0-1d07-43fe-8bdc-11439590bbab") : secret "cluster-monitoring-operator-tls" not found
Sep 09 13:26:44.981548 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:44.980502    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-tls\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:44.981548 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:44.980637    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6210e253-1f3a-480f-84de-2cc61e20c535-service-ca-bundle\") pod \"router-default-5fb8d9b79d-nxqx6\" (UID: \"6210e253-1f3a-480f-84de-2cc61e20c535\") " pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:26:44.981548 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:44.980735    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6210e253-1f3a-480f-84de-2cc61e20c535-metrics-certs\") pod \"router-default-5fb8d9b79d-nxqx6\" (UID: \"6210e253-1f3a-480f-84de-2cc61e20c535\") " pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:26:44.981548 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:44.980889    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 13:26:44.981548 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:44.980960    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6210e253-1f3a-480f-84de-2cc61e20c535-metrics-certs podName:6210e253-1f3a-480f-84de-2cc61e20c535 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:48.980938666 +0000 UTC m=+42.340788540 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6210e253-1f3a-480f-84de-2cc61e20c535-metrics-certs") pod "router-default-5fb8d9b79d-nxqx6" (UID: "6210e253-1f3a-480f-84de-2cc61e20c535") : secret "router-metrics-certs-default" not found
Sep 09 13:26:44.981548 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:44.981374    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 13:26:44.981548 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:44.981392    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65cf7ccc98-jg9dz: secret "image-registry-tls" not found
Sep 09 13:26:44.981548 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:44.981441    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-tls podName:2adb3e0a-ed98-4c24-8f7e-7209c7a4324d nodeName:}" failed. No retries permitted until 2026-09-09 13:26:48.98142592 +0000 UTC m=+42.341275740 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-tls") pod "image-registry-65cf7ccc98-jg9dz" (UID: "2adb3e0a-ed98-4c24-8f7e-7209c7a4324d") : secret "image-registry-tls" not found
Sep 09 13:26:44.981548 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:44.981510    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6210e253-1f3a-480f-84de-2cc61e20c535-service-ca-bundle podName:6210e253-1f3a-480f-84de-2cc61e20c535 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:48.981498338 +0000 UTC m=+42.341348167 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6210e253-1f3a-480f-84de-2cc61e20c535-service-ca-bundle") pod "router-default-5fb8d9b79d-nxqx6" (UID: "6210e253-1f3a-480f-84de-2cc61e20c535") : configmap references non-existent config key: service-ca.crt
Sep 09 13:26:45.082410 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:45.081457    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-shzt9\" (UID: \"6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c\") " pod="openshift-network-console/networking-console-plugin-79977f9844-shzt9"
Sep 09 13:26:45.082410 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:45.081540    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be3c3076-7d51-4348-ba53-1d350b9f69b6-metrics-tls\") pod \"dns-default-4dnj2\" (UID: \"be3c3076-7d51-4348-ba53-1d350b9f69b6\") " pod="openshift-dns/dns-default-4dnj2"
Sep 09 13:26:45.082410 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:45.081685    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 13:26:45.082410 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:45.081745    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be3c3076-7d51-4348-ba53-1d350b9f69b6-metrics-tls podName:be3c3076-7d51-4348-ba53-1d350b9f69b6 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:49.081726405 +0000 UTC m=+42.441576223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/be3c3076-7d51-4348-ba53-1d350b9f69b6-metrics-tls") pod "dns-default-4dnj2" (UID: "be3c3076-7d51-4348-ba53-1d350b9f69b6") : secret "dns-default-metrics-tls" not found
Sep 09 13:26:45.082410 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:45.082310    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 13:26:45.082410 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:45.082370    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c-networking-console-plugin-cert podName:6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c nodeName:}" failed. No retries permitted until 2026-09-09 13:26:49.08235287 +0000 UTC m=+42.442202689 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-shzt9" (UID: "6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c") : secret "networking-console-plugin-cert" not found
Sep 09 13:26:45.894587 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:45.894551    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q5njn_ab687c9c-b837-4387-a696-7f24afd7711f/konnectivity-agent/0.log" limit=1024
Sep 09 13:26:46.329880 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:46.329748    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q5njn_ab687c9c-b837-4387-a696-7f24afd7711f/konnectivity-agent/0.log" limit=1024
Sep 09 13:26:46.377896 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:46.377348    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-d6qbv" event={"ID":"97bffee4-6cb2-4047-8a04-e92e83309608","Type":"ContainerStarted","Data":"92a2613da8faadbba2064c031fe1bfd892cec5fff71c79ac08feee900eb06957"}
Sep 09 13:26:46.412253 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:46.411100    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-d6qbv" podStartSLOduration=35.55065817 podStartE2EDuration="38.411076333s" totalImagesPullingTime="2.860418163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:26:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:26:46.410144859 +0000 UTC m=+39.769994678" watchObservedRunningTime="2026-09-09 13:26:46.411076333 +0000 UTC m=+39.770926160"
Sep 09 13:26:46.413955 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:46.413851    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q6cft" event={"ID":"6dde766b-3e64-4762-ab24-25b30b1b4465","Type":"ContainerStarted","Data":"164fc77806ea41fa4944b007a71e697714c24433eb93d8a9ee2dcf6638620d08"}
Sep 09 13:26:46.463073 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:46.461671    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-q6cft" podStartSLOduration=9.677261642 podStartE2EDuration="39.461646796s" totalImagesPullingTime="29.784385154s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 13:26:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:26:46.461140791 +0000 UTC m=+39.820990616" watchObservedRunningTime="2026-09-09 13:26:46.461646796 +0000 UTC m=+39.821496621"
Sep 09 13:26:46.543981 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:46.543869    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q5njn_ab687c9c-b837-4387-a696-7f24afd7711f/konnectivity-agent/0.log" limit=1024
Sep 09 13:26:46.779595 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:46.779475    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q5njn_ab687c9c-b837-4387-a696-7f24afd7711f/konnectivity-agent/0.log" limit=1024
Sep 09 13:26:46.988284 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:46.988178    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q5njn_ab687c9c-b837-4387-a696-7f24afd7711f/konnectivity-agent/0.log" limit=1024
Sep 09 13:26:48.923149 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:48.923108    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2cd7847c-15f0-4e5a-aef5-69342b3c6499-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hsrwj\" (UID: \"2cd7847c-15f0-4e5a-aef5-69342b3c6499\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hsrwj"
Sep 09 13:26:48.923149 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:48.923155    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbe27bd0-1d07-43fe-8bdc-11439590bbab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wdm7n\" (UID: \"bbe27bd0-1d07-43fe-8bdc-11439590bbab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wdm7n"
Sep 09 13:26:48.923672 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:48.923266    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 13:26:48.923672 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:48.923290    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 13:26:48.923672 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:48.923324    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbe27bd0-1d07-43fe-8bdc-11439590bbab-cluster-monitoring-operator-tls podName:bbe27bd0-1d07-43fe-8bdc-11439590bbab nodeName:}" failed. No retries permitted until 2026-09-09 13:26:56.92330903 +0000 UTC m=+50.283158852 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bbe27bd0-1d07-43fe-8bdc-11439590bbab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-wdm7n" (UID: "bbe27bd0-1d07-43fe-8bdc-11439590bbab") : secret "cluster-monitoring-operator-tls" not found
Sep 09 13:26:48.923672 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:48.923360    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cd7847c-15f0-4e5a-aef5-69342b3c6499-samples-operator-tls podName:2cd7847c-15f0-4e5a-aef5-69342b3c6499 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:56.92334292 +0000 UTC m=+50.283192733 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2cd7847c-15f0-4e5a-aef5-69342b3c6499-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-hsrwj" (UID: "2cd7847c-15f0-4e5a-aef5-69342b3c6499") : secret "samples-operator-tls" not found
Sep 09 13:26:49.024776 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:49.024731    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6210e253-1f3a-480f-84de-2cc61e20c535-service-ca-bundle\") pod \"router-default-5fb8d9b79d-nxqx6\" (UID: \"6210e253-1f3a-480f-84de-2cc61e20c535\") " pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:26:49.024964 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:49.024895    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6210e253-1f3a-480f-84de-2cc61e20c535-metrics-certs\") pod \"router-default-5fb8d9b79d-nxqx6\" (UID: \"6210e253-1f3a-480f-84de-2cc61e20c535\") " pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:26:49.024964 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:49.024952    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6210e253-1f3a-480f-84de-2cc61e20c535-service-ca-bundle podName:6210e253-1f3a-480f-84de-2cc61e20c535 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:57.024932421 +0000 UTC m=+50.384782239 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6210e253-1f3a-480f-84de-2cc61e20c535-service-ca-bundle") pod "router-default-5fb8d9b79d-nxqx6" (UID: "6210e253-1f3a-480f-84de-2cc61e20c535") : configmap references non-existent config key: service-ca.crt
Sep 09 13:26:49.025094 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:49.025016    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 13:26:49.025094 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:49.025071    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6210e253-1f3a-480f-84de-2cc61e20c535-metrics-certs podName:6210e253-1f3a-480f-84de-2cc61e20c535 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:57.025056122 +0000 UTC m=+50.384905950 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6210e253-1f3a-480f-84de-2cc61e20c535-metrics-certs") pod "router-default-5fb8d9b79d-nxqx6" (UID: "6210e253-1f3a-480f-84de-2cc61e20c535") : secret "router-metrics-certs-default" not found
Sep 09 13:26:49.025094 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:49.025015    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-tls\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:49.025094 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:49.025081    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 13:26:49.025262 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:49.025098    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65cf7ccc98-jg9dz: secret "image-registry-tls" not found
Sep 09 13:26:49.025262 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:49.025128    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-tls podName:2adb3e0a-ed98-4c24-8f7e-7209c7a4324d nodeName:}" failed. No retries permitted until 2026-09-09 13:26:57.025118667 +0000 UTC m=+50.384968503 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-tls") pod "image-registry-65cf7ccc98-jg9dz" (UID: "2adb3e0a-ed98-4c24-8f7e-7209c7a4324d") : secret "image-registry-tls" not found
Sep 09 13:26:49.125788 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:49.125751    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-shzt9\" (UID: \"6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c\") " pod="openshift-network-console/networking-console-plugin-79977f9844-shzt9"
Sep 09 13:26:49.125984 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:49.125842    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be3c3076-7d51-4348-ba53-1d350b9f69b6-metrics-tls\") pod \"dns-default-4dnj2\" (UID: \"be3c3076-7d51-4348-ba53-1d350b9f69b6\") " pod="openshift-dns/dns-default-4dnj2"
Sep 09 13:26:49.125984 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:49.125918    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 13:26:49.126085 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:49.125998    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c-networking-console-plugin-cert podName:6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c nodeName:}" failed. No retries permitted until 2026-09-09 13:26:57.125975227 +0000 UTC m=+50.485825054 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-shzt9" (UID: "6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c") : secret "networking-console-plugin-cert" not found
Sep 09 13:26:49.126085 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:49.125999    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 13:26:49.126085 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:49.126039    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be3c3076-7d51-4348-ba53-1d350b9f69b6-metrics-tls podName:be3c3076-7d51-4348-ba53-1d350b9f69b6 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:57.126029689 +0000 UTC m=+50.485879508 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/be3c3076-7d51-4348-ba53-1d350b9f69b6-metrics-tls") pod "dns-default-4dnj2" (UID: "be3c3076-7d51-4348-ba53-1d350b9f69b6") : secret "dns-default-metrics-tls" not found
Sep 09 13:26:49.676412 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:49.676369    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q5njn_ab687c9c-b837-4387-a696-7f24afd7711f/konnectivity-agent/0.log" limit=1024
Sep 09 13:26:49.929080 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:49.928972    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q5njn_ab687c9c-b837-4387-a696-7f24afd7711f/konnectivity-agent/0.log" limit=1024
Sep 09 13:26:51.863769 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:51.863723    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-q5njn_ab687c9c-b837-4387-a696-7f24afd7711f/konnectivity-agent/0.log" limit=1024
Sep 09 13:26:57.001920 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:57.001874    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2cd7847c-15f0-4e5a-aef5-69342b3c6499-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hsrwj\" (UID: \"2cd7847c-15f0-4e5a-aef5-69342b3c6499\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hsrwj"
Sep 09 13:26:57.001920 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:57.001922    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbe27bd0-1d07-43fe-8bdc-11439590bbab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wdm7n\" (UID: \"bbe27bd0-1d07-43fe-8bdc-11439590bbab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wdm7n"
Sep 09 13:26:57.002363 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:57.002030    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 13:26:57.002363 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:57.002034    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 13:26:57.002363 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:57.002102    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbe27bd0-1d07-43fe-8bdc-11439590bbab-cluster-monitoring-operator-tls podName:bbe27bd0-1d07-43fe-8bdc-11439590bbab nodeName:}" failed. No retries permitted until 2026-09-09 13:27:13.00208563 +0000 UTC m=+66.361935456 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bbe27bd0-1d07-43fe-8bdc-11439590bbab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-wdm7n" (UID: "bbe27bd0-1d07-43fe-8bdc-11439590bbab") : secret "cluster-monitoring-operator-tls" not found
Sep 09 13:26:57.002363 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:57.002115    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cd7847c-15f0-4e5a-aef5-69342b3c6499-samples-operator-tls podName:2cd7847c-15f0-4e5a-aef5-69342b3c6499 nodeName:}" failed. No retries permitted until 2026-09-09 13:27:13.002109437 +0000 UTC m=+66.361959241 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2cd7847c-15f0-4e5a-aef5-69342b3c6499-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-hsrwj" (UID: "2cd7847c-15f0-4e5a-aef5-69342b3c6499") : secret "samples-operator-tls" not found
Sep 09 13:26:57.102549 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:57.102502    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6210e253-1f3a-480f-84de-2cc61e20c535-service-ca-bundle\") pod \"router-default-5fb8d9b79d-nxqx6\" (UID: \"6210e253-1f3a-480f-84de-2cc61e20c535\") " pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:26:57.102864 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:57.102608    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6210e253-1f3a-480f-84de-2cc61e20c535-metrics-certs\") pod \"router-default-5fb8d9b79d-nxqx6\" (UID: \"6210e253-1f3a-480f-84de-2cc61e20c535\") " pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:26:57.102864 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:57.102666    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-tls\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:26:57.102864 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:57.102724    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6210e253-1f3a-480f-84de-2cc61e20c535-service-ca-bundle podName:6210e253-1f3a-480f-84de-2cc61e20c535 nodeName:}" failed. No retries permitted until 2026-09-09 13:27:13.102690909 +0000 UTC m=+66.462540726 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6210e253-1f3a-480f-84de-2cc61e20c535-service-ca-bundle") pod "router-default-5fb8d9b79d-nxqx6" (UID: "6210e253-1f3a-480f-84de-2cc61e20c535") : configmap references non-existent config key: service-ca.crt
Sep 09 13:26:57.102864 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:57.102752    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 13:26:57.102864 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:57.102775    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 13:26:57.102864 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:57.102789    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65cf7ccc98-jg9dz: secret "image-registry-tls" not found
Sep 09 13:26:57.102864 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:57.102826    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6210e253-1f3a-480f-84de-2cc61e20c535-metrics-certs podName:6210e253-1f3a-480f-84de-2cc61e20c535 nodeName:}" failed. No retries permitted until 2026-09-09 13:27:13.102793818 +0000 UTC m=+66.462643641 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6210e253-1f3a-480f-84de-2cc61e20c535-metrics-certs") pod "router-default-5fb8d9b79d-nxqx6" (UID: "6210e253-1f3a-480f-84de-2cc61e20c535") : secret "router-metrics-certs-default" not found
Sep 09 13:26:57.102864 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:57.102852    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-tls podName:2adb3e0a-ed98-4c24-8f7e-7209c7a4324d nodeName:}" failed. No retries permitted until 2026-09-09 13:27:13.102837098 +0000 UTC m=+66.462686918 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-tls") pod "image-registry-65cf7ccc98-jg9dz" (UID: "2adb3e0a-ed98-4c24-8f7e-7209c7a4324d") : secret "image-registry-tls" not found
Sep 09 13:26:57.203926 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:57.203895    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be3c3076-7d51-4348-ba53-1d350b9f69b6-metrics-tls\") pod \"dns-default-4dnj2\" (UID: \"be3c3076-7d51-4348-ba53-1d350b9f69b6\") " pod="openshift-dns/dns-default-4dnj2"
Sep 09 13:26:57.204077 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:57.204033    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-shzt9\" (UID: \"6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c\") " pod="openshift-network-console/networking-console-plugin-79977f9844-shzt9"
Sep 09 13:26:57.204077 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:57.204058    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 13:26:57.204146 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:57.204122    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 13:26:57.204146 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:57.204127    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be3c3076-7d51-4348-ba53-1d350b9f69b6-metrics-tls podName:be3c3076-7d51-4348-ba53-1d350b9f69b6 nodeName:}" failed. No retries permitted until 2026-09-09 13:27:13.204109593 +0000 UTC m=+66.563959401 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/be3c3076-7d51-4348-ba53-1d350b9f69b6-metrics-tls") pod "dns-default-4dnj2" (UID: "be3c3076-7d51-4348-ba53-1d350b9f69b6") : secret "dns-default-metrics-tls" not found
Sep 09 13:26:57.204223 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:57.204163    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c-networking-console-plugin-cert podName:6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c nodeName:}" failed. No retries permitted until 2026-09-09 13:27:13.204150433 +0000 UTC m=+66.564000237 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-shzt9" (UID: "6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c") : secret "networking-console-plugin-cert" not found
Sep 09 13:26:58.454147 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.454109    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl" event={"ID":"6e9b0e9f-87db-45fd-abd3-bd3956bc9805","Type":"ContainerStarted","Data":"2b481886734a9e4be5a00582563b63beeaf1eaaf23e55ea5aeca62dccb90a4c3"}
Sep 09 13:26:58.454724 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.454700    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl" event={"ID":"6e9b0e9f-87db-45fd-abd3-bd3956bc9805","Type":"ContainerStarted","Data":"fb5c13f940daca399c6c8e667d6b3192aa81abf6b17a6ff1082dc9dc758eace0"}
Sep 09 13:26:58.454782 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.454731    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl" event={"ID":"6e9b0e9f-87db-45fd-abd3-bd3956bc9805","Type":"ContainerStarted","Data":"c501a0d40833230277170bc18e155acb058ca167cc560ed238457dee1600204f"}
Sep 09 13:26:58.456023 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.456000    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ftm96_af902e8b-d693-4341-9979-03cf69548cfb/console-operator/0.log"
Sep 09 13:26:58.456166 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.456039    2566 generic.go:356] "Generic (PLEG): container finished" podID="af902e8b-d693-4341-9979-03cf69548cfb" containerID="c270fc2c1bbd7f557863cdf48f04e145ee22ee899aec81247be9b5fc5d43f4b8" exitCode=255
Sep 09 13:26:58.456166 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.456068    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-ftm96" event={"ID":"af902e8b-d693-4341-9979-03cf69548cfb","Type":"ContainerDied","Data":"c270fc2c1bbd7f557863cdf48f04e145ee22ee899aec81247be9b5fc5d43f4b8"}
Sep 09 13:26:58.456364 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.456346    2566 scope.go:117] "RemoveContainer" containerID="c270fc2c1bbd7f557863cdf48f04e145ee22ee899aec81247be9b5fc5d43f4b8"
Sep 09 13:26:58.458056 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.458015    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-z7s72" event={"ID":"e2fed7f4-5c70-4dc5-a61d-880e4c65492f","Type":"ContainerStarted","Data":"7b34e7a8103208d9cfb4f7e9e62f8e1e4829f9c0c138e32e388c49a27e4f9731"}
Sep 09 13:26:58.459728 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.459686    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ffff64488-lk689" event={"ID":"edc23e7d-c615-47bf-8614-0ed3cd38c69b","Type":"ContainerStarted","Data":"ae4e7cce763e7933b54b4a3d68dc02eee5b2b6934cb9b56daed70e863f485ca9"}
Sep 09 13:26:58.459933 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.459897    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ffff64488-lk689"
Sep 09 13:26:58.461584 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.461565    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ffff64488-lk689"
Sep 09 13:26:58.462153 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.462129    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d4cfdb4-2wxv2" event={"ID":"e858dafd-7a28-4576-9897-802ab13a56c4","Type":"ContainerStarted","Data":"25bbc406cef5d00ee3ceb20d142b3aebafc9739eac07bc8711644a2843f74e48"}
Sep 09 13:26:58.463934 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.463907    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-sdx78" event={"ID":"717fe8eb-5931-4e1b-abce-d011c8db18dd","Type":"ContainerStarted","Data":"7d202e74be7726ea75fafb56fb96588492d2bfd4dc39af387fa20d5fff765b95"}
Sep 09 13:26:58.465880 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.465855    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-pvnqg" event={"ID":"c838ec52-950d-4e56-a558-657a595ef5ac","Type":"ContainerStarted","Data":"bad30525649c93fd683f6a182c07e1f8724ffd5b0284b1855abbdb6569180538"}
Sep 09 13:26:58.467735 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.467698    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-thhvt" event={"ID":"fee30cf9-8a3a-466d-8460-8ec8b363ff6d","Type":"ContainerStarted","Data":"cb8f5ba3f92667ef2cd99362262674d652584bcbdbefaec42071cd2075210063"}
Sep 09 13:26:58.469534 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.469514    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-5hgw6" event={"ID":"613994fe-2f81-47d4-81f3-1b6a63682f5b","Type":"ContainerStarted","Data":"7d58610baac44de22c7e4350e5a38f8d868c188cab68dff82ef8883f0224d047"}
Sep 09 13:26:58.491643 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.491581    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl" podStartSLOduration=4.265734985 podStartE2EDuration="18.491556345s" totalImagesPullingTime="14.22582136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:26:58.488840972 +0000 UTC m=+51.848690798" watchObservedRunningTime="2026-09-09 13:26:58.491556345 +0000 UTC m=+51.851406168"
Sep 09 13:26:58.528933 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.528865    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-z7s72" podStartSLOduration=36.350005304 podStartE2EDuration="50.528840923s" totalImagesPullingTime="14.178835619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:26:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:26:58.527986919 +0000 UTC m=+51.887836746" watchObservedRunningTime="2026-09-09 13:26:58.528840923 +0000 UTC m=+51.888690753"
Sep 09 13:26:58.705173 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.705049    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6ffff64488-lk689" podStartSLOduration=4.462466384 podStartE2EDuration="18.705030071s" totalImagesPullingTime="14.242563687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:26:58.704784117 +0000 UTC m=+52.064633958" watchObservedRunningTime="2026-09-09 13:26:58.705030071 +0000 UTC m=+52.064879930"
Sep 09 13:26:58.750067 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.750003    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-sdx78" podStartSLOduration=36.523105032 podStartE2EDuration="50.749979161s" totalImagesPullingTime="14.226874129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:26:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:26:58.746248534 +0000 UTC m=+52.106098354" watchObservedRunningTime="2026-09-09 13:26:58.749979161 +0000 UTC m=+52.109828991"
Sep 09 13:26:58.800322 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.799849    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-thhvt" podStartSLOduration=3.5022230260000002 podStartE2EDuration="17.799825889s" totalImagesPullingTime="14.297602863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:26:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:26:58.796977902 +0000 UTC m=+52.156827754" watchObservedRunningTime="2026-09-09 13:26:58.799825889 +0000 UTC m=+52.159675709"
Sep 09 13:26:58.835794 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.834477    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sckf8"]
Sep 09 13:26:58.835794 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.835764    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-5hgw6" podStartSLOduration=38.267155424 podStartE2EDuration="50.83574475s" totalImagesPullingTime="12.568589326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:26:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:26:58.831311066 +0000 UTC m=+52.191160895" watchObservedRunningTime="2026-09-09 13:26:58.83574475 +0000 UTC m=+52.195594577"
Sep 09 13:26:58.851458 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.851429    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sckf8"
Sep 09 13:26:58.859373 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.859030    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jg6wc\""
Sep 09 13:26:58.860231 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.859630    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 13:26:58.861214 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.860930    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 13:26:58.873155 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.873096    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-pvnqg" podStartSLOduration=33.643181754 podStartE2EDuration="47.873076855s" totalImagesPullingTime="14.229895101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:26:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:26:58.871006445 +0000 UTC m=+52.230856468" watchObservedRunningTime="2026-09-09 13:26:58.873076855 +0000 UTC m=+52.232926684"
Sep 09 13:26:58.875006 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.874982    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sckf8"]
Sep 09 13:26:58.920777 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.917256    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d4cfdb4-2wxv2" podStartSLOduration=4.689200879 podStartE2EDuration="18.917233713s" totalImagesPullingTime="14.228032834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:26:58.915346716 +0000 UTC m=+52.275196555" watchObservedRunningTime="2026-09-09 13:26:58.917233713 +0000 UTC m=+52.277083537"
Sep 09 13:26:58.921744 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.921720    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/403adf37-3fd7-4d5b-bda9-0b28e7623e08-crio-socket\") pod \"insights-runtime-extractor-sckf8\" (UID: \"403adf37-3fd7-4d5b-bda9-0b28e7623e08\") " pod="openshift-insights/insights-runtime-extractor-sckf8"
Sep 09 13:26:58.921836 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.921759    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76dtw\" (UniqueName: \"kubernetes.io/projected/403adf37-3fd7-4d5b-bda9-0b28e7623e08-kube-api-access-76dtw\") pod \"insights-runtime-extractor-sckf8\" (UID: \"403adf37-3fd7-4d5b-bda9-0b28e7623e08\") " pod="openshift-insights/insights-runtime-extractor-sckf8"
Sep 09 13:26:58.921836 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.921785    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/403adf37-3fd7-4d5b-bda9-0b28e7623e08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sckf8\" (UID: \"403adf37-3fd7-4d5b-bda9-0b28e7623e08\") " pod="openshift-insights/insights-runtime-extractor-sckf8"
Sep 09 13:26:58.921974 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.921949    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/403adf37-3fd7-4d5b-bda9-0b28e7623e08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sckf8\" (UID: \"403adf37-3fd7-4d5b-bda9-0b28e7623e08\") " pod="openshift-insights/insights-runtime-extractor-sckf8"
Sep 09 13:26:58.922006 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:58.921992    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/403adf37-3fd7-4d5b-bda9-0b28e7623e08-data-volume\") pod \"insights-runtime-extractor-sckf8\" (UID: \"403adf37-3fd7-4d5b-bda9-0b28e7623e08\") " pod="openshift-insights/insights-runtime-extractor-sckf8"
Sep 09 13:26:59.023061 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:59.022951    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/403adf37-3fd7-4d5b-bda9-0b28e7623e08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sckf8\" (UID: \"403adf37-3fd7-4d5b-bda9-0b28e7623e08\") " pod="openshift-insights/insights-runtime-extractor-sckf8"
Sep 09 13:26:59.023061 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:59.023006    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/403adf37-3fd7-4d5b-bda9-0b28e7623e08-data-volume\") pod \"insights-runtime-extractor-sckf8\" (UID: \"403adf37-3fd7-4d5b-bda9-0b28e7623e08\") " pod="openshift-insights/insights-runtime-extractor-sckf8"
Sep 09 13:26:59.023296 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:59.023123    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 13:26:59.023296 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:59.023144    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/403adf37-3fd7-4d5b-bda9-0b28e7623e08-crio-socket\") pod \"insights-runtime-extractor-sckf8\" (UID: \"403adf37-3fd7-4d5b-bda9-0b28e7623e08\") " pod="openshift-insights/insights-runtime-extractor-sckf8"
Sep 09 13:26:59.023296 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:59.023178    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76dtw\" (UniqueName: \"kubernetes.io/projected/403adf37-3fd7-4d5b-bda9-0b28e7623e08-kube-api-access-76dtw\") pod \"insights-runtime-extractor-sckf8\" (UID: \"403adf37-3fd7-4d5b-bda9-0b28e7623e08\") " pod="openshift-insights/insights-runtime-extractor-sckf8"
Sep 09 13:26:59.023296 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:59.023202    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/403adf37-3fd7-4d5b-bda9-0b28e7623e08-insights-runtime-extractor-tls podName:403adf37-3fd7-4d5b-bda9-0b28e7623e08 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:59.523180384 +0000 UTC m=+52.883030204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/403adf37-3fd7-4d5b-bda9-0b28e7623e08-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sckf8" (UID: "403adf37-3fd7-4d5b-bda9-0b28e7623e08") : secret "insights-runtime-extractor-tls" not found
Sep 09 13:26:59.023296 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:59.023240    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/403adf37-3fd7-4d5b-bda9-0b28e7623e08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sckf8\" (UID: \"403adf37-3fd7-4d5b-bda9-0b28e7623e08\") " pod="openshift-insights/insights-runtime-extractor-sckf8"
Sep 09 13:26:59.023296 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:59.023273    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/403adf37-3fd7-4d5b-bda9-0b28e7623e08-crio-socket\") pod \"insights-runtime-extractor-sckf8\" (UID: \"403adf37-3fd7-4d5b-bda9-0b28e7623e08\") " pod="openshift-insights/insights-runtime-extractor-sckf8"
Sep 09 13:26:59.023514 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:59.023348    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 13:26:59.023514 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:59.023405    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/403adf37-3fd7-4d5b-bda9-0b28e7623e08-kube-rbac-proxy-cm podName:403adf37-3fd7-4d5b-bda9-0b28e7623e08 nodeName:}" failed. No retries permitted until 2026-09-09 13:26:59.523388781 +0000 UTC m=+52.883238600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/403adf37-3fd7-4d5b-bda9-0b28e7623e08-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sckf8" (UID: "403adf37-3fd7-4d5b-bda9-0b28e7623e08") : configmap "kube-rbac-proxy" not found
Sep 09 13:26:59.023514 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:59.023462    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/403adf37-3fd7-4d5b-bda9-0b28e7623e08-data-volume\") pod \"insights-runtime-extractor-sckf8\" (UID: \"403adf37-3fd7-4d5b-bda9-0b28e7623e08\") " pod="openshift-insights/insights-runtime-extractor-sckf8"
Sep 09 13:26:59.042066 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:59.042029    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76dtw\" (UniqueName: \"kubernetes.io/projected/403adf37-3fd7-4d5b-bda9-0b28e7623e08-kube-api-access-76dtw\") pod \"insights-runtime-extractor-sckf8\" (UID: \"403adf37-3fd7-4d5b-bda9-0b28e7623e08\") " pod="openshift-insights/insights-runtime-extractor-sckf8"
Sep 09 13:26:59.475118 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:59.475088    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ftm96_af902e8b-d693-4341-9979-03cf69548cfb/console-operator/1.log"
Sep 09 13:26:59.475573 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:59.475556    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ftm96_af902e8b-d693-4341-9979-03cf69548cfb/console-operator/0.log"
Sep 09 13:26:59.475631 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:59.475595    2566 generic.go:356] "Generic (PLEG): container finished" podID="af902e8b-d693-4341-9979-03cf69548cfb" containerID="6b1feef95a1561222a6da05cb459969a9cd016166c9af66f3079e8563b518645" exitCode=255
Sep 09 13:26:59.475691 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:59.475676    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-ftm96" event={"ID":"af902e8b-d693-4341-9979-03cf69548cfb","Type":"ContainerDied","Data":"6b1feef95a1561222a6da05cb459969a9cd016166c9af66f3079e8563b518645"}
Sep 09 13:26:59.475751 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:59.475723    2566 scope.go:117] "RemoveContainer" containerID="c270fc2c1bbd7f557863cdf48f04e145ee22ee899aec81247be9b5fc5d43f4b8"
Sep 09 13:26:59.476055 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:59.476031    2566 scope.go:117] "RemoveContainer" containerID="6b1feef95a1561222a6da05cb459969a9cd016166c9af66f3079e8563b518645"
Sep 09 13:26:59.476263 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:59.476237    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-ftm96_openshift-console-operator(af902e8b-d693-4341-9979-03cf69548cfb)\"" pod="openshift-console-operator/console-operator-85b454874b-ftm96" podUID="af902e8b-d693-4341-9979-03cf69548cfb"
Sep 09 13:26:59.528435 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:59.528404    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/403adf37-3fd7-4d5b-bda9-0b28e7623e08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sckf8\" (UID: \"403adf37-3fd7-4d5b-bda9-0b28e7623e08\") " pod="openshift-insights/insights-runtime-extractor-sckf8"
Sep 09 13:26:59.528629 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:59.528514    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 13:26:59.528629 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:59.528604    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/403adf37-3fd7-4d5b-bda9-0b28e7623e08-kube-rbac-proxy-cm podName:403adf37-3fd7-4d5b-bda9-0b28e7623e08 nodeName:}" failed. No retries permitted until 2026-09-09 13:27:00.528584805 +0000 UTC m=+53.888434623 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/403adf37-3fd7-4d5b-bda9-0b28e7623e08-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sckf8" (UID: "403adf37-3fd7-4d5b-bda9-0b28e7623e08") : configmap "kube-rbac-proxy" not found
Sep 09 13:26:59.528977 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:26:59.528945    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/403adf37-3fd7-4d5b-bda9-0b28e7623e08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sckf8\" (UID: \"403adf37-3fd7-4d5b-bda9-0b28e7623e08\") " pod="openshift-insights/insights-runtime-extractor-sckf8"
Sep 09 13:26:59.529112 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:59.529090    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 13:26:59.529180 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:26:59.529164    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/403adf37-3fd7-4d5b-bda9-0b28e7623e08-insights-runtime-extractor-tls podName:403adf37-3fd7-4d5b-bda9-0b28e7623e08 nodeName:}" failed. No retries permitted until 2026-09-09 13:27:00.529149278 +0000 UTC m=+53.888999099 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/403adf37-3fd7-4d5b-bda9-0b28e7623e08-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sckf8" (UID: "403adf37-3fd7-4d5b-bda9-0b28e7623e08") : secret "insights-runtime-extractor-tls" not found
Sep 09 13:27:00.174949 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:00.174923    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sw9qv_f06384ff-80d7-4e3a-a9bd-4925d55d6824/dns-node-resolver/0.log"
Sep 09 13:27:00.481829 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:00.481729    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ftm96_af902e8b-d693-4341-9979-03cf69548cfb/console-operator/1.log"
Sep 09 13:27:00.482193 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:00.482163    2566 scope.go:117] "RemoveContainer" containerID="6b1feef95a1561222a6da05cb459969a9cd016166c9af66f3079e8563b518645"
Sep 09 13:27:00.482374 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:27:00.482355    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-ftm96_openshift-console-operator(af902e8b-d693-4341-9979-03cf69548cfb)\"" pod="openshift-console-operator/console-operator-85b454874b-ftm96" podUID="af902e8b-d693-4341-9979-03cf69548cfb"
Sep 09 13:27:00.539036 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:00.538991    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/403adf37-3fd7-4d5b-bda9-0b28e7623e08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sckf8\" (UID: \"403adf37-3fd7-4d5b-bda9-0b28e7623e08\") " pod="openshift-insights/insights-runtime-extractor-sckf8"
Sep 09 13:27:00.539230 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:00.539117    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/403adf37-3fd7-4d5b-bda9-0b28e7623e08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sckf8\" (UID: \"403adf37-3fd7-4d5b-bda9-0b28e7623e08\") " pod="openshift-insights/insights-runtime-extractor-sckf8"
Sep 09 13:27:00.539230 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:27:00.539127    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 13:27:00.539230 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:27:00.539204    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/403adf37-3fd7-4d5b-bda9-0b28e7623e08-kube-rbac-proxy-cm podName:403adf37-3fd7-4d5b-bda9-0b28e7623e08 nodeName:}" failed. No retries permitted until 2026-09-09 13:27:02.539183465 +0000 UTC m=+55.899033288 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/403adf37-3fd7-4d5b-bda9-0b28e7623e08-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sckf8" (UID: "403adf37-3fd7-4d5b-bda9-0b28e7623e08") : configmap "kube-rbac-proxy" not found
Sep 09 13:27:00.539408 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:27:00.539261    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 13:27:00.539408 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:27:00.539321    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/403adf37-3fd7-4d5b-bda9-0b28e7623e08-insights-runtime-extractor-tls podName:403adf37-3fd7-4d5b-bda9-0b28e7623e08 nodeName:}" failed. No retries permitted until 2026-09-09 13:27:02.539305421 +0000 UTC m=+55.899155239 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/403adf37-3fd7-4d5b-bda9-0b28e7623e08-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sckf8" (UID: "403adf37-3fd7-4d5b-bda9-0b28e7623e08") : secret "insights-runtime-extractor-tls" not found
Sep 09 13:27:01.175083 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:01.175056    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qrq28_e61e15eb-3d0d-4a92-a91a-0987c9fd4eeb/node-ca/0.log"
Sep 09 13:27:01.532199 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:01.532106    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-ftm96"
Sep 09 13:27:01.532199 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:01.532163    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-ftm96"
Sep 09 13:27:01.532586 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:01.532499    2566 scope.go:117] "RemoveContainer" containerID="6b1feef95a1561222a6da05cb459969a9cd016166c9af66f3079e8563b518645"
Sep 09 13:27:01.532667 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:27:01.532648    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-ftm96_openshift-console-operator(af902e8b-d693-4341-9979-03cf69548cfb)\"" pod="openshift-console-operator/console-operator-85b454874b-ftm96" podUID="af902e8b-d693-4341-9979-03cf69548cfb"
Sep 09 13:27:02.009320 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:02.009285    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-thhvt_fee30cf9-8a3a-466d-8460-8ec8b363ff6d/serve-healthcheck-canary/0.log"
Sep 09 13:27:02.560399 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:02.560353    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/403adf37-3fd7-4d5b-bda9-0b28e7623e08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sckf8\" (UID: \"403adf37-3fd7-4d5b-bda9-0b28e7623e08\") " pod="openshift-insights/insights-runtime-extractor-sckf8"
Sep 09 13:27:02.560794 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:27:02.560480    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 13:27:02.560794 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:02.560544    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/403adf37-3fd7-4d5b-bda9-0b28e7623e08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sckf8\" (UID: \"403adf37-3fd7-4d5b-bda9-0b28e7623e08\") " pod="openshift-insights/insights-runtime-extractor-sckf8"
Sep 09 13:27:02.560794 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:27:02.560559    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/403adf37-3fd7-4d5b-bda9-0b28e7623e08-kube-rbac-proxy-cm podName:403adf37-3fd7-4d5b-bda9-0b28e7623e08 nodeName:}" failed. No retries permitted until 2026-09-09 13:27:06.560543105 +0000 UTC m=+59.920392935 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/403adf37-3fd7-4d5b-bda9-0b28e7623e08-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sckf8" (UID: "403adf37-3fd7-4d5b-bda9-0b28e7623e08") : configmap "kube-rbac-proxy" not found
Sep 09 13:27:02.560794 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:27:02.560631    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 13:27:02.560794 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:27:02.560705    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/403adf37-3fd7-4d5b-bda9-0b28e7623e08-insights-runtime-extractor-tls podName:403adf37-3fd7-4d5b-bda9-0b28e7623e08 nodeName:}" failed. No retries permitted until 2026-09-09 13:27:06.560691227 +0000 UTC m=+59.920541044 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/403adf37-3fd7-4d5b-bda9-0b28e7623e08-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sckf8" (UID: "403adf37-3fd7-4d5b-bda9-0b28e7623e08") : secret "insights-runtime-extractor-tls" not found
Sep 09 13:27:06.598111 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:06.598066    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/403adf37-3fd7-4d5b-bda9-0b28e7623e08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sckf8\" (UID: \"403adf37-3fd7-4d5b-bda9-0b28e7623e08\") " pod="openshift-insights/insights-runtime-extractor-sckf8"
Sep 09 13:27:06.598550 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:06.598196    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/403adf37-3fd7-4d5b-bda9-0b28e7623e08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sckf8\" (UID: \"403adf37-3fd7-4d5b-bda9-0b28e7623e08\") " pod="openshift-insights/insights-runtime-extractor-sckf8"
Sep 09 13:27:06.598550 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:27:06.598221    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 13:27:06.598550 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:27:06.598267    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 13:27:06.598550 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:27:06.598298    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/403adf37-3fd7-4d5b-bda9-0b28e7623e08-insights-runtime-extractor-tls podName:403adf37-3fd7-4d5b-bda9-0b28e7623e08 nodeName:}" failed. No retries permitted until 2026-09-09 13:27:14.598282003 +0000 UTC m=+67.958131811 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/403adf37-3fd7-4d5b-bda9-0b28e7623e08-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sckf8" (UID: "403adf37-3fd7-4d5b-bda9-0b28e7623e08") : secret "insights-runtime-extractor-tls" not found
Sep 09 13:27:06.598550 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:27:06.598326    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/403adf37-3fd7-4d5b-bda9-0b28e7623e08-kube-rbac-proxy-cm podName:403adf37-3fd7-4d5b-bda9-0b28e7623e08 nodeName:}" failed. No retries permitted until 2026-09-09 13:27:14.598313895 +0000 UTC m=+67.958163698 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/403adf37-3fd7-4d5b-bda9-0b28e7623e08-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sckf8" (UID: "403adf37-3fd7-4d5b-bda9-0b28e7623e08") : configmap "kube-rbac-proxy" not found
Sep 09 13:27:08.303519 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:08.303487    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m4s7g"
Sep 09 13:27:12.863676 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:12.863638    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ddda0bf-9853-4015-a006-c9e44d47acd5-metrics-certs\") pod \"network-metrics-daemon-q5dxg\" (UID: \"9ddda0bf-9853-4015-a006-c9e44d47acd5\") " pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:27:12.864191 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:12.863755    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ecc5e12-af64-4305-83ed-37552f51b76c-original-pull-secret\") pod \"global-pull-secret-syncer-plsz4\" (UID: \"0ecc5e12-af64-4305-83ed-37552f51b76c\") " pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:27:12.868272 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:12.868242    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 13:27:12.868540 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:12.868520    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 13:27:12.874626 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:27:12.874598    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 13:27:12.874708 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:27:12.874695    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ddda0bf-9853-4015-a006-c9e44d47acd5-metrics-certs podName:9ddda0bf-9853-4015-a006-c9e44d47acd5 nodeName:}" failed. No retries permitted until 2026-09-09 13:28:16.874675153 +0000 UTC m=+130.234524974 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ddda0bf-9853-4015-a006-c9e44d47acd5-metrics-certs") pod "network-metrics-daemon-q5dxg" (UID: "9ddda0bf-9853-4015-a006-c9e44d47acd5") : secret "metrics-daemon-secret" not found
Sep 09 13:27:12.876760 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:12.876735    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ecc5e12-af64-4305-83ed-37552f51b76c-original-pull-secret\") pod \"global-pull-secret-syncer-plsz4\" (UID: \"0ecc5e12-af64-4305-83ed-37552f51b76c\") " pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:27:13.063649 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.063609    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plsz4"
Sep 09 13:27:13.064955 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.064925    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-552v9\" (UniqueName: \"kubernetes.io/projected/49845a79-9f07-4b8c-9f9d-ad2a1035d184-kube-api-access-552v9\") pod \"network-check-target-zbqpk\" (UID: \"49845a79-9f07-4b8c-9f9d-ad2a1035d184\") " pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:27:13.065084 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.065010    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2cd7847c-15f0-4e5a-aef5-69342b3c6499-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hsrwj\" (UID: \"2cd7847c-15f0-4e5a-aef5-69342b3c6499\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hsrwj"
Sep 09 13:27:13.065084 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.065032    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbe27bd0-1d07-43fe-8bdc-11439590bbab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wdm7n\" (UID: \"bbe27bd0-1d07-43fe-8bdc-11439590bbab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wdm7n"
Sep 09 13:27:13.065155 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:27:13.065136    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 13:27:13.065214 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:27:13.065205    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbe27bd0-1d07-43fe-8bdc-11439590bbab-cluster-monitoring-operator-tls podName:bbe27bd0-1d07-43fe-8bdc-11439590bbab nodeName:}" failed. No retries permitted until 2026-09-09 13:27:45.065191997 +0000 UTC m=+98.425041814 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bbe27bd0-1d07-43fe-8bdc-11439590bbab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-wdm7n" (UID: "bbe27bd0-1d07-43fe-8bdc-11439590bbab") : secret "cluster-monitoring-operator-tls" not found
Sep 09 13:27:13.067336 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.067310    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2cd7847c-15f0-4e5a-aef5-69342b3c6499-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-hsrwj\" (UID: \"2cd7847c-15f0-4e5a-aef5-69342b3c6499\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hsrwj"
Sep 09 13:27:13.067463 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.067449    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-552v9\" (UniqueName: \"kubernetes.io/projected/49845a79-9f07-4b8c-9f9d-ad2a1035d184-kube-api-access-552v9\") pod \"network-check-target-zbqpk\" (UID: \"49845a79-9f07-4b8c-9f9d-ad2a1035d184\") " pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:27:13.071364 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.071340    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ps8jz\""
Sep 09 13:27:13.080394 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.080356    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:27:13.166552 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.166517    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6210e253-1f3a-480f-84de-2cc61e20c535-service-ca-bundle\") pod \"router-default-5fb8d9b79d-nxqx6\" (UID: \"6210e253-1f3a-480f-84de-2cc61e20c535\") " pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:27:13.166719 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.166618    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6210e253-1f3a-480f-84de-2cc61e20c535-metrics-certs\") pod \"router-default-5fb8d9b79d-nxqx6\" (UID: \"6210e253-1f3a-480f-84de-2cc61e20c535\") " pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:27:13.166719 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:27:13.166701    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6210e253-1f3a-480f-84de-2cc61e20c535-service-ca-bundle podName:6210e253-1f3a-480f-84de-2cc61e20c535 nodeName:}" failed. No retries permitted until 2026-09-09 13:27:45.166678796 +0000 UTC m=+98.526528620 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6210e253-1f3a-480f-84de-2cc61e20c535-service-ca-bundle") pod "router-default-5fb8d9b79d-nxqx6" (UID: "6210e253-1f3a-480f-84de-2cc61e20c535") : configmap references non-existent config key: service-ca.crt
Sep 09 13:27:13.166865 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.166738    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-tls\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:27:13.169959 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.169902    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-tls\") pod \"image-registry-65cf7ccc98-jg9dz\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") " pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:27:13.171077 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.171024    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6210e253-1f3a-480f-84de-2cc61e20c535-metrics-certs\") pod \"router-default-5fb8d9b79d-nxqx6\" (UID: \"6210e253-1f3a-480f-84de-2cc61e20c535\") " pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:27:13.226734 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.226695    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-q2hz7\""
Sep 09 13:27:13.234957 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.234922    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hsrwj"
Sep 09 13:27:13.267244 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.267208    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be3c3076-7d51-4348-ba53-1d350b9f69b6-metrics-tls\") pod \"dns-default-4dnj2\" (UID: \"be3c3076-7d51-4348-ba53-1d350b9f69b6\") " pod="openshift-dns/dns-default-4dnj2"
Sep 09 13:27:13.267418 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.267386    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-shzt9\" (UID: \"6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c\") " pod="openshift-network-console/networking-console-plugin-79977f9844-shzt9"
Sep 09 13:27:13.267606 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:27:13.267509    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 13:27:13.267606 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:27:13.267578    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c-networking-console-plugin-cert podName:6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c nodeName:}" failed. No retries permitted until 2026-09-09 13:27:45.267558936 +0000 UTC m=+98.627408753 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-shzt9" (UID: "6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c") : secret "networking-console-plugin-cert" not found
Sep 09 13:27:13.269735 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.269709    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be3c3076-7d51-4348-ba53-1d350b9f69b6-metrics-tls\") pod \"dns-default-4dnj2\" (UID: \"be3c3076-7d51-4348-ba53-1d350b9f69b6\") " pod="openshift-dns/dns-default-4dnj2"
Sep 09 13:27:13.367900 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.367866    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4wzf7\""
Sep 09 13:27:13.372682 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.372605    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-plsz4"]
Sep 09 13:27:13.375177 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.375156    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:27:13.375476 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:27:13.375455    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ecc5e12_af64_4305_83ed_37552f51b76c.slice/crio-28ff7ca60dcabaf2b2d13dc10fe1a89a8e1c63b18718e9e9eada2e208b9554a1 WatchSource:0}: Error finding container 28ff7ca60dcabaf2b2d13dc10fe1a89a8e1c63b18718e9e9eada2e208b9554a1: Status 404 returned error can't find the container with id 28ff7ca60dcabaf2b2d13dc10fe1a89a8e1c63b18718e9e9eada2e208b9554a1
Sep 09 13:27:13.397963 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.397931    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zbqpk"]
Sep 09 13:27:13.401610 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:27:13.401578    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49845a79_9f07_4b8c_9f9d_ad2a1035d184.slice/crio-1fa50be2ff9988fef68992d23bf9dfd3c8cd5e485091bbf5fe64206253d9a0e8 WatchSource:0}: Error finding container 1fa50be2ff9988fef68992d23bf9dfd3c8cd5e485091bbf5fe64206253d9a0e8: Status 404 returned error can't find the container with id 1fa50be2ff9988fef68992d23bf9dfd3c8cd5e485091bbf5fe64206253d9a0e8
Sep 09 13:27:13.498996 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.498972    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ckz5q\""
Sep 09 13:27:13.507005 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.506975    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4dnj2"
Sep 09 13:27:13.523375 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.523338    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zbqpk" event={"ID":"49845a79-9f07-4b8c-9f9d-ad2a1035d184","Type":"ContainerStarted","Data":"935b14ee63eb9b80c03b1acd7892c224b295f2c30ffe10d31e8d9f785d29cdd4"}
Sep 09 13:27:13.523509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.523384    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zbqpk" event={"ID":"49845a79-9f07-4b8c-9f9d-ad2a1035d184","Type":"ContainerStarted","Data":"1fa50be2ff9988fef68992d23bf9dfd3c8cd5e485091bbf5fe64206253d9a0e8"}
Sep 09 13:27:13.523509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.523436    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:27:13.524785 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.524758    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-plsz4" event={"ID":"0ecc5e12-af64-4305-83ed-37552f51b76c","Type":"ContainerStarted","Data":"28ff7ca60dcabaf2b2d13dc10fe1a89a8e1c63b18718e9e9eada2e208b9554a1"}
Sep 09 13:27:13.536579 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.536558    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hsrwj"]
Sep 09 13:27:13.545555 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.545503    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zbqpk" podStartSLOduration=66.545486342 podStartE2EDuration="1m6.545486342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:27:13.543995455 +0000 UTC m=+66.903845281" watchObservedRunningTime="2026-09-09 13:27:13.545486342 +0000 UTC m=+66.905336169"
Sep 09 13:27:13.677145 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.677070    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65cf7ccc98-jg9dz"]
Sep 09 13:27:13.680531 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:27:13.680509    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2adb3e0a_ed98_4c24_8f7e_7209c7a4324d.slice/crio-0fbd28a57e59ed98b9a4a898abfaf941e20a1489ea8529c896c63406aa6ab364 WatchSource:0}: Error finding container 0fbd28a57e59ed98b9a4a898abfaf941e20a1489ea8529c896c63406aa6ab364: Status 404 returned error can't find the container with id 0fbd28a57e59ed98b9a4a898abfaf941e20a1489ea8529c896c63406aa6ab364
Sep 09 13:27:13.807304 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:27:13.807278    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe3c3076_7d51_4348_ba53_1d350b9f69b6.slice/crio-a930c5648e29c5c2b98e81048d6296f73915d68708a680962b17e28f16174d69 WatchSource:0}: Error finding container a930c5648e29c5c2b98e81048d6296f73915d68708a680962b17e28f16174d69: Status 404 returned error can't find the container with id a930c5648e29c5c2b98e81048d6296f73915d68708a680962b17e28f16174d69
Sep 09 13:27:13.809352 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:13.809228    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4dnj2"]
Sep 09 13:27:14.529519 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:14.529482    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4dnj2" event={"ID":"be3c3076-7d51-4348-ba53-1d350b9f69b6","Type":"ContainerStarted","Data":"a930c5648e29c5c2b98e81048d6296f73915d68708a680962b17e28f16174d69"}
Sep 09 13:27:14.530741 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:14.530721    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hsrwj" event={"ID":"2cd7847c-15f0-4e5a-aef5-69342b3c6499","Type":"ContainerStarted","Data":"240b22d9155e8915131b710ed369d15100334c3da206aedfc30a3012307a2203"}
Sep 09 13:27:14.532269 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:14.532247    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz" event={"ID":"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d","Type":"ContainerStarted","Data":"2fabfc1c78d34e729ea54d80fbe22542616caaee81089352af8aad7c60c43016"}
Sep 09 13:27:14.532350 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:14.532285    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz" event={"ID":"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d","Type":"ContainerStarted","Data":"0fbd28a57e59ed98b9a4a898abfaf941e20a1489ea8529c896c63406aa6ab364"}
Sep 09 13:27:14.532433 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:14.532393    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:27:14.560187 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:14.560131    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz" podStartSLOduration=67.56011245 podStartE2EDuration="1m7.56011245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:27:14.558377076 +0000 UTC m=+67.918226901" watchObservedRunningTime="2026-09-09 13:27:14.56011245 +0000 UTC m=+67.919962275"
Sep 09 13:27:14.680403 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:14.680363    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/403adf37-3fd7-4d5b-bda9-0b28e7623e08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sckf8\" (UID: \"403adf37-3fd7-4d5b-bda9-0b28e7623e08\") " pod="openshift-insights/insights-runtime-extractor-sckf8"
Sep 09 13:27:14.680566 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:27:14.680516    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 13:27:14.680566 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:14.680529    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/403adf37-3fd7-4d5b-bda9-0b28e7623e08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sckf8\" (UID: \"403adf37-3fd7-4d5b-bda9-0b28e7623e08\") " pod="openshift-insights/insights-runtime-extractor-sckf8"
Sep 09 13:27:14.681034 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:27:14.680982    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/403adf37-3fd7-4d5b-bda9-0b28e7623e08-insights-runtime-extractor-tls podName:403adf37-3fd7-4d5b-bda9-0b28e7623e08 nodeName:}" failed. No retries permitted until 2026-09-09 13:27:30.680563709 +0000 UTC m=+84.040413530 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/403adf37-3fd7-4d5b-bda9-0b28e7623e08-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sckf8" (UID: "403adf37-3fd7-4d5b-bda9-0b28e7623e08") : secret "insights-runtime-extractor-tls" not found
Sep 09 13:27:14.681201 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:14.681176    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/403adf37-3fd7-4d5b-bda9-0b28e7623e08-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sckf8\" (UID: \"403adf37-3fd7-4d5b-bda9-0b28e7623e08\") " pod="openshift-insights/insights-runtime-extractor-sckf8"
Sep 09 13:27:15.144615 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:15.144571    2566 scope.go:117] "RemoveContainer" containerID="6b1feef95a1561222a6da05cb459969a9cd016166c9af66f3079e8563b518645"
Sep 09 13:27:15.538308 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:15.538223    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ftm96_af902e8b-d693-4341-9979-03cf69548cfb/console-operator/2.log"
Sep 09 13:27:15.538774 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:15.538757    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ftm96_af902e8b-d693-4341-9979-03cf69548cfb/console-operator/1.log"
Sep 09 13:27:15.538854 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:15.538797    2566 generic.go:356] "Generic (PLEG): container finished" podID="af902e8b-d693-4341-9979-03cf69548cfb" containerID="83c91244c5bb3c32556ce830c01d204f11b3a861b6661eea96ef41965557777f" exitCode=255
Sep 09 13:27:15.538928 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:15.538853    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-ftm96" event={"ID":"af902e8b-d693-4341-9979-03cf69548cfb","Type":"ContainerDied","Data":"83c91244c5bb3c32556ce830c01d204f11b3a861b6661eea96ef41965557777f"}
Sep 09 13:27:15.538928 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:15.538917    2566 scope.go:117] "RemoveContainer" containerID="6b1feef95a1561222a6da05cb459969a9cd016166c9af66f3079e8563b518645"
Sep 09 13:27:15.539428 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:15.539404    2566 scope.go:117] "RemoveContainer" containerID="83c91244c5bb3c32556ce830c01d204f11b3a861b6661eea96ef41965557777f"
Sep 09 13:27:15.539697 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:27:15.539574    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-ftm96_openshift-console-operator(af902e8b-d693-4341-9979-03cf69548cfb)\"" pod="openshift-console-operator/console-operator-85b454874b-ftm96" podUID="af902e8b-d693-4341-9979-03cf69548cfb"
Sep 09 13:27:16.545102 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:16.545076    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ftm96_af902e8b-d693-4341-9979-03cf69548cfb/console-operator/2.log"
Sep 09 13:27:16.547367 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:16.547339    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hsrwj" event={"ID":"2cd7847c-15f0-4e5a-aef5-69342b3c6499","Type":"ContainerStarted","Data":"e8a3aea545988cc0e8d80f537611d97eaa8550d510c4119be413faf04dd15e9b"}
Sep 09 13:27:16.547526 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:16.547394    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hsrwj" event={"ID":"2cd7847c-15f0-4e5a-aef5-69342b3c6499","Type":"ContainerStarted","Data":"1d6a45e86b4aad6e92b74e61e2de30e499d976cf003be8f80b97a609f324d4e8"}
Sep 09 13:27:16.569900 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:16.569849    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-hsrwj" podStartSLOduration=66.31835759 podStartE2EDuration="1m8.569829691s" totalImagesPullingTime="2.251472101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:26:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:27:16.568050903 +0000 UTC m=+69.927900730" watchObservedRunningTime="2026-09-09 13:27:16.569829691 +0000 UTC m=+69.929679509"
Sep 09 13:27:17.561295 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:17.561241    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4dnj2" event={"ID":"be3c3076-7d51-4348-ba53-1d350b9f69b6","Type":"ContainerStarted","Data":"0620c99ffa257e6a54afe02d01a0c0a3ff49621cb5d4744d7e783751bf00a3f0"}
Sep 09 13:27:18.567407 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:18.567366    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4dnj2" event={"ID":"be3c3076-7d51-4348-ba53-1d350b9f69b6","Type":"ContainerStarted","Data":"ec38e25f06f2c5256c81e5c427fc8bb3abccb393f8d670d86ac184676da6ca7c"}
Sep 09 13:27:18.567993 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:18.567524    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4dnj2"
Sep 09 13:27:18.587227 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:18.587165    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4dnj2" podStartSLOduration=33.975537426 podStartE2EDuration="37.587139245s" totalImagesPullingTime="3.611601819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:26:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:27:18.585521537 +0000 UTC m=+71.945371377" watchObservedRunningTime="2026-09-09 13:27:18.587139245 +0000 UTC m=+71.946989072"
Sep 09 13:27:19.572270 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:19.572177    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-plsz4" event={"ID":"0ecc5e12-af64-4305-83ed-37552f51b76c","Type":"ContainerStarted","Data":"808c45ff9b171f5dbdf1dead221745b334e7f7d0206c6db5842d24397dddafb8"}
Sep 09 13:27:19.589751 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:19.589700    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-plsz4" podStartSLOduration=66.772622639 podStartE2EDuration="1m12.589682574s" totalImagesPullingTime="5.817059935s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 13:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:27:19.588389896 +0000 UTC m=+72.948239732" watchObservedRunningTime="2026-09-09 13:27:19.589682574 +0000 UTC m=+72.949532429"
Sep 09 13:27:21.532575 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:21.532537    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-ftm96"
Sep 09 13:27:21.532996 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:21.532601    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-ftm96"
Sep 09 13:27:21.532996 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:21.532957    2566 scope.go:117] "RemoveContainer" containerID="83c91244c5bb3c32556ce830c01d204f11b3a861b6661eea96ef41965557777f"
Sep 09 13:27:21.533126 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:27:21.533106    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-ftm96_openshift-console-operator(af902e8b-d693-4341-9979-03cf69548cfb)\"" pod="openshift-console-operator/console-operator-85b454874b-ftm96" podUID="af902e8b-d693-4341-9979-03cf69548cfb"
Sep 09 13:27:28.574709 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:28.574675    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4dnj2"
Sep 09 13:27:30.726722 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:30.726585    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/403adf37-3fd7-4d5b-bda9-0b28e7623e08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sckf8\" (UID: \"403adf37-3fd7-4d5b-bda9-0b28e7623e08\") " pod="openshift-insights/insights-runtime-extractor-sckf8"
Sep 09 13:27:30.729282 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:30.729253    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/403adf37-3fd7-4d5b-bda9-0b28e7623e08-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sckf8\" (UID: \"403adf37-3fd7-4d5b-bda9-0b28e7623e08\") " pod="openshift-insights/insights-runtime-extractor-sckf8"
Sep 09 13:27:30.966977 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:30.966941    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jg6wc\""
Sep 09 13:27:30.974760 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:30.974729    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sckf8"
Sep 09 13:27:31.274368 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:31.274285    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sckf8"]
Sep 09 13:27:31.278748 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:27:31.278717    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403adf37_3fd7_4d5b_bda9_0b28e7623e08.slice/crio-68d1d0a86b87ad2c47bf57c5393a6e88ef49ba5d8b036080d78259bcdf0d5aec WatchSource:0}: Error finding container 68d1d0a86b87ad2c47bf57c5393a6e88ef49ba5d8b036080d78259bcdf0d5aec: Status 404 returned error can't find the container with id 68d1d0a86b87ad2c47bf57c5393a6e88ef49ba5d8b036080d78259bcdf0d5aec
Sep 09 13:27:31.614650 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:31.614612    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sckf8" event={"ID":"403adf37-3fd7-4d5b-bda9-0b28e7623e08","Type":"ContainerStarted","Data":"87055b2ca3ccb8f367fb5646d85c3752c08abbe1e4ec2d83a5334914a19122b5"}
Sep 09 13:27:31.614800 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:31.614669    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sckf8" event={"ID":"403adf37-3fd7-4d5b-bda9-0b28e7623e08","Type":"ContainerStarted","Data":"68d1d0a86b87ad2c47bf57c5393a6e88ef49ba5d8b036080d78259bcdf0d5aec"}
Sep 09 13:27:32.620595 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:32.620558    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sckf8" event={"ID":"403adf37-3fd7-4d5b-bda9-0b28e7623e08","Type":"ContainerStarted","Data":"a93bcd1fd7a52b5319783853d5f0928326c10dbda0873a2ce73a08bbe3d55303"}
Sep 09 13:27:33.380144 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:33.380108    2566 patch_prober.go:28] interesting pod/image-registry-65cf7ccc98-jg9dz container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 13:27:33.380327 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:33.380167    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz" podUID="2adb3e0a-ed98-4c24-8f7e-7209c7a4324d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 13:27:34.630749 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:34.630711    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sckf8" event={"ID":"403adf37-3fd7-4d5b-bda9-0b28e7623e08","Type":"ContainerStarted","Data":"b1779bd6c7d9589f9375ec87dc4ed3facfe44dcec086a165bfe20320398209aa"}
Sep 09 13:27:34.651367 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:34.651309    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sckf8" podStartSLOduration=34.466941457 podStartE2EDuration="36.651290991s" totalImagesPullingTime="2.184349534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 13:26:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:27:34.649399423 +0000 UTC m=+88.009249247" watchObservedRunningTime="2026-09-09 13:27:34.651290991 +0000 UTC m=+88.011140816"
Sep 09 13:27:35.544152 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:35.544105    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:27:37.145678 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:37.145641    2566 scope.go:117] "RemoveContainer" containerID="83c91244c5bb3c32556ce830c01d204f11b3a861b6661eea96ef41965557777f"
Sep 09 13:27:37.643734 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:37.643703    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ftm96_af902e8b-d693-4341-9979-03cf69548cfb/console-operator/2.log"
Sep 09 13:27:37.643931 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:37.643786    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-ftm96" event={"ID":"af902e8b-d693-4341-9979-03cf69548cfb","Type":"ContainerStarted","Data":"9fa6e619f82d0352c9440e34af805bf4f3af6a434c8bf47657d2ddb6360e6460"}
Sep 09 13:27:37.644111 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:37.644084    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-ftm96"
Sep 09 13:27:37.664317 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:37.664259    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-ftm96" podStartSLOduration=75.438392578 podStartE2EDuration="1m29.664240738s" totalImagesPullingTime="14.22584816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:26:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:27:37.663412709 +0000 UTC m=+91.023262536" watchObservedRunningTime="2026-09-09 13:27:37.664240738 +0000 UTC m=+91.024090564"
Sep 09 13:27:38.071028 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:38.070944    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-ftm96"
Sep 09 13:27:44.534842 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:44.534759    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zbqpk"
Sep 09 13:27:45.139473 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:45.139426    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbe27bd0-1d07-43fe-8bdc-11439590bbab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wdm7n\" (UID: \"bbe27bd0-1d07-43fe-8bdc-11439590bbab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wdm7n"
Sep 09 13:27:45.142032 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:45.142000    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbe27bd0-1d07-43fe-8bdc-11439590bbab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wdm7n\" (UID: \"bbe27bd0-1d07-43fe-8bdc-11439590bbab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wdm7n"
Sep 09 13:27:45.240921 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:45.240877    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6210e253-1f3a-480f-84de-2cc61e20c535-service-ca-bundle\") pod \"router-default-5fb8d9b79d-nxqx6\" (UID: \"6210e253-1f3a-480f-84de-2cc61e20c535\") " pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:27:45.241519 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:45.241497    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6210e253-1f3a-480f-84de-2cc61e20c535-service-ca-bundle\") pod \"router-default-5fb8d9b79d-nxqx6\" (UID: \"6210e253-1f3a-480f-84de-2cc61e20c535\") " pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:27:45.335201 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:45.335165    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-shkt9\""
Sep 09 13:27:45.342301 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:45.342269    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-shzt9\" (UID: \"6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c\") " pod="openshift-network-console/networking-console-plugin-79977f9844-shzt9"
Sep 09 13:27:45.342729 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:45.342713    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wdm7n"
Sep 09 13:27:45.344742 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:45.344713    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-shzt9\" (UID: \"6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c\") " pod="openshift-network-console/networking-console-plugin-79977f9844-shzt9"
Sep 09 13:27:45.412520 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:45.412483    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xmzdt\""
Sep 09 13:27:45.420615 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:45.420573    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:27:45.585083 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:45.585050    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-fxsgl\""
Sep 09 13:27:45.592864 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:45.592835    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-shzt9"
Sep 09 13:27:45.617011 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:45.616982    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wdm7n"]
Sep 09 13:27:45.619743 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:27:45.619713    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe27bd0_1d07_43fe_8bdc_11439590bbab.slice/crio-a0cf820aef28687bdc37cdc91e7a9c5b9da73506f2cf549bc57bde992c92fcc3 WatchSource:0}: Error finding container a0cf820aef28687bdc37cdc91e7a9c5b9da73506f2cf549bc57bde992c92fcc3: Status 404 returned error can't find the container with id a0cf820aef28687bdc37cdc91e7a9c5b9da73506f2cf549bc57bde992c92fcc3
Sep 09 13:27:45.675033 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:45.674993    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wdm7n" event={"ID":"bbe27bd0-1d07-43fe-8bdc-11439590bbab","Type":"ContainerStarted","Data":"a0cf820aef28687bdc37cdc91e7a9c5b9da73506f2cf549bc57bde992c92fcc3"}
Sep 09 13:27:45.702604 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:45.702571    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5fb8d9b79d-nxqx6"]
Sep 09 13:27:45.705336 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:27:45.705307    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6210e253_1f3a_480f_84de_2cc61e20c535.slice/crio-0c36c6ced3abac57c0bd8af4986977a1daf94202dc3618496599fd01abbb0c90 WatchSource:0}: Error finding container 0c36c6ced3abac57c0bd8af4986977a1daf94202dc3618496599fd01abbb0c90: Status 404 returned error can't find the container with id 0c36c6ced3abac57c0bd8af4986977a1daf94202dc3618496599fd01abbb0c90
Sep 09 13:27:45.865848 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:45.865804    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-shzt9"]
Sep 09 13:27:45.869075 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:27:45.869049    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a2933d5_93f7_4cc3_ae7c_30f4c9444b3c.slice/crio-32ce22af98ba1fffe91ed81468ed763b521c99cbbef50b921d185fbf27321d1b WatchSource:0}: Error finding container 32ce22af98ba1fffe91ed81468ed763b521c99cbbef50b921d185fbf27321d1b: Status 404 returned error can't find the container with id 32ce22af98ba1fffe91ed81468ed763b521c99cbbef50b921d185fbf27321d1b
Sep 09 13:27:46.681660 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:46.681614    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6" event={"ID":"6210e253-1f3a-480f-84de-2cc61e20c535","Type":"ContainerStarted","Data":"86440916d184bcaf0869c2dfab8f1d30908a1739fbe0d7a8bc6ee4938d154ec0"}
Sep 09 13:27:46.682107 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:46.681682    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6" event={"ID":"6210e253-1f3a-480f-84de-2cc61e20c535","Type":"ContainerStarted","Data":"0c36c6ced3abac57c0bd8af4986977a1daf94202dc3618496599fd01abbb0c90"}
Sep 09 13:27:46.683102 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:46.683072    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-shzt9" event={"ID":"6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c","Type":"ContainerStarted","Data":"32ce22af98ba1fffe91ed81468ed763b521c99cbbef50b921d185fbf27321d1b"}
Sep 09 13:27:46.704139 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:46.704078    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6" podStartSLOduration=98.704053275 podStartE2EDuration="1m38.704053275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:27:46.702682324 +0000 UTC m=+100.062532176" watchObservedRunningTime="2026-09-09 13:27:46.704053275 +0000 UTC m=+100.063903101"
Sep 09 13:27:47.420857 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:47.420745    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:27:47.425073 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:47.424795    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:27:47.689525 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:47.689433    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-shzt9" event={"ID":"6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c","Type":"ContainerStarted","Data":"8d529ccf3783e2ac6f4de28d754a6da231660af444500602afb81ecd8e482356"}
Sep 09 13:27:47.690015 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:47.689947    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:27:47.691173 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:47.691145    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5fb8d9b79d-nxqx6"
Sep 09 13:27:47.704563 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:47.704501    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-shzt9" podStartSLOduration=65.496713909 podStartE2EDuration="1m6.704481491s" totalImagesPullingTime="1.207767582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:26:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:27:47.703890915 +0000 UTC m=+101.063740742" watchObservedRunningTime="2026-09-09 13:27:47.704481491 +0000 UTC m=+101.064331319"
Sep 09 13:27:48.695313 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:48.695270    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wdm7n" event={"ID":"bbe27bd0-1d07-43fe-8bdc-11439590bbab","Type":"ContainerStarted","Data":"f480918141570debbf1f4db241a9a96285cc3ab1919e8bd9403225a2e8d5fe54"}
Sep 09 13:27:48.712855 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:48.712781    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wdm7n" podStartSLOduration=98.397485755 podStartE2EDuration="1m40.712762001s" totalImagesPullingTime="2.315276246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:26:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:27:48.711992735 +0000 UTC m=+102.071842565" watchObservedRunningTime="2026-09-09 13:27:48.712762001 +0000 UTC m=+102.072611825"
Sep 09 13:27:54.031674 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:27:54.031631    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65cf7ccc98-jg9dz"]
Sep 09 13:28:04.921638 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:04.921601    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bkbmm"]
Sep 09 13:28:04.925357 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:04.925335    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:04.927975 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:04.927951    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 13:28:04.928416 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:04.928397    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 13:28:04.928487 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:04.928397    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 13:28:04.928487 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:04.928408    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 13:28:04.928487 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:04.928470    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lqv6c\""
Sep 09 13:28:05.014163 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.014124    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0b37b18-368c-4506-83a3-f55bece72293-sys\") pod \"node-exporter-bkbmm\" (UID: \"b0b37b18-368c-4506-83a3-f55bece72293\") " pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:05.014346 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.014199    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhl9g\" (UniqueName: \"kubernetes.io/projected/b0b37b18-368c-4506-83a3-f55bece72293-kube-api-access-lhl9g\") pod \"node-exporter-bkbmm\" (UID: \"b0b37b18-368c-4506-83a3-f55bece72293\") " pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:05.014346 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.014244    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b0b37b18-368c-4506-83a3-f55bece72293-node-exporter-accelerators-collector-config\") pod \"node-exporter-bkbmm\" (UID: \"b0b37b18-368c-4506-83a3-f55bece72293\") " pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:05.014346 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.014284    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b0b37b18-368c-4506-83a3-f55bece72293-node-exporter-wtmp\") pod \"node-exporter-bkbmm\" (UID: \"b0b37b18-368c-4506-83a3-f55bece72293\") " pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:05.014346 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.014305    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b0b37b18-368c-4506-83a3-f55bece72293-node-exporter-textfile\") pod \"node-exporter-bkbmm\" (UID: \"b0b37b18-368c-4506-83a3-f55bece72293\") " pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:05.014346 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.014328    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b0b37b18-368c-4506-83a3-f55bece72293-root\") pod \"node-exporter-bkbmm\" (UID: \"b0b37b18-368c-4506-83a3-f55bece72293\") " pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:05.014569 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.014366    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b0b37b18-368c-4506-83a3-f55bece72293-node-exporter-tls\") pod \"node-exporter-bkbmm\" (UID: \"b0b37b18-368c-4506-83a3-f55bece72293\") " pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:05.014569 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.014439    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0b37b18-368c-4506-83a3-f55bece72293-metrics-client-ca\") pod \"node-exporter-bkbmm\" (UID: \"b0b37b18-368c-4506-83a3-f55bece72293\") " pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:05.014569 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.014473    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0b37b18-368c-4506-83a3-f55bece72293-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bkbmm\" (UID: \"b0b37b18-368c-4506-83a3-f55bece72293\") " pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:05.115189 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.115146    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b0b37b18-368c-4506-83a3-f55bece72293-node-exporter-textfile\") pod \"node-exporter-bkbmm\" (UID: \"b0b37b18-368c-4506-83a3-f55bece72293\") " pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:05.115401 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.115202    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b0b37b18-368c-4506-83a3-f55bece72293-root\") pod \"node-exporter-bkbmm\" (UID: \"b0b37b18-368c-4506-83a3-f55bece72293\") " pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:05.115401 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.115226    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b0b37b18-368c-4506-83a3-f55bece72293-node-exporter-tls\") pod \"node-exporter-bkbmm\" (UID: \"b0b37b18-368c-4506-83a3-f55bece72293\") " pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:05.115401 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.115254    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0b37b18-368c-4506-83a3-f55bece72293-metrics-client-ca\") pod \"node-exporter-bkbmm\" (UID: \"b0b37b18-368c-4506-83a3-f55bece72293\") " pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:05.115401 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.115292    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0b37b18-368c-4506-83a3-f55bece72293-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bkbmm\" (UID: \"b0b37b18-368c-4506-83a3-f55bece72293\") " pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:05.115401 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.115344    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b0b37b18-368c-4506-83a3-f55bece72293-root\") pod \"node-exporter-bkbmm\" (UID: \"b0b37b18-368c-4506-83a3-f55bece72293\") " pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:05.115662 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.115416    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0b37b18-368c-4506-83a3-f55bece72293-sys\") pod \"node-exporter-bkbmm\" (UID: \"b0b37b18-368c-4506-83a3-f55bece72293\") " pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:05.115662 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.115501    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhl9g\" (UniqueName: \"kubernetes.io/projected/b0b37b18-368c-4506-83a3-f55bece72293-kube-api-access-lhl9g\") pod \"node-exporter-bkbmm\" (UID: \"b0b37b18-368c-4506-83a3-f55bece72293\") " pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:05.115662 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.115506    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b0b37b18-368c-4506-83a3-f55bece72293-node-exporter-textfile\") pod \"node-exporter-bkbmm\" (UID: \"b0b37b18-368c-4506-83a3-f55bece72293\") " pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:05.115662 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.115544    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b0b37b18-368c-4506-83a3-f55bece72293-node-exporter-accelerators-collector-config\") pod \"node-exporter-bkbmm\" (UID: \"b0b37b18-368c-4506-83a3-f55bece72293\") " pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:05.115662 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.115578    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b0b37b18-368c-4506-83a3-f55bece72293-node-exporter-wtmp\") pod \"node-exporter-bkbmm\" (UID: \"b0b37b18-368c-4506-83a3-f55bece72293\") " pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:05.115662 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.115549    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0b37b18-368c-4506-83a3-f55bece72293-sys\") pod \"node-exporter-bkbmm\" (UID: \"b0b37b18-368c-4506-83a3-f55bece72293\") " pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:05.115964 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.115739    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b0b37b18-368c-4506-83a3-f55bece72293-node-exporter-wtmp\") pod \"node-exporter-bkbmm\" (UID: \"b0b37b18-368c-4506-83a3-f55bece72293\") " pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:05.115964 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.115915    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0b37b18-368c-4506-83a3-f55bece72293-metrics-client-ca\") pod \"node-exporter-bkbmm\" (UID: \"b0b37b18-368c-4506-83a3-f55bece72293\") " pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:05.116124 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.116103    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b0b37b18-368c-4506-83a3-f55bece72293-node-exporter-accelerators-collector-config\") pod \"node-exporter-bkbmm\" (UID: \"b0b37b18-368c-4506-83a3-f55bece72293\") " pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:05.117760 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.117737    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b0b37b18-368c-4506-83a3-f55bece72293-node-exporter-tls\") pod \"node-exporter-bkbmm\" (UID: \"b0b37b18-368c-4506-83a3-f55bece72293\") " pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:05.117885 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.117782    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0b37b18-368c-4506-83a3-f55bece72293-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bkbmm\" (UID: \"b0b37b18-368c-4506-83a3-f55bece72293\") " pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:05.126694 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.126667    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhl9g\" (UniqueName: \"kubernetes.io/projected/b0b37b18-368c-4506-83a3-f55bece72293-kube-api-access-lhl9g\") pod \"node-exporter-bkbmm\" (UID: \"b0b37b18-368c-4506-83a3-f55bece72293\") " pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:05.234615 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.234530    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bkbmm"
Sep 09 13:28:05.245179 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:28:05.245150    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b37b18_368c_4506_83a3_f55bece72293.slice/crio-73784cfb85326258997ce212fdf0b886ffb62f74214831184d4b4c189d1d94a2 WatchSource:0}: Error finding container 73784cfb85326258997ce212fdf0b886ffb62f74214831184d4b4c189d1d94a2: Status 404 returned error can't find the container with id 73784cfb85326258997ce212fdf0b886ffb62f74214831184d4b4c189d1d94a2
Sep 09 13:28:05.757700 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:05.757654    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bkbmm" event={"ID":"b0b37b18-368c-4506-83a3-f55bece72293","Type":"ContainerStarted","Data":"73784cfb85326258997ce212fdf0b886ffb62f74214831184d4b4c189d1d94a2"}
Sep 09 13:28:06.763998 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:06.763896    2566 generic.go:356] "Generic (PLEG): container finished" podID="b0b37b18-368c-4506-83a3-f55bece72293" containerID="115e3fef78659f84ffdc21c590c4d01a05062665ab14beaecaa664caf18cc1c6" exitCode=0
Sep 09 13:28:06.763998 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:06.763988    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bkbmm" event={"ID":"b0b37b18-368c-4506-83a3-f55bece72293","Type":"ContainerDied","Data":"115e3fef78659f84ffdc21c590c4d01a05062665ab14beaecaa664caf18cc1c6"}
Sep 09 13:28:07.769496 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:07.769454    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bkbmm" event={"ID":"b0b37b18-368c-4506-83a3-f55bece72293","Type":"ContainerStarted","Data":"0332bb408654d4c09e6d474e64a79b91f0155387c5e87cbbcaabf194d86c664d"}
Sep 09 13:28:07.769935 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:07.769526    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bkbmm" event={"ID":"b0b37b18-368c-4506-83a3-f55bece72293","Type":"ContainerStarted","Data":"3c288610a7fe81650050b558a37d36d69f10799d800897750a3126b7a283f8fd"}
Sep 09 13:28:07.789608 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:07.789551    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bkbmm" podStartSLOduration=2.502253937 podStartE2EDuration="3.789534797s" totalImagesPullingTime="1.224485914s" totalInitContainerRuntime="62.794946ms" isStatefulPod=true podCreationTimestamp="2026-09-09 13:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:28:07.787889882 +0000 UTC m=+121.147739708" watchObservedRunningTime="2026-09-09 13:28:07.789534797 +0000 UTC m=+121.149384623"
Sep 09 13:28:13.792729 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:13.792642    2566 generic.go:356] "Generic (PLEG): container finished" podID="717fe8eb-5931-4e1b-abce-d011c8db18dd" containerID="7d202e74be7726ea75fafb56fb96588492d2bfd4dc39af387fa20d5fff765b95" exitCode=0
Sep 09 13:28:13.792729 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:13.792713    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-sdx78" event={"ID":"717fe8eb-5931-4e1b-abce-d011c8db18dd","Type":"ContainerDied","Data":"7d202e74be7726ea75fafb56fb96588492d2bfd4dc39af387fa20d5fff765b95"}
Sep 09 13:28:13.793184 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:13.793046    2566 scope.go:117] "RemoveContainer" containerID="7d202e74be7726ea75fafb56fb96588492d2bfd4dc39af387fa20d5fff765b95"
Sep 09 13:28:14.798536 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:14.798503    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-sdx78" event={"ID":"717fe8eb-5931-4e1b-abce-d011c8db18dd","Type":"ContainerStarted","Data":"5211b423e3272b7b80e627f897856939237dcd8a32d3e213aa95fa699d97a0fa"}
Sep 09 13:28:16.923726 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:16.923682    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ddda0bf-9853-4015-a006-c9e44d47acd5-metrics-certs\") pod \"network-metrics-daemon-q5dxg\" (UID: \"9ddda0bf-9853-4015-a006-c9e44d47acd5\") " pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:28:16.926141 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:16.926114    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ddda0bf-9853-4015-a006-c9e44d47acd5-metrics-certs\") pod \"network-metrics-daemon-q5dxg\" (UID: \"9ddda0bf-9853-4015-a006-c9e44d47acd5\") " pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:28:16.960196 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:16.960165    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zpj6j\""
Sep 09 13:28:16.967858 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:16.967838    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q5dxg"
Sep 09 13:28:17.254276 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:17.254189    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q5dxg"]
Sep 09 13:28:17.257011 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:28:17.256983    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ddda0bf_9853_4015_a006_c9e44d47acd5.slice/crio-3f115721cf60848969424c80196dfbb7c118825a349d267a698c574caa810944 WatchSource:0}: Error finding container 3f115721cf60848969424c80196dfbb7c118825a349d267a698c574caa810944: Status 404 returned error can't find the container with id 3f115721cf60848969424c80196dfbb7c118825a349d267a698c574caa810944
Sep 09 13:28:17.810804 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:17.810766    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q5dxg" event={"ID":"9ddda0bf-9853-4015-a006-c9e44d47acd5","Type":"ContainerStarted","Data":"3f115721cf60848969424c80196dfbb7c118825a349d267a698c574caa810944"}
Sep 09 13:28:19.051996 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.051937    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz" podUID="2adb3e0a-ed98-4c24-8f7e-7209c7a4324d" containerName="registry" containerID="cri-o://2fabfc1c78d34e729ea54d80fbe22542616caaee81089352af8aad7c60c43016" gracePeriod=30
Sep 09 13:28:19.298145 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.298121    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:28:19.346454 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.346424    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qv4b\" (UniqueName: \"kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-kube-api-access-7qv4b\") pod \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") "
Sep 09 13:28:19.346646 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.346498    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-certificates\") pod \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") "
Sep 09 13:28:19.346646 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.346547    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-image-registry-private-configuration\") pod \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") "
Sep 09 13:28:19.346796 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.346673    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-bound-sa-token\") pod \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") "
Sep 09 13:28:19.346877 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.346858    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-trusted-ca\") pod \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") "
Sep 09 13:28:19.346931 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.346892    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-tls\") pod \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") "
Sep 09 13:28:19.346931 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.346901    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2adb3e0a-ed98-4c24-8f7e-7209c7a4324d" (UID: "2adb3e0a-ed98-4c24-8f7e-7209c7a4324d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:28:19.347042 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.346929    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-ca-trust-extracted\") pod \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") "
Sep 09 13:28:19.347042 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.346977    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-installation-pull-secrets\") pod \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\" (UID: \"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d\") "
Sep 09 13:28:19.347345 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.347291    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2adb3e0a-ed98-4c24-8f7e-7209c7a4324d" (UID: "2adb3e0a-ed98-4c24-8f7e-7209c7a4324d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:28:19.347479 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.347377    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-trusted-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:28:19.347479 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.347396    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-certificates\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:28:19.349347 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.349322    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2adb3e0a-ed98-4c24-8f7e-7209c7a4324d" (UID: "2adb3e0a-ed98-4c24-8f7e-7209c7a4324d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:28:19.349465 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.349386    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-kube-api-access-7qv4b" (OuterVolumeSpecName: "kube-api-access-7qv4b") pod "2adb3e0a-ed98-4c24-8f7e-7209c7a4324d" (UID: "2adb3e0a-ed98-4c24-8f7e-7209c7a4324d"). InnerVolumeSpecName "kube-api-access-7qv4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:28:19.349465 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.349444    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2adb3e0a-ed98-4c24-8f7e-7209c7a4324d" (UID: "2adb3e0a-ed98-4c24-8f7e-7209c7a4324d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:28:19.349559 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.349535    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2adb3e0a-ed98-4c24-8f7e-7209c7a4324d" (UID: "2adb3e0a-ed98-4c24-8f7e-7209c7a4324d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:28:19.349599 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.349563    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2adb3e0a-ed98-4c24-8f7e-7209c7a4324d" (UID: "2adb3e0a-ed98-4c24-8f7e-7209c7a4324d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:28:19.355325 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.355294    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2adb3e0a-ed98-4c24-8f7e-7209c7a4324d" (UID: "2adb3e0a-ed98-4c24-8f7e-7209c7a4324d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:28:19.448271 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.448234    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qv4b\" (UniqueName: \"kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-kube-api-access-7qv4b\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:28:19.448271 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.448264    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-image-registry-private-configuration\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:28:19.448271 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.448275    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-bound-sa-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:28:19.448481 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.448284    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-registry-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:28:19.448481 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.448292    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-ca-trust-extracted\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:28:19.448481 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.448300    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d-installation-pull-secrets\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:28:19.819672 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.819588    2566 generic.go:356] "Generic (PLEG): container finished" podID="2adb3e0a-ed98-4c24-8f7e-7209c7a4324d" containerID="2fabfc1c78d34e729ea54d80fbe22542616caaee81089352af8aad7c60c43016" exitCode=0
Sep 09 13:28:19.819672 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.819637    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz"
Sep 09 13:28:19.819888 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.819670    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz" event={"ID":"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d","Type":"ContainerDied","Data":"2fabfc1c78d34e729ea54d80fbe22542616caaee81089352af8aad7c60c43016"}
Sep 09 13:28:19.819888 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.819716    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65cf7ccc98-jg9dz" event={"ID":"2adb3e0a-ed98-4c24-8f7e-7209c7a4324d","Type":"ContainerDied","Data":"0fbd28a57e59ed98b9a4a898abfaf941e20a1489ea8529c896c63406aa6ab364"}
Sep 09 13:28:19.819888 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.819738    2566 scope.go:117] "RemoveContainer" containerID="2fabfc1c78d34e729ea54d80fbe22542616caaee81089352af8aad7c60c43016"
Sep 09 13:28:19.821673 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.821653    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q5dxg" event={"ID":"9ddda0bf-9853-4015-a006-c9e44d47acd5","Type":"ContainerStarted","Data":"2c253db672f02d544bea8187a2257e329b7202656911e180ccfde1e1c83e0e44"}
Sep 09 13:28:19.821794 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.821691    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q5dxg" event={"ID":"9ddda0bf-9853-4015-a006-c9e44d47acd5","Type":"ContainerStarted","Data":"46ea4bb12868f35f413fa7d726d22d130f3b268eb462d167ea888297b5ac2f44"}
Sep 09 13:28:19.828475 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.828460    2566 scope.go:117] "RemoveContainer" containerID="2fabfc1c78d34e729ea54d80fbe22542616caaee81089352af8aad7c60c43016"
Sep 09 13:28:19.828736 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:28:19.828717    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fabfc1c78d34e729ea54d80fbe22542616caaee81089352af8aad7c60c43016\": container with ID starting with 2fabfc1c78d34e729ea54d80fbe22542616caaee81089352af8aad7c60c43016 not found: ID does not exist" containerID="2fabfc1c78d34e729ea54d80fbe22542616caaee81089352af8aad7c60c43016"
Sep 09 13:28:19.828793 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.828744    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fabfc1c78d34e729ea54d80fbe22542616caaee81089352af8aad7c60c43016"} err="failed to get container status \"2fabfc1c78d34e729ea54d80fbe22542616caaee81089352af8aad7c60c43016\": rpc error: code = NotFound desc = could not find container \"2fabfc1c78d34e729ea54d80fbe22542616caaee81089352af8aad7c60c43016\": container with ID starting with 2fabfc1c78d34e729ea54d80fbe22542616caaee81089352af8aad7c60c43016 not found: ID does not exist"
Sep 09 13:28:19.839931 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.839887    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-q5dxg" podStartSLOduration=131.286057453 podStartE2EDuration="2m12.839871334s" totalImagesPullingTime="1.553813881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:28:19.838800411 +0000 UTC m=+133.198650238" watchObservedRunningTime="2026-09-09 13:28:19.839871334 +0000 UTC m=+133.199721160"
Sep 09 13:28:19.859760 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.859731    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65cf7ccc98-jg9dz"]
Sep 09 13:28:19.864018 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:19.863990    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-65cf7ccc98-jg9dz"]
Sep 09 13:28:21.149152 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:21.149118    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2adb3e0a-ed98-4c24-8f7e-7209c7a4324d" path="/var/lib/kubelet/pods/2adb3e0a-ed98-4c24-8f7e-7209c7a4324d/volumes"
Sep 09 13:28:21.644838 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:21.644785    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl" podUID="6e9b0e9f-87db-45fd-abd3-bd3956bc9805" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 13:28:28.855313 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:28.855280    2566 generic.go:356] "Generic (PLEG): container finished" podID="613994fe-2f81-47d4-81f3-1b6a63682f5b" containerID="7d58610baac44de22c7e4350e5a38f8d868c188cab68dff82ef8883f0224d047" exitCode=0
Sep 09 13:28:28.855742 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:28.855359    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-5hgw6" event={"ID":"613994fe-2f81-47d4-81f3-1b6a63682f5b","Type":"ContainerDied","Data":"7d58610baac44de22c7e4350e5a38f8d868c188cab68dff82ef8883f0224d047"}
Sep 09 13:28:28.855742 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:28.855719    2566 scope.go:117] "RemoveContainer" containerID="7d58610baac44de22c7e4350e5a38f8d868c188cab68dff82ef8883f0224d047"
Sep 09 13:28:28.857102 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:28.857081    2566 generic.go:356] "Generic (PLEG): container finished" podID="e2fed7f4-5c70-4dc5-a61d-880e4c65492f" containerID="7b34e7a8103208d9cfb4f7e9e62f8e1e4829f9c0c138e32e388c49a27e4f9731" exitCode=0
Sep 09 13:28:28.857160 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:28.857119    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-z7s72" event={"ID":"e2fed7f4-5c70-4dc5-a61d-880e4c65492f","Type":"ContainerDied","Data":"7b34e7a8103208d9cfb4f7e9e62f8e1e4829f9c0c138e32e388c49a27e4f9731"}
Sep 09 13:28:28.857414 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:28.857399    2566 scope.go:117] "RemoveContainer" containerID="7b34e7a8103208d9cfb4f7e9e62f8e1e4829f9c0c138e32e388c49a27e4f9731"
Sep 09 13:28:29.863923 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:29.863882    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-z7s72" event={"ID":"e2fed7f4-5c70-4dc5-a61d-880e4c65492f","Type":"ContainerStarted","Data":"4bbfe02ce5734afcaac589317c704449dca28cf977bb7d4b6822302e7cfc8031"}
Sep 09 13:28:29.865782 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:29.865756    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-5hgw6" event={"ID":"613994fe-2f81-47d4-81f3-1b6a63682f5b","Type":"ContainerStarted","Data":"985a1fd1ae56c9b09b495c27cd7481e02947522b3f7330bdc237267134dbb8e9"}
Sep 09 13:28:31.644644 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:31.644599    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl" podUID="6e9b0e9f-87db-45fd-abd3-bd3956bc9805" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 13:28:41.644480 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:41.644434    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl" podUID="6e9b0e9f-87db-45fd-abd3-bd3956bc9805" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 13:28:41.644920 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:41.644511    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl"
Sep 09 13:28:41.645121 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:41.645101    2566 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"2b481886734a9e4be5a00582563b63beeaf1eaaf23e55ea5aeca62dccb90a4c3"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 09 13:28:41.645159 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:41.645146    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl" podUID="6e9b0e9f-87db-45fd-abd3-bd3956bc9805" containerName="service-proxy" containerID="cri-o://2b481886734a9e4be5a00582563b63beeaf1eaaf23e55ea5aeca62dccb90a4c3" gracePeriod=30
Sep 09 13:28:41.914583 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:41.914489    2566 generic.go:356] "Generic (PLEG): container finished" podID="6e9b0e9f-87db-45fd-abd3-bd3956bc9805" containerID="2b481886734a9e4be5a00582563b63beeaf1eaaf23e55ea5aeca62dccb90a4c3" exitCode=2
Sep 09 13:28:41.914583 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:41.914560    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl" event={"ID":"6e9b0e9f-87db-45fd-abd3-bd3956bc9805","Type":"ContainerDied","Data":"2b481886734a9e4be5a00582563b63beeaf1eaaf23e55ea5aeca62dccb90a4c3"}
Sep 09 13:28:41.914773 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:28:41.914606    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d5f6575bc-cfqzl" event={"ID":"6e9b0e9f-87db-45fd-abd3-bd3956bc9805","Type":"ContainerStarted","Data":"c781241555ec02d9edbac90366fefc275ab5368cc60a2dcf5f7475dc17a77e6f"}
Sep 09 13:31:07.072103 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:07.072065    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ftm96_af902e8b-d693-4341-9979-03cf69548cfb/console-operator/2.log"
Sep 09 13:31:07.080328 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:07.080302    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ftm96_af902e8b-d693-4341-9979-03cf69548cfb/console-operator/2.log"
Sep 09 13:31:07.086432 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:07.086396    2566 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 13:31:50.409769 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:50.409733    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8469c9486d-jqtgr"]
Sep 09 13:31:50.410239 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:50.410068    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2adb3e0a-ed98-4c24-8f7e-7209c7a4324d" containerName="registry"
Sep 09 13:31:50.410239 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:50.410079    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2adb3e0a-ed98-4c24-8f7e-7209c7a4324d" containerName="registry"
Sep 09 13:31:50.410239 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:50.410139    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2adb3e0a-ed98-4c24-8f7e-7209c7a4324d" containerName="registry"
Sep 09 13:31:50.413091 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:50.413070    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8469c9486d-jqtgr"
Sep 09 13:31:50.418654 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:50.418629    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 13:31:50.419120 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:50.419094    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 13:31:50.419273 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:50.419126    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-bk6bx\""
Sep 09 13:31:50.419273 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:50.419104    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 13:31:50.419767 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:50.419562    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 13:31:50.433872 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:50.433830    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8469c9486d-jqtgr"]
Sep 09 13:31:50.436489 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:50.436455    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmjmp\" (UniqueName: \"kubernetes.io/projected/bff62eb2-b9c8-4142-8d12-4b968be9de07-kube-api-access-vmjmp\") pod \"opendatahub-operator-controller-manager-8469c9486d-jqtgr\" (UID: \"bff62eb2-b9c8-4142-8d12-4b968be9de07\") " pod="opendatahub/opendatahub-operator-controller-manager-8469c9486d-jqtgr"
Sep 09 13:31:50.436631 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:50.436506    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bff62eb2-b9c8-4142-8d12-4b968be9de07-manifests\") pod \"opendatahub-operator-controller-manager-8469c9486d-jqtgr\" (UID: \"bff62eb2-b9c8-4142-8d12-4b968be9de07\") " pod="opendatahub/opendatahub-operator-controller-manager-8469c9486d-jqtgr"
Sep 09 13:31:50.436631 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:50.436561    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bff62eb2-b9c8-4142-8d12-4b968be9de07-tmp\") pod \"opendatahub-operator-controller-manager-8469c9486d-jqtgr\" (UID: \"bff62eb2-b9c8-4142-8d12-4b968be9de07\") " pod="opendatahub/opendatahub-operator-controller-manager-8469c9486d-jqtgr"
Sep 09 13:31:50.436631 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:50.436602    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bff62eb2-b9c8-4142-8d12-4b968be9de07-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8469c9486d-jqtgr\" (UID: \"bff62eb2-b9c8-4142-8d12-4b968be9de07\") " pod="opendatahub/opendatahub-operator-controller-manager-8469c9486d-jqtgr"
Sep 09 13:31:50.436761 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:50.436632    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bff62eb2-b9c8-4142-8d12-4b968be9de07-webhook-cert\") pod \"opendatahub-operator-controller-manager-8469c9486d-jqtgr\" (UID: \"bff62eb2-b9c8-4142-8d12-4b968be9de07\") " pod="opendatahub/opendatahub-operator-controller-manager-8469c9486d-jqtgr"
Sep 09 13:31:50.537266 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:50.537151    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmjmp\" (UniqueName: \"kubernetes.io/projected/bff62eb2-b9c8-4142-8d12-4b968be9de07-kube-api-access-vmjmp\") pod \"opendatahub-operator-controller-manager-8469c9486d-jqtgr\" (UID: \"bff62eb2-b9c8-4142-8d12-4b968be9de07\") " pod="opendatahub/opendatahub-operator-controller-manager-8469c9486d-jqtgr"
Sep 09 13:31:50.537473 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:50.537299    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bff62eb2-b9c8-4142-8d12-4b968be9de07-manifests\") pod \"opendatahub-operator-controller-manager-8469c9486d-jqtgr\" (UID: \"bff62eb2-b9c8-4142-8d12-4b968be9de07\") " pod="opendatahub/opendatahub-operator-controller-manager-8469c9486d-jqtgr"
Sep 09 13:31:50.537473 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:50.537334    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bff62eb2-b9c8-4142-8d12-4b968be9de07-tmp\") pod \"opendatahub-operator-controller-manager-8469c9486d-jqtgr\" (UID: \"bff62eb2-b9c8-4142-8d12-4b968be9de07\") " pod="opendatahub/opendatahub-operator-controller-manager-8469c9486d-jqtgr"
Sep 09 13:31:50.537473 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:50.537374    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bff62eb2-b9c8-4142-8d12-4b968be9de07-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8469c9486d-jqtgr\" (UID: \"bff62eb2-b9c8-4142-8d12-4b968be9de07\") " pod="opendatahub/opendatahub-operator-controller-manager-8469c9486d-jqtgr"
Sep 09 13:31:50.537473 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:50.537405    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bff62eb2-b9c8-4142-8d12-4b968be9de07-webhook-cert\") pod \"opendatahub-operator-controller-manager-8469c9486d-jqtgr\" (UID: \"bff62eb2-b9c8-4142-8d12-4b968be9de07\") " pod="opendatahub/opendatahub-operator-controller-manager-8469c9486d-jqtgr"
Sep 09 13:31:50.537693 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:50.537668    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bff62eb2-b9c8-4142-8d12-4b968be9de07-manifests\") pod \"opendatahub-operator-controller-manager-8469c9486d-jqtgr\" (UID: \"bff62eb2-b9c8-4142-8d12-4b968be9de07\") " pod="opendatahub/opendatahub-operator-controller-manager-8469c9486d-jqtgr"
Sep 09 13:31:50.539795 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:50.539766    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bff62eb2-b9c8-4142-8d12-4b968be9de07-tmp\") pod \"opendatahub-operator-controller-manager-8469c9486d-jqtgr\" (UID: \"bff62eb2-b9c8-4142-8d12-4b968be9de07\") " pod="opendatahub/opendatahub-operator-controller-manager-8469c9486d-jqtgr"
Sep 09 13:31:50.539969 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:50.539944    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bff62eb2-b9c8-4142-8d12-4b968be9de07-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8469c9486d-jqtgr\" (UID: \"bff62eb2-b9c8-4142-8d12-4b968be9de07\") " pod="opendatahub/opendatahub-operator-controller-manager-8469c9486d-jqtgr"
Sep 09 13:31:50.540012 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:50.539976    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bff62eb2-b9c8-4142-8d12-4b968be9de07-webhook-cert\") pod \"opendatahub-operator-controller-manager-8469c9486d-jqtgr\" (UID: \"bff62eb2-b9c8-4142-8d12-4b968be9de07\") " pod="opendatahub/opendatahub-operator-controller-manager-8469c9486d-jqtgr"
Sep 09 13:31:50.544901 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:50.544878    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmjmp\" (UniqueName: \"kubernetes.io/projected/bff62eb2-b9c8-4142-8d12-4b968be9de07-kube-api-access-vmjmp\") pod \"opendatahub-operator-controller-manager-8469c9486d-jqtgr\" (UID: \"bff62eb2-b9c8-4142-8d12-4b968be9de07\") " pod="opendatahub/opendatahub-operator-controller-manager-8469c9486d-jqtgr"
Sep 09 13:31:50.723453 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:50.723343    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8469c9486d-jqtgr"
Sep 09 13:31:51.024281 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:51.024255    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8469c9486d-jqtgr"]
Sep 09 13:31:51.027086 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:31:51.027055    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff62eb2_b9c8_4142_8d12_4b968be9de07.slice/crio-0a305431eca3054707e59901f13c6ee1dca2a55da716beb40a850d5a74c62d22 WatchSource:0}: Error finding container 0a305431eca3054707e59901f13c6ee1dca2a55da716beb40a850d5a74c62d22: Status 404 returned error can't find the container with id 0a305431eca3054707e59901f13c6ee1dca2a55da716beb40a850d5a74c62d22
Sep 09 13:31:51.028913 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:51.028892    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 13:31:51.653911 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:51.653868    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8469c9486d-jqtgr" event={"ID":"bff62eb2-b9c8-4142-8d12-4b968be9de07","Type":"ContainerStarted","Data":"0a305431eca3054707e59901f13c6ee1dca2a55da716beb40a850d5a74c62d22"}
Sep 09 13:31:54.669258 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:54.669221    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8469c9486d-jqtgr" event={"ID":"bff62eb2-b9c8-4142-8d12-4b968be9de07","Type":"ContainerStarted","Data":"4a7c4bea239bd686cad6036258f667c447c838d7537901f8bc6dbdcdb2cccad6"}
Sep 09 13:31:55.675204 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:55.675169    2566 generic.go:356] "Generic (PLEG): container finished" podID="bff62eb2-b9c8-4142-8d12-4b968be9de07" containerID="4a7c4bea239bd686cad6036258f667c447c838d7537901f8bc6dbdcdb2cccad6" exitCode=0
Sep 09 13:31:55.675606 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:55.675262    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8469c9486d-jqtgr" event={"ID":"bff62eb2-b9c8-4142-8d12-4b968be9de07","Type":"ContainerDied","Data":"4a7c4bea239bd686cad6036258f667c447c838d7537901f8bc6dbdcdb2cccad6"}
Sep 09 13:31:56.681887 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:56.681850    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8469c9486d-jqtgr" event={"ID":"bff62eb2-b9c8-4142-8d12-4b968be9de07","Type":"ContainerStarted","Data":"a4e91ba3722d78566fc4657dc4cd3512d1b9e914d9cf93c178a171504db29406"}
Sep 09 13:31:56.682253 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:56.681954    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8469c9486d-jqtgr"
Sep 09 13:31:56.716174 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:31:56.716111    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8469c9486d-jqtgr" podStartSLOduration=2.349789422 podStartE2EDuration="6.716088217s" totalImagesPullingTime="3.29274182s" totalInitContainerRuntime="1.073556975s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:31:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:31:56.713848344 +0000 UTC m=+350.073698164" watchObservedRunningTime="2026-09-09 13:31:56.716088217 +0000 UTC m=+350.075938043"
Sep 09 13:32:07.688879 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:07.688784    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8469c9486d-jqtgr"
Sep 09 13:32:20.536635 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:20.536593    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-595c6d67d9-zbpjd"]
Sep 09 13:32:20.547869 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:20.547837    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-595c6d67d9-zbpjd"
Sep 09 13:32:20.551179 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:20.551151    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 13:32:20.551395 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:20.551371    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 13:32:20.553849 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:20.553807    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-x7rbd\""
Sep 09 13:32:20.561054 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:20.561022    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-595c6d67d9-zbpjd"]
Sep 09 13:32:20.678895 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:20.678863    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10f201f3-a37d-454a-a0ba-e0230efcbcfd-tmp\") pod \"kube-auth-proxy-595c6d67d9-zbpjd\" (UID: \"10f201f3-a37d-454a-a0ba-e0230efcbcfd\") " pod="openshift-ingress/kube-auth-proxy-595c6d67d9-zbpjd"
Sep 09 13:32:20.679068 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:20.678904    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f201f3-a37d-454a-a0ba-e0230efcbcfd-tls-certs\") pod \"kube-auth-proxy-595c6d67d9-zbpjd\" (UID: \"10f201f3-a37d-454a-a0ba-e0230efcbcfd\") " pod="openshift-ingress/kube-auth-proxy-595c6d67d9-zbpjd"
Sep 09 13:32:20.679068 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:20.678978    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45n2b\" (UniqueName: \"kubernetes.io/projected/10f201f3-a37d-454a-a0ba-e0230efcbcfd-kube-api-access-45n2b\") pod \"kube-auth-proxy-595c6d67d9-zbpjd\" (UID: \"10f201f3-a37d-454a-a0ba-e0230efcbcfd\") " pod="openshift-ingress/kube-auth-proxy-595c6d67d9-zbpjd"
Sep 09 13:32:20.779453 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:20.779412    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10f201f3-a37d-454a-a0ba-e0230efcbcfd-tmp\") pod \"kube-auth-proxy-595c6d67d9-zbpjd\" (UID: \"10f201f3-a37d-454a-a0ba-e0230efcbcfd\") " pod="openshift-ingress/kube-auth-proxy-595c6d67d9-zbpjd"
Sep 09 13:32:20.779650 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:20.779460    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f201f3-a37d-454a-a0ba-e0230efcbcfd-tls-certs\") pod \"kube-auth-proxy-595c6d67d9-zbpjd\" (UID: \"10f201f3-a37d-454a-a0ba-e0230efcbcfd\") " pod="openshift-ingress/kube-auth-proxy-595c6d67d9-zbpjd"
Sep 09 13:32:20.779650 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:20.779508    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45n2b\" (UniqueName: \"kubernetes.io/projected/10f201f3-a37d-454a-a0ba-e0230efcbcfd-kube-api-access-45n2b\") pod \"kube-auth-proxy-595c6d67d9-zbpjd\" (UID: \"10f201f3-a37d-454a-a0ba-e0230efcbcfd\") " pod="openshift-ingress/kube-auth-proxy-595c6d67d9-zbpjd"
Sep 09 13:32:20.782264 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:20.782218    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10f201f3-a37d-454a-a0ba-e0230efcbcfd-tmp\") pod \"kube-auth-proxy-595c6d67d9-zbpjd\" (UID: \"10f201f3-a37d-454a-a0ba-e0230efcbcfd\") " pod="openshift-ingress/kube-auth-proxy-595c6d67d9-zbpjd"
Sep 09 13:32:20.782429 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:20.782342    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f201f3-a37d-454a-a0ba-e0230efcbcfd-tls-certs\") pod \"kube-auth-proxy-595c6d67d9-zbpjd\" (UID: \"10f201f3-a37d-454a-a0ba-e0230efcbcfd\") " pod="openshift-ingress/kube-auth-proxy-595c6d67d9-zbpjd"
Sep 09 13:32:20.798793 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:20.798713    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45n2b\" (UniqueName: \"kubernetes.io/projected/10f201f3-a37d-454a-a0ba-e0230efcbcfd-kube-api-access-45n2b\") pod \"kube-auth-proxy-595c6d67d9-zbpjd\" (UID: \"10f201f3-a37d-454a-a0ba-e0230efcbcfd\") " pod="openshift-ingress/kube-auth-proxy-595c6d67d9-zbpjd"
Sep 09 13:32:20.857540 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:20.857498    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-595c6d67d9-zbpjd"
Sep 09 13:32:21.136161 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:21.136124    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-595c6d67d9-zbpjd"]
Sep 09 13:32:21.141635 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:32:21.141604    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f201f3_a37d_454a_a0ba_e0230efcbcfd.slice/crio-c862e696faa4985671624f660b9f74540a89c591b77a4f81d8870b6e2482678d WatchSource:0}: Error finding container c862e696faa4985671624f660b9f74540a89c591b77a4f81d8870b6e2482678d: Status 404 returned error can't find the container with id c862e696faa4985671624f660b9f74540a89c591b77a4f81d8870b6e2482678d
Sep 09 13:32:21.793212 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:21.793169    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-595c6d67d9-zbpjd" event={"ID":"10f201f3-a37d-454a-a0ba-e0230efcbcfd","Type":"ContainerStarted","Data":"c862e696faa4985671624f660b9f74540a89c591b77a4f81d8870b6e2482678d"}
Sep 09 13:32:24.810888 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:24.810845    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-595c6d67d9-zbpjd" event={"ID":"10f201f3-a37d-454a-a0ba-e0230efcbcfd","Type":"ContainerStarted","Data":"6fc0a52d703fc770b6e6de52cac03832ce4216df54d2846b27d7d5b88b5132f3"}
Sep 09 13:32:24.828680 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:24.826977    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-595c6d67d9-zbpjd" podStartSLOduration=2.087834444 podStartE2EDuration="4.826954454s" totalImagesPullingTime="2.73912001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:32:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:32:24.826245422 +0000 UTC m=+378.186095251" watchObservedRunningTime="2026-09-09 13:32:24.826954454 +0000 UTC m=+378.186804281"
Sep 09 13:32:29.284121 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.284082    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"]
Sep 09 13:32:29.287700 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.287679    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.290280 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.290242    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-qn6gv\""
Sep 09 13:32:29.291034 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.291006    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 13:32:29.310282 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.310248    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"]
Sep 09 13:32:29.457556 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.457513    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/766fc6d1-54c1-4a76-844a-f828adb2442d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn\" (UID: \"766fc6d1-54c1-4a76-844a-f828adb2442d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.457556 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.457564    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/766fc6d1-54c1-4a76-844a-f828adb2442d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn\" (UID: \"766fc6d1-54c1-4a76-844a-f828adb2442d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.457808 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.457587    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88ttr\" (UniqueName: \"kubernetes.io/projected/766fc6d1-54c1-4a76-844a-f828adb2442d-kube-api-access-88ttr\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn\" (UID: \"766fc6d1-54c1-4a76-844a-f828adb2442d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.457808 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.457684    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/766fc6d1-54c1-4a76-844a-f828adb2442d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn\" (UID: \"766fc6d1-54c1-4a76-844a-f828adb2442d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.457808 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.457721    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/766fc6d1-54c1-4a76-844a-f828adb2442d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn\" (UID: \"766fc6d1-54c1-4a76-844a-f828adb2442d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.457808 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.457751    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/766fc6d1-54c1-4a76-844a-f828adb2442d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn\" (UID: \"766fc6d1-54c1-4a76-844a-f828adb2442d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.457808 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.457800    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/766fc6d1-54c1-4a76-844a-f828adb2442d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn\" (UID: \"766fc6d1-54c1-4a76-844a-f828adb2442d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.458082 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.457866    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/766fc6d1-54c1-4a76-844a-f828adb2442d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn\" (UID: \"766fc6d1-54c1-4a76-844a-f828adb2442d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.458082 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.457894    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/766fc6d1-54c1-4a76-844a-f828adb2442d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn\" (UID: \"766fc6d1-54c1-4a76-844a-f828adb2442d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.558750 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.558646    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/766fc6d1-54c1-4a76-844a-f828adb2442d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn\" (UID: \"766fc6d1-54c1-4a76-844a-f828adb2442d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.558750 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.558701    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88ttr\" (UniqueName: \"kubernetes.io/projected/766fc6d1-54c1-4a76-844a-f828adb2442d-kube-api-access-88ttr\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn\" (UID: \"766fc6d1-54c1-4a76-844a-f828adb2442d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.559022 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.558753    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/766fc6d1-54c1-4a76-844a-f828adb2442d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn\" (UID: \"766fc6d1-54c1-4a76-844a-f828adb2442d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.559022 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.558781    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/766fc6d1-54c1-4a76-844a-f828adb2442d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn\" (UID: \"766fc6d1-54c1-4a76-844a-f828adb2442d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.559022 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.558849    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/766fc6d1-54c1-4a76-844a-f828adb2442d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn\" (UID: \"766fc6d1-54c1-4a76-844a-f828adb2442d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.559022 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.558875    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/766fc6d1-54c1-4a76-844a-f828adb2442d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn\" (UID: \"766fc6d1-54c1-4a76-844a-f828adb2442d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.559022 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.558897    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/766fc6d1-54c1-4a76-844a-f828adb2442d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn\" (UID: \"766fc6d1-54c1-4a76-844a-f828adb2442d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.559022 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.558932    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/766fc6d1-54c1-4a76-844a-f828adb2442d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn\" (UID: \"766fc6d1-54c1-4a76-844a-f828adb2442d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.559022 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.558998    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/766fc6d1-54c1-4a76-844a-f828adb2442d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn\" (UID: \"766fc6d1-54c1-4a76-844a-f828adb2442d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.559623 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.559592    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/766fc6d1-54c1-4a76-844a-f828adb2442d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn\" (UID: \"766fc6d1-54c1-4a76-844a-f828adb2442d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.559794 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.559759    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/766fc6d1-54c1-4a76-844a-f828adb2442d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn\" (UID: \"766fc6d1-54c1-4a76-844a-f828adb2442d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.559928 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.559688    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/766fc6d1-54c1-4a76-844a-f828adb2442d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn\" (UID: \"766fc6d1-54c1-4a76-844a-f828adb2442d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.559928 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.559648    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/766fc6d1-54c1-4a76-844a-f828adb2442d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn\" (UID: \"766fc6d1-54c1-4a76-844a-f828adb2442d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.560075 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.560055    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/766fc6d1-54c1-4a76-844a-f828adb2442d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn\" (UID: \"766fc6d1-54c1-4a76-844a-f828adb2442d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.561554 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.561533    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/766fc6d1-54c1-4a76-844a-f828adb2442d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn\" (UID: \"766fc6d1-54c1-4a76-844a-f828adb2442d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.561689 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.561674    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/766fc6d1-54c1-4a76-844a-f828adb2442d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn\" (UID: \"766fc6d1-54c1-4a76-844a-f828adb2442d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.573670 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.573634    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-68df89f6f5-kknpr"]
Sep 09 13:32:29.577363 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.577344    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68df89f6f5-kknpr"
Sep 09 13:32:29.579890 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.579867    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 13:32:29.580964 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.580944    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 13:32:29.581437 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.581415    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 13:32:29.581437 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.581437    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-6jc27\""
Sep 09 13:32:29.581602 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.581454    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 13:32:29.581602 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.581490    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 13:32:29.584358 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.584329    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/766fc6d1-54c1-4a76-844a-f828adb2442d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn\" (UID: \"766fc6d1-54c1-4a76-844a-f828adb2442d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.584714 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.584694    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88ttr\" (UniqueName: \"kubernetes.io/projected/766fc6d1-54c1-4a76-844a-f828adb2442d-kube-api-access-88ttr\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn\" (UID: \"766fc6d1-54c1-4a76-844a-f828adb2442d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.595781 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.595750    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68df89f6f5-kknpr"]
Sep 09 13:32:29.598611 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.598583    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:29.660852 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.660753    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5d616539-8e2a-4076-9abf-ea8dd8f8cbc3-manager-config\") pod \"lws-controller-manager-68df89f6f5-kknpr\" (UID: \"5d616539-8e2a-4076-9abf-ea8dd8f8cbc3\") " pod="openshift-lws-operator/lws-controller-manager-68df89f6f5-kknpr"
Sep 09 13:32:29.661020 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.660914    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2q75\" (UniqueName: \"kubernetes.io/projected/5d616539-8e2a-4076-9abf-ea8dd8f8cbc3-kube-api-access-v2q75\") pod \"lws-controller-manager-68df89f6f5-kknpr\" (UID: \"5d616539-8e2a-4076-9abf-ea8dd8f8cbc3\") " pod="openshift-lws-operator/lws-controller-manager-68df89f6f5-kknpr"
Sep 09 13:32:29.661020 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.660976    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5d616539-8e2a-4076-9abf-ea8dd8f8cbc3-metrics-cert\") pod \"lws-controller-manager-68df89f6f5-kknpr\" (UID: \"5d616539-8e2a-4076-9abf-ea8dd8f8cbc3\") " pod="openshift-lws-operator/lws-controller-manager-68df89f6f5-kknpr"
Sep 09 13:32:29.661020 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.661011    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d616539-8e2a-4076-9abf-ea8dd8f8cbc3-cert\") pod \"lws-controller-manager-68df89f6f5-kknpr\" (UID: \"5d616539-8e2a-4076-9abf-ea8dd8f8cbc3\") " pod="openshift-lws-operator/lws-controller-manager-68df89f6f5-kknpr"
Sep 09 13:32:29.762208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.762161    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5d616539-8e2a-4076-9abf-ea8dd8f8cbc3-metrics-cert\") pod \"lws-controller-manager-68df89f6f5-kknpr\" (UID: \"5d616539-8e2a-4076-9abf-ea8dd8f8cbc3\") " pod="openshift-lws-operator/lws-controller-manager-68df89f6f5-kknpr"
Sep 09 13:32:29.762208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.762214    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d616539-8e2a-4076-9abf-ea8dd8f8cbc3-cert\") pod \"lws-controller-manager-68df89f6f5-kknpr\" (UID: \"5d616539-8e2a-4076-9abf-ea8dd8f8cbc3\") " pod="openshift-lws-operator/lws-controller-manager-68df89f6f5-kknpr"
Sep 09 13:32:29.762473 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.762250    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5d616539-8e2a-4076-9abf-ea8dd8f8cbc3-manager-config\") pod \"lws-controller-manager-68df89f6f5-kknpr\" (UID: \"5d616539-8e2a-4076-9abf-ea8dd8f8cbc3\") " pod="openshift-lws-operator/lws-controller-manager-68df89f6f5-kknpr"
Sep 09 13:32:29.762473 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.762289    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2q75\" (UniqueName: \"kubernetes.io/projected/5d616539-8e2a-4076-9abf-ea8dd8f8cbc3-kube-api-access-v2q75\") pod \"lws-controller-manager-68df89f6f5-kknpr\" (UID: \"5d616539-8e2a-4076-9abf-ea8dd8f8cbc3\") " pod="openshift-lws-operator/lws-controller-manager-68df89f6f5-kknpr"
Sep 09 13:32:29.763072 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.763042    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5d616539-8e2a-4076-9abf-ea8dd8f8cbc3-manager-config\") pod \"lws-controller-manager-68df89f6f5-kknpr\" (UID: \"5d616539-8e2a-4076-9abf-ea8dd8f8cbc3\") " pod="openshift-lws-operator/lws-controller-manager-68df89f6f5-kknpr"
Sep 09 13:32:29.765016 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.764990    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5d616539-8e2a-4076-9abf-ea8dd8f8cbc3-metrics-cert\") pod \"lws-controller-manager-68df89f6f5-kknpr\" (UID: \"5d616539-8e2a-4076-9abf-ea8dd8f8cbc3\") " pod="openshift-lws-operator/lws-controller-manager-68df89f6f5-kknpr"
Sep 09 13:32:29.765109 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.764990    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d616539-8e2a-4076-9abf-ea8dd8f8cbc3-cert\") pod \"lws-controller-manager-68df89f6f5-kknpr\" (UID: \"5d616539-8e2a-4076-9abf-ea8dd8f8cbc3\") " pod="openshift-lws-operator/lws-controller-manager-68df89f6f5-kknpr"
Sep 09 13:32:29.777914 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.777884    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2q75\" (UniqueName: \"kubernetes.io/projected/5d616539-8e2a-4076-9abf-ea8dd8f8cbc3-kube-api-access-v2q75\") pod \"lws-controller-manager-68df89f6f5-kknpr\" (UID: \"5d616539-8e2a-4076-9abf-ea8dd8f8cbc3\") " pod="openshift-lws-operator/lws-controller-manager-68df89f6f5-kknpr"
Sep 09 13:32:29.895294 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.895255    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68df89f6f5-kknpr"
Sep 09 13:32:29.900174 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:29.900149    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"]
Sep 09 13:32:29.905954 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:32:29.905917    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod766fc6d1_54c1_4a76_844a_f828adb2442d.slice/crio-58e4685f0d6326fa9703c7a28c56bf239ffab03ea40b0b14253b32d5fe7d8253 WatchSource:0}: Error finding container 58e4685f0d6326fa9703c7a28c56bf239ffab03ea40b0b14253b32d5fe7d8253: Status 404 returned error can't find the container with id 58e4685f0d6326fa9703c7a28c56bf239ffab03ea40b0b14253b32d5fe7d8253
Sep 09 13:32:30.195080 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:30.195001    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68df89f6f5-kknpr"]
Sep 09 13:32:30.198364 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:32:30.198332    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d616539_8e2a_4076_9abf_ea8dd8f8cbc3.slice/crio-5b2ca7f3de5681009e6079ce3466c4c16190ea5baa1d2b0c9f96da5759509ea2 WatchSource:0}: Error finding container 5b2ca7f3de5681009e6079ce3466c4c16190ea5baa1d2b0c9f96da5759509ea2: Status 404 returned error can't find the container with id 5b2ca7f3de5681009e6079ce3466c4c16190ea5baa1d2b0c9f96da5759509ea2
Sep 09 13:32:30.840009 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:30.839975    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn" event={"ID":"766fc6d1-54c1-4a76-844a-f828adb2442d","Type":"ContainerStarted","Data":"58e4685f0d6326fa9703c7a28c56bf239ffab03ea40b0b14253b32d5fe7d8253"}
Sep 09 13:32:30.841223 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:30.841197    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68df89f6f5-kknpr" event={"ID":"5d616539-8e2a-4076-9abf-ea8dd8f8cbc3","Type":"ContainerStarted","Data":"5b2ca7f3de5681009e6079ce3466c4c16190ea5baa1d2b0c9f96da5759509ea2"}
Sep 09 13:32:34.382562 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:34.382524    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:32:34.382844 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:34.382596    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:32:34.382844 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:34.382624    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:32:34.865902 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:34.865863    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn" event={"ID":"766fc6d1-54c1-4a76-844a-f828adb2442d","Type":"ContainerStarted","Data":"17421369c75b57d25213def7451deb6ef44ec841501b772650d965d79f721c2e"}
Sep 09 13:32:34.867391 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:34.867363    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68df89f6f5-kknpr" event={"ID":"5d616539-8e2a-4076-9abf-ea8dd8f8cbc3","Type":"ContainerStarted","Data":"0ab5a581276c625ec01074955a24fdad70ee44102f68e6137690e520934efaa2"}
Sep 09 13:32:34.867521 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:34.867492    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-68df89f6f5-kknpr"
Sep 09 13:32:34.890274 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:34.890218    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn" podStartSLOduration=1.415973546 podStartE2EDuration="5.890200705s" totalImagesPullingTime="4.474227159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:32:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:32:34.889324202 +0000 UTC m=+388.249174033" watchObservedRunningTime="2026-09-09 13:32:34.890200705 +0000 UTC m=+388.250050531"
Sep 09 13:32:34.909849 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:34.909778    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-68df89f6f5-kknpr" podStartSLOduration=1.72957581 podStartE2EDuration="5.909761229s" totalImagesPullingTime="4.180185419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:32:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:32:34.909251016 +0000 UTC m=+388.269100842" watchObservedRunningTime="2026-09-09 13:32:34.909761229 +0000 UTC m=+388.269611054"
Sep 09 13:32:35.599717 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:35.599682    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:35.735737 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:35.735706    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:35.871730 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:35.871646    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:35.872494 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:35.872478    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn"
Sep 09 13:32:45.873348 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:32:45.873317    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-68df89f6f5-kknpr"
Sep 09 13:34:51.031582 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:34:51.031545    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-29wtg"]
Sep 09 13:34:51.034742 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:34:51.034724    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-29wtg"
Sep 09 13:34:51.036898 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:34:51.036871    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 13:34:51.037042 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:34:51.037024    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 13:34:51.037652 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:34:51.037635    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pdffk\""
Sep 09 13:34:51.042450 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:34:51.042429    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-29wtg"]
Sep 09 13:34:51.173082 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:34:51.173038    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq9mq\" (UniqueName: \"kubernetes.io/projected/2697393f-2811-4085-af3c-5b99822c9a89-kube-api-access-xq9mq\") pod \"authorino-59c974bd48-29wtg\" (UID: \"2697393f-2811-4085-af3c-5b99822c9a89\") " pod="kuadrant-system/authorino-59c974bd48-29wtg"
Sep 09 13:34:51.274466 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:34:51.274427    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq9mq\" (UniqueName: \"kubernetes.io/projected/2697393f-2811-4085-af3c-5b99822c9a89-kube-api-access-xq9mq\") pod \"authorino-59c974bd48-29wtg\" (UID: \"2697393f-2811-4085-af3c-5b99822c9a89\") " pod="kuadrant-system/authorino-59c974bd48-29wtg"
Sep 09 13:34:51.284516 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:34:51.284462    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq9mq\" (UniqueName: \"kubernetes.io/projected/2697393f-2811-4085-af3c-5b99822c9a89-kube-api-access-xq9mq\") pod \"authorino-59c974bd48-29wtg\" (UID: \"2697393f-2811-4085-af3c-5b99822c9a89\") " pod="kuadrant-system/authorino-59c974bd48-29wtg"
Sep 09 13:34:51.344387 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:34:51.344349    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-29wtg"
Sep 09 13:34:51.621070 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:34:51.621038    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-29wtg"]
Sep 09 13:34:51.624396 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:34:51.624373    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2697393f_2811_4085_af3c_5b99822c9a89.slice/crio-4d3cc4f4b21457fef05332bce7592ded1c1543204dc871e293287a8bcbfbdc30 WatchSource:0}: Error finding container 4d3cc4f4b21457fef05332bce7592ded1c1543204dc871e293287a8bcbfbdc30: Status 404 returned error can't find the container with id 4d3cc4f4b21457fef05332bce7592ded1c1543204dc871e293287a8bcbfbdc30
Sep 09 13:34:52.461406 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:34:52.461367    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-29wtg" event={"ID":"2697393f-2811-4085-af3c-5b99822c9a89","Type":"ContainerStarted","Data":"4d3cc4f4b21457fef05332bce7592ded1c1543204dc871e293287a8bcbfbdc30"}
Sep 09 13:34:54.472476 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:34:54.472394    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-29wtg" event={"ID":"2697393f-2811-4085-af3c-5b99822c9a89","Type":"ContainerStarted","Data":"b20d9bacd041bd92c433043ee19a7f2cb60266c6ec195c8e33c9b6351e3acb5a"}
Sep 09 13:34:54.491649 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:34:54.491603    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-29wtg" podStartSLOduration=0.939335758 podStartE2EDuration="3.491587961s" totalImagesPullingTime="2.552252203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:34:54.490143195 +0000 UTC m=+527.849993022" watchObservedRunningTime="2026-09-09 13:34:54.491587961 +0000 UTC m=+527.851437786"
Sep 09 13:35:33.348511 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:35:33.348411    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-29wtg"]
Sep 09 13:35:33.349209 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:35:33.348608    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-29wtg" podUID="2697393f-2811-4085-af3c-5b99822c9a89" containerName="authorino" containerID="cri-o://b20d9bacd041bd92c433043ee19a7f2cb60266c6ec195c8e33c9b6351e3acb5a" gracePeriod=30
Sep 09 13:35:33.586806 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:35:33.586776    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-29wtg"
Sep 09 13:35:33.614302 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:35:33.614176    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xq9mq\" (UniqueName: \"kubernetes.io/projected/2697393f-2811-4085-af3c-5b99822c9a89-kube-api-access-xq9mq\") pod \"2697393f-2811-4085-af3c-5b99822c9a89\" (UID: \"2697393f-2811-4085-af3c-5b99822c9a89\") "
Sep 09 13:35:33.616539 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:35:33.616511    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2697393f-2811-4085-af3c-5b99822c9a89-kube-api-access-xq9mq" (OuterVolumeSpecName: "kube-api-access-xq9mq") pod "2697393f-2811-4085-af3c-5b99822c9a89" (UID: "2697393f-2811-4085-af3c-5b99822c9a89"). InnerVolumeSpecName "kube-api-access-xq9mq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:35:33.648755 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:35:33.648722    2566 generic.go:356] "Generic (PLEG): container finished" podID="2697393f-2811-4085-af3c-5b99822c9a89" containerID="b20d9bacd041bd92c433043ee19a7f2cb60266c6ec195c8e33c9b6351e3acb5a" exitCode=0
Sep 09 13:35:33.648970 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:35:33.648769    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-29wtg"
Sep 09 13:35:33.648970 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:35:33.648825    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-29wtg" event={"ID":"2697393f-2811-4085-af3c-5b99822c9a89","Type":"ContainerDied","Data":"b20d9bacd041bd92c433043ee19a7f2cb60266c6ec195c8e33c9b6351e3acb5a"}
Sep 09 13:35:33.648970 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:35:33.648877    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-29wtg" event={"ID":"2697393f-2811-4085-af3c-5b99822c9a89","Type":"ContainerDied","Data":"4d3cc4f4b21457fef05332bce7592ded1c1543204dc871e293287a8bcbfbdc30"}
Sep 09 13:35:33.648970 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:35:33.648900    2566 scope.go:117] "RemoveContainer" containerID="b20d9bacd041bd92c433043ee19a7f2cb60266c6ec195c8e33c9b6351e3acb5a"
Sep 09 13:35:33.657369 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:35:33.657352    2566 scope.go:117] "RemoveContainer" containerID="b20d9bacd041bd92c433043ee19a7f2cb60266c6ec195c8e33c9b6351e3acb5a"
Sep 09 13:35:33.657639 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:35:33.657621    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b20d9bacd041bd92c433043ee19a7f2cb60266c6ec195c8e33c9b6351e3acb5a\": container with ID starting with b20d9bacd041bd92c433043ee19a7f2cb60266c6ec195c8e33c9b6351e3acb5a not found: ID does not exist" containerID="b20d9bacd041bd92c433043ee19a7f2cb60266c6ec195c8e33c9b6351e3acb5a"
Sep 09 13:35:33.657686 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:35:33.657648    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b20d9bacd041bd92c433043ee19a7f2cb60266c6ec195c8e33c9b6351e3acb5a"} err="failed to get container status \"b20d9bacd041bd92c433043ee19a7f2cb60266c6ec195c8e33c9b6351e3acb5a\": rpc error: code = NotFound desc = could not find container \"b20d9bacd041bd92c433043ee19a7f2cb60266c6ec195c8e33c9b6351e3acb5a\": container with ID starting with b20d9bacd041bd92c433043ee19a7f2cb60266c6ec195c8e33c9b6351e3acb5a not found: ID does not exist"
Sep 09 13:35:33.668917 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:35:33.668891    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-29wtg"]
Sep 09 13:35:33.672473 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:35:33.672453    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-29wtg"]
Sep 09 13:35:33.715201 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:35:33.715164    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq9mq\" (UniqueName: \"kubernetes.io/projected/2697393f-2811-4085-af3c-5b99822c9a89-kube-api-access-xq9mq\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:35:35.149423 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:35:35.149388    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2697393f-2811-4085-af3c-5b99822c9a89" path="/var/lib/kubelet/pods/2697393f-2811-4085-af3c-5b99822c9a89/volumes"
Sep 09 13:36:07.100483 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:07.100454    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ftm96_af902e8b-d693-4341-9979-03cf69548cfb/console-operator/2.log"
Sep 09 13:36:07.113906 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:07.113881    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ftm96_af902e8b-d693-4341-9979-03cf69548cfb/console-operator/2.log"
Sep 09 13:36:51.172654 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.172620    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z"]
Sep 09 13:36:51.173084 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.172952    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2697393f-2811-4085-af3c-5b99822c9a89" containerName="authorino"
Sep 09 13:36:51.173084 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.172962    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2697393f-2811-4085-af3c-5b99822c9a89" containerName="authorino"
Sep 09 13:36:51.173084 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.173015    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2697393f-2811-4085-af3c-5b99822c9a89" containerName="authorino"
Sep 09 13:36:51.175945 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.175929    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z"
Sep 09 13:36:51.178966 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.178939    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 09 13:36:51.178966 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.178954    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 13:36:51.178966 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.178954    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ztzlx\""
Sep 09 13:36:51.179191 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.179031    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 13:36:51.187873 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.187848    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z"]
Sep 09 13:36:51.289214 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.289165    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/366979d9-163d-42be-aa7a-258f9ad1b098-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5t22z\" (UID: \"366979d9-163d-42be-aa7a-258f9ad1b098\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z"
Sep 09 13:36:51.289214 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.289214    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d95r\" (UniqueName: \"kubernetes.io/projected/366979d9-163d-42be-aa7a-258f9ad1b098-kube-api-access-4d95r\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5t22z\" (UID: \"366979d9-163d-42be-aa7a-258f9ad1b098\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z"
Sep 09 13:36:51.289482 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.289244    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/366979d9-163d-42be-aa7a-258f9ad1b098-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5t22z\" (UID: \"366979d9-163d-42be-aa7a-258f9ad1b098\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z"
Sep 09 13:36:51.289482 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.289307    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/366979d9-163d-42be-aa7a-258f9ad1b098-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5t22z\" (UID: \"366979d9-163d-42be-aa7a-258f9ad1b098\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z"
Sep 09 13:36:51.289482 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.289329    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/366979d9-163d-42be-aa7a-258f9ad1b098-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5t22z\" (UID: \"366979d9-163d-42be-aa7a-258f9ad1b098\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z"
Sep 09 13:36:51.289482 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.289360    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/366979d9-163d-42be-aa7a-258f9ad1b098-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5t22z\" (UID: \"366979d9-163d-42be-aa7a-258f9ad1b098\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z"
Sep 09 13:36:51.390163 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.390123    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/366979d9-163d-42be-aa7a-258f9ad1b098-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5t22z\" (UID: \"366979d9-163d-42be-aa7a-258f9ad1b098\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z"
Sep 09 13:36:51.390163 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.390165    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/366979d9-163d-42be-aa7a-258f9ad1b098-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5t22z\" (UID: \"366979d9-163d-42be-aa7a-258f9ad1b098\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z"
Sep 09 13:36:51.390429 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.390186    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/366979d9-163d-42be-aa7a-258f9ad1b098-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5t22z\" (UID: \"366979d9-163d-42be-aa7a-258f9ad1b098\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z"
Sep 09 13:36:51.390429 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.390228    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/366979d9-163d-42be-aa7a-258f9ad1b098-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5t22z\" (UID: \"366979d9-163d-42be-aa7a-258f9ad1b098\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z"
Sep 09 13:36:51.390429 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.390255    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d95r\" (UniqueName: \"kubernetes.io/projected/366979d9-163d-42be-aa7a-258f9ad1b098-kube-api-access-4d95r\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5t22z\" (UID: \"366979d9-163d-42be-aa7a-258f9ad1b098\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z"
Sep 09 13:36:51.390429 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.390274    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/366979d9-163d-42be-aa7a-258f9ad1b098-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5t22z\" (UID: \"366979d9-163d-42be-aa7a-258f9ad1b098\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z"
Sep 09 13:36:51.390668 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.390645    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/366979d9-163d-42be-aa7a-258f9ad1b098-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5t22z\" (UID: \"366979d9-163d-42be-aa7a-258f9ad1b098\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z"
Sep 09 13:36:51.390744 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.390680    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/366979d9-163d-42be-aa7a-258f9ad1b098-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5t22z\" (UID: \"366979d9-163d-42be-aa7a-258f9ad1b098\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z"
Sep 09 13:36:51.390744 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.390653    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/366979d9-163d-42be-aa7a-258f9ad1b098-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5t22z\" (UID: \"366979d9-163d-42be-aa7a-258f9ad1b098\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z"
Sep 09 13:36:51.392486 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.392467    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/366979d9-163d-42be-aa7a-258f9ad1b098-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5t22z\" (UID: \"366979d9-163d-42be-aa7a-258f9ad1b098\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z"
Sep 09 13:36:51.392805 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.392788    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/366979d9-163d-42be-aa7a-258f9ad1b098-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5t22z\" (UID: \"366979d9-163d-42be-aa7a-258f9ad1b098\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z"
Sep 09 13:36:51.399033 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.399006    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d95r\" (UniqueName: \"kubernetes.io/projected/366979d9-163d-42be-aa7a-258f9ad1b098-kube-api-access-4d95r\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5t22z\" (UID: \"366979d9-163d-42be-aa7a-258f9ad1b098\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z"
Sep 09 13:36:51.486376 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.486284    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z"
Sep 09 13:36:51.757598 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.757572    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z"]
Sep 09 13:36:51.759620 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:36:51.759589    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod366979d9_163d_42be_aa7a_258f9ad1b098.slice/crio-b962e0e5120ab890bbb6c7e7890f798d011377e4ac2064e552ee4a6bae73b9d5 WatchSource:0}: Error finding container b962e0e5120ab890bbb6c7e7890f798d011377e4ac2064e552ee4a6bae73b9d5: Status 404 returned error can't find the container with id b962e0e5120ab890bbb6c7e7890f798d011377e4ac2064e552ee4a6bae73b9d5
Sep 09 13:36:51.761208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.761187    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 13:36:51.988319 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:51.988276    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z" event={"ID":"366979d9-163d-42be-aa7a-258f9ad1b098","Type":"ContainerStarted","Data":"b962e0e5120ab890bbb6c7e7890f798d011377e4ac2064e552ee4a6bae73b9d5"}
Sep 09 13:36:52.994521 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:52.994418    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z" event={"ID":"366979d9-163d-42be-aa7a-258f9ad1b098","Type":"ContainerStarted","Data":"fb46e1433076ab15e21743137f17d7c32e33f2d56cc0c884f2578d7a897bdeb6"}
Sep 09 13:36:53.012734 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:36:53.012676    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z" podStartSLOduration=1.06548619 podStartE2EDuration="2.012654264s" totalImagesPullingTime="947.168074ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:36:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:36:53.011806295 +0000 UTC m=+646.371656122" watchObservedRunningTime="2026-09-09 13:36:53.012654264 +0000 UTC m=+646.372504094"
Sep 09 13:37:01.486695 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:01.486640    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z"
Sep 09 13:37:01.487169 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:01.486735    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z"
Sep 09 13:37:01.499448 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:01.499417    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z"
Sep 09 13:37:02.044841 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:02.044798    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5t22z"
Sep 09 13:37:04.357594 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:04.357556    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-h784p"]
Sep 09 13:37:04.361454 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:04.361430    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h784p"
Sep 09 13:37:04.363807 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:04.363784    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 09 13:37:04.370541 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:04.370515    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-h784p"]
Sep 09 13:37:04.504306 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:04.504265    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l9xm\" (UniqueName: \"kubernetes.io/projected/ef9d42fc-968c-411e-9981-b24e72db2c42-kube-api-access-9l9xm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h784p\" (UID: \"ef9d42fc-968c-411e-9981-b24e72db2c42\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h784p"
Sep 09 13:37:04.504538 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:04.504351    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef9d42fc-968c-411e-9981-b24e72db2c42-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h784p\" (UID: \"ef9d42fc-968c-411e-9981-b24e72db2c42\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h784p"
Sep 09 13:37:04.504538 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:04.504388    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef9d42fc-968c-411e-9981-b24e72db2c42-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h784p\" (UID: \"ef9d42fc-968c-411e-9981-b24e72db2c42\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h784p"
Sep 09 13:37:04.504538 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:04.504444    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef9d42fc-968c-411e-9981-b24e72db2c42-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h784p\" (UID: \"ef9d42fc-968c-411e-9981-b24e72db2c42\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h784p"
Sep 09 13:37:04.504538 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:04.504484    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef9d42fc-968c-411e-9981-b24e72db2c42-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h784p\" (UID: \"ef9d42fc-968c-411e-9981-b24e72db2c42\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h784p"
Sep 09 13:37:04.504538 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:04.504511    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef9d42fc-968c-411e-9981-b24e72db2c42-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h784p\" (UID: \"ef9d42fc-968c-411e-9981-b24e72db2c42\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h784p"
Sep 09 13:37:04.605677 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:04.605623    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9xm\" (UniqueName: \"kubernetes.io/projected/ef9d42fc-968c-411e-9981-b24e72db2c42-kube-api-access-9l9xm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h784p\" (UID: \"ef9d42fc-968c-411e-9981-b24e72db2c42\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h784p"
Sep 09 13:37:04.605920 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:04.605719    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef9d42fc-968c-411e-9981-b24e72db2c42-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h784p\" (UID: \"ef9d42fc-968c-411e-9981-b24e72db2c42\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h784p"
Sep 09 13:37:04.605920 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:04.605754    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef9d42fc-968c-411e-9981-b24e72db2c42-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h784p\" (UID: \"ef9d42fc-968c-411e-9981-b24e72db2c42\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h784p"
Sep 09 13:37:04.605920 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:04.605799    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef9d42fc-968c-411e-9981-b24e72db2c42-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h784p\" (UID: \"ef9d42fc-968c-411e-9981-b24e72db2c42\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h784p"
Sep 09 13:37:04.605920 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:04.605848    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef9d42fc-968c-411e-9981-b24e72db2c42-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h784p\" (UID: \"ef9d42fc-968c-411e-9981-b24e72db2c42\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h784p"
Sep 09 13:37:04.605920 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:04.605878    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef9d42fc-968c-411e-9981-b24e72db2c42-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h784p\" (UID: \"ef9d42fc-968c-411e-9981-b24e72db2c42\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h784p"
Sep 09 13:37:04.606198 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:04.606134    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef9d42fc-968c-411e-9981-b24e72db2c42-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h784p\" (UID: \"ef9d42fc-968c-411e-9981-b24e72db2c42\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h784p"
Sep 09 13:37:04.606268 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:04.606244    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef9d42fc-968c-411e-9981-b24e72db2c42-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h784p\" (UID: \"ef9d42fc-968c-411e-9981-b24e72db2c42\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h784p"
Sep 09 13:37:04.606330 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:04.606271    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef9d42fc-968c-411e-9981-b24e72db2c42-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h784p\" (UID: \"ef9d42fc-968c-411e-9981-b24e72db2c42\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h784p"
Sep 09 13:37:04.608185 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:04.608138    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef9d42fc-968c-411e-9981-b24e72db2c42-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h784p\" (UID: \"ef9d42fc-968c-411e-9981-b24e72db2c42\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h784p"
Sep 09 13:37:04.608490 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:04.608471    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef9d42fc-968c-411e-9981-b24e72db2c42-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h784p\" (UID: \"ef9d42fc-968c-411e-9981-b24e72db2c42\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h784p"
Sep 09 13:37:04.615945 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:04.615924    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l9xm\" (UniqueName: \"kubernetes.io/projected/ef9d42fc-968c-411e-9981-b24e72db2c42-kube-api-access-9l9xm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h784p\" (UID: \"ef9d42fc-968c-411e-9981-b24e72db2c42\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h784p"
Sep 09 13:37:04.675038 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:04.674989    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h784p"
Sep 09 13:37:04.946403 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:04.946319    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-h784p"]
Sep 09 13:37:04.949620 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:37:04.949591    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9d42fc_968c_411e_9981_b24e72db2c42.slice/crio-b32df163869433a0e43b1ac613e22b37604996d783e18e0141737d490a8e3340 WatchSource:0}: Error finding container b32df163869433a0e43b1ac613e22b37604996d783e18e0141737d490a8e3340: Status 404 returned error can't find the container with id b32df163869433a0e43b1ac613e22b37604996d783e18e0141737d490a8e3340
Sep 09 13:37:05.047480 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:05.047442    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h784p" event={"ID":"ef9d42fc-968c-411e-9981-b24e72db2c42","Type":"ContainerStarted","Data":"b32df163869433a0e43b1ac613e22b37604996d783e18e0141737d490a8e3340"}
Sep 09 13:37:06.053963 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:06.053922    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h784p" event={"ID":"ef9d42fc-968c-411e-9981-b24e72db2c42","Type":"ContainerStarted","Data":"5324bd111605be197958c4706dd62a99eeeb1564919ce76c05d99dcf8ec2b7e7"}
Sep 09 13:37:06.072827 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:06.072749    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h784p" podStartSLOduration=1.895090916 podStartE2EDuration="2.072726601s" totalImagesPullingTime="177.635685ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:37:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:37:06.070775865 +0000 UTC m=+659.430625714" watchObservedRunningTime="2026-09-09 13:37:06.072726601 +0000 UTC m=+659.432576444"
Sep 09 13:37:14.676082 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:14.676035    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h784p"
Sep 09 13:37:14.676478 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:14.676112    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h784p"
Sep 09 13:37:14.688327 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:14.688300    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h784p"
Sep 09 13:37:15.103233 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:15.103194    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h784p"
Sep 09 13:37:25.468839 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:25.468788    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24"]
Sep 09 13:37:25.475152 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:25.475126    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24"
Sep 09 13:37:25.477244 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:25.477212    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 09 13:37:25.480316 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:25.480290    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24"]
Sep 09 13:37:25.594615 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:25.594575    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e49c0e6c-c8e4-4356-be97-b1d89e8384d5-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-r4s24\" (UID: \"e49c0e6c-c8e4-4356-be97-b1d89e8384d5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24"
Sep 09 13:37:25.594615 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:25.594614    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e49c0e6c-c8e4-4356-be97-b1d89e8384d5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-r4s24\" (UID: \"e49c0e6c-c8e4-4356-be97-b1d89e8384d5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24"
Sep 09 13:37:25.594901 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:25.594646    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwwmf\" (UniqueName: \"kubernetes.io/projected/e49c0e6c-c8e4-4356-be97-b1d89e8384d5-kube-api-access-mwwmf\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-r4s24\" (UID: \"e49c0e6c-c8e4-4356-be97-b1d89e8384d5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24"
Sep 09 13:37:25.594901 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:25.594684    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e49c0e6c-c8e4-4356-be97-b1d89e8384d5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-r4s24\" (UID: \"e49c0e6c-c8e4-4356-be97-b1d89e8384d5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24"
Sep 09 13:37:25.594901 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:25.594746    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e49c0e6c-c8e4-4356-be97-b1d89e8384d5-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-r4s24\" (UID: \"e49c0e6c-c8e4-4356-be97-b1d89e8384d5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24"
Sep 09 13:37:25.594901 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:25.594784    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e49c0e6c-c8e4-4356-be97-b1d89e8384d5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-r4s24\" (UID: \"e49c0e6c-c8e4-4356-be97-b1d89e8384d5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24"
Sep 09 13:37:25.695659 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:25.695621    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e49c0e6c-c8e4-4356-be97-b1d89e8384d5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-r4s24\" (UID: \"e49c0e6c-c8e4-4356-be97-b1d89e8384d5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24"
Sep 09 13:37:25.695873 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:25.695667    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e49c0e6c-c8e4-4356-be97-b1d89e8384d5-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-r4s24\" (UID: \"e49c0e6c-c8e4-4356-be97-b1d89e8384d5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24"
Sep 09 13:37:25.695873 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:25.695690    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e49c0e6c-c8e4-4356-be97-b1d89e8384d5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-r4s24\" (UID: \"e49c0e6c-c8e4-4356-be97-b1d89e8384d5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24"
Sep 09 13:37:25.695873 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:25.695726    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwwmf\" (UniqueName: \"kubernetes.io/projected/e49c0e6c-c8e4-4356-be97-b1d89e8384d5-kube-api-access-mwwmf\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-r4s24\" (UID: \"e49c0e6c-c8e4-4356-be97-b1d89e8384d5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24"
Sep 09 13:37:25.695873 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:25.695745    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e49c0e6c-c8e4-4356-be97-b1d89e8384d5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-r4s24\" (UID: \"e49c0e6c-c8e4-4356-be97-b1d89e8384d5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24"
Sep 09 13:37:25.695873 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:25.695804    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e49c0e6c-c8e4-4356-be97-b1d89e8384d5-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-r4s24\" (UID: \"e49c0e6c-c8e4-4356-be97-b1d89e8384d5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24"
Sep 09 13:37:25.696138 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:25.696040    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e49c0e6c-c8e4-4356-be97-b1d89e8384d5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-r4s24\" (UID: \"e49c0e6c-c8e4-4356-be97-b1d89e8384d5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24"
Sep 09 13:37:25.696194 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:25.696151    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e49c0e6c-c8e4-4356-be97-b1d89e8384d5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-r4s24\" (UID: \"e49c0e6c-c8e4-4356-be97-b1d89e8384d5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24"
Sep 09 13:37:25.696246 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:25.696190    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e49c0e6c-c8e4-4356-be97-b1d89e8384d5-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-r4s24\" (UID: \"e49c0e6c-c8e4-4356-be97-b1d89e8384d5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24"
Sep 09 13:37:25.697993 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:25.697972    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e49c0e6c-c8e4-4356-be97-b1d89e8384d5-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-r4s24\" (UID: \"e49c0e6c-c8e4-4356-be97-b1d89e8384d5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24"
Sep 09 13:37:25.698333 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:25.698311    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e49c0e6c-c8e4-4356-be97-b1d89e8384d5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-r4s24\" (UID: \"e49c0e6c-c8e4-4356-be97-b1d89e8384d5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24"
Sep 09 13:37:25.703588 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:25.703569    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwwmf\" (UniqueName: \"kubernetes.io/projected/e49c0e6c-c8e4-4356-be97-b1d89e8384d5-kube-api-access-mwwmf\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-r4s24\" (UID: \"e49c0e6c-c8e4-4356-be97-b1d89e8384d5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24"
Sep 09 13:37:25.784728 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:25.784639    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24"
Sep 09 13:37:26.057280 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:26.057253    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24"]
Sep 09 13:37:26.059455 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:37:26.059413    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode49c0e6c_c8e4_4356_be97_b1d89e8384d5.slice/crio-7a4bf96eec3eeeea302b5224eb4f318c13a54ab0619c33fdd93587c16b413bbe WatchSource:0}: Error finding container 7a4bf96eec3eeeea302b5224eb4f318c13a54ab0619c33fdd93587c16b413bbe: Status 404 returned error can't find the container with id 7a4bf96eec3eeeea302b5224eb4f318c13a54ab0619c33fdd93587c16b413bbe
Sep 09 13:37:26.140210 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:26.140174    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24" event={"ID":"e49c0e6c-c8e4-4356-be97-b1d89e8384d5","Type":"ContainerStarted","Data":"7a4bf96eec3eeeea302b5224eb4f318c13a54ab0619c33fdd93587c16b413bbe"}
Sep 09 13:37:27.149148 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:27.149114    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24" event={"ID":"e49c0e6c-c8e4-4356-be97-b1d89e8384d5","Type":"ContainerStarted","Data":"76b04824c3616f6a5f38f5cf78fbd8dc0f8759a9e9bc21d840c259a5ead1371e"}
Sep 09 13:37:27.163358 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:27.163302    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24" podStartSLOduration=1.9377804140000001 podStartE2EDuration="2.163281659s" totalImagesPullingTime="225.501245ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:37:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:37:27.16293249 +0000 UTC m=+680.522782316" watchObservedRunningTime="2026-09-09 13:37:27.163281659 +0000 UTC m=+680.523131487"
Sep 09 13:37:35.785032 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:35.784989    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24"
Sep 09 13:37:35.785513 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:35.785055    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24"
Sep 09 13:37:35.797346 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:35.797319    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24"
Sep 09 13:37:36.194488 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:37:36.194460    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-r4s24"
Sep 09 13:38:59.671762 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.671724    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"]
Sep 09 13:38:59.675568 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.675540    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:38:59.677771 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.677745    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4f59a0-openshift-default-dockercfg-flb8q\""
Sep 09 13:38:59.690246 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.690217    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"]
Sep 09 13:38:59.724283 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.724249    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-istio-envoy\") pod \"e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") " pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:38:59.724449 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.724305    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-istio-data\") pod \"e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") " pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:38:59.724449 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.724390    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8d6623e-e731-4ace-8f3b-f44db09987c8-istiod-ca-cert\") pod \"e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") " pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:38:59.724449 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.724434    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8d6623e-e731-4ace-8f3b-f44db09987c8-istio-podinfo\") pod \"e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") " pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:38:59.724553 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.724512    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-workload-socket\") pod \"e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") " pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:38:59.724553 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.724531    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-credential-socket\") pod \"e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") " pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:38:59.724615 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.724551    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stk2w\" (UniqueName: \"kubernetes.io/projected/c8d6623e-e731-4ace-8f3b-f44db09987c8-kube-api-access-stk2w\") pod \"e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") " pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:38:59.724615 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.724594    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8d6623e-e731-4ace-8f3b-f44db09987c8-istio-token\") pod \"e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") " pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:38:59.724679 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.724620    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-workload-certs\") pod \"e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") " pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:38:59.825599 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.825565    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-istio-envoy\") pod \"e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") " pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:38:59.825765 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.825614    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-istio-data\") pod \"e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") " pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:38:59.825765 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.825652    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8d6623e-e731-4ace-8f3b-f44db09987c8-istiod-ca-cert\") pod \"e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") " pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:38:59.825765 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.825670    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8d6623e-e731-4ace-8f3b-f44db09987c8-istio-podinfo\") pod \"e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") " pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:38:59.825765 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.825730    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-workload-socket\") pod \"e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") " pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:38:59.825765 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.825754    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-credential-socket\") pod \"e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") " pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:38:59.826042 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.825778    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stk2w\" (UniqueName: \"kubernetes.io/projected/c8d6623e-e731-4ace-8f3b-f44db09987c8-kube-api-access-stk2w\") pod \"e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") " pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:38:59.826042 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.825808    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8d6623e-e731-4ace-8f3b-f44db09987c8-istio-token\") pod \"e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") " pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:38:59.826042 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.825847    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-workload-certs\") pod \"e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") " pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:38:59.826209 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.826189    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-workload-certs\") pod \"e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") " pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:38:59.826270 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.826238    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-credential-socket\") pod \"e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") " pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:38:59.826396 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.826375    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-istio-data\") pod \"e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") " pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:38:59.826449 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.826391    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-workload-socket\") pod \"e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") " pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:38:59.826488 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.826464    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8d6623e-e731-4ace-8f3b-f44db09987c8-istiod-ca-cert\") pod \"e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") " pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:38:59.827935 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.827912    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-istio-envoy\") pod \"e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") " pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:38:59.828318 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.828301    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8d6623e-e731-4ace-8f3b-f44db09987c8-istio-podinfo\") pod \"e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") " pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:38:59.833085 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.833067    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8d6623e-e731-4ace-8f3b-f44db09987c8-istio-token\") pod \"e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") " pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:38:59.833222 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.833205    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stk2w\" (UniqueName: \"kubernetes.io/projected/c8d6623e-e731-4ace-8f3b-f44db09987c8-kube-api-access-stk2w\") pod \"e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") " pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:38:59.986521 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:38:59.986433    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:39:00.263773 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:00.263740    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"]
Sep 09 13:39:00.266684 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:39:00.266654    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d6623e_e731_4ace_8f3b_f44db09987c8.slice/crio-139f26adcccd0a4e4ec25466a40b315ce0e40f768a3eb3d64844d34f80210ef7 WatchSource:0}: Error finding container 139f26adcccd0a4e4ec25466a40b315ce0e40f768a3eb3d64844d34f80210ef7: Status 404 returned error can't find the container with id 139f26adcccd0a4e4ec25466a40b315ce0e40f768a3eb3d64844d34f80210ef7
Sep 09 13:39:00.269277 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:00.269241    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:39:00.269337 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:00.269308    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:39:00.269371 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:00.269340    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:39:00.565022 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:00.564931    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh" event={"ID":"c8d6623e-e731-4ace-8f3b-f44db09987c8","Type":"ContainerStarted","Data":"49d2710ad58674557cfd0a8c1cfe7569b379b0b15112bad6ea0939d197f5326c"}
Sep 09 13:39:00.565022 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:00.565014    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh" event={"ID":"c8d6623e-e731-4ace-8f3b-f44db09987c8","Type":"ContainerStarted","Data":"139f26adcccd0a4e4ec25466a40b315ce0e40f768a3eb3d64844d34f80210ef7"}
Sep 09 13:39:00.590586 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:00.590525    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh" podStartSLOduration=1.5905077140000001 podStartE2EDuration="1.590507714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:39:00.587971135 +0000 UTC m=+773.947820960" watchObservedRunningTime="2026-09-09 13:39:00.590507714 +0000 UTC m=+773.950357540"
Sep 09 13:39:00.987229 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:00.987199    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:39:01.992026 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:01.991995    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:39:02.574870 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:02.574839    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:39:02.575931 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:02.575908    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:39:05.315699 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:05.315668    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7949-809753"]
Sep 09 13:39:05.318963 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:05.318944    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7949-809753"
Sep 09 13:39:05.321039 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:05.321017    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qg8jq\""
Sep 09 13:39:05.325502 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:05.325468    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7949-809753"]
Sep 09 13:39:05.378520 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:05.378491    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvshw\" (UniqueName: \"kubernetes.io/projected/c08cdbbd-d96b-4003-85bd-80811312a24d-kube-api-access-fvshw\") pod \"test-curl-7949-809753\" (UID: \"c08cdbbd-d96b-4003-85bd-80811312a24d\") " pod="opendatahub/test-curl-7949-809753"
Sep 09 13:39:05.479406 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:05.479375    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvshw\" (UniqueName: \"kubernetes.io/projected/c08cdbbd-d96b-4003-85bd-80811312a24d-kube-api-access-fvshw\") pod \"test-curl-7949-809753\" (UID: \"c08cdbbd-d96b-4003-85bd-80811312a24d\") " pod="opendatahub/test-curl-7949-809753"
Sep 09 13:39:05.487081 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:05.487055    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvshw\" (UniqueName: \"kubernetes.io/projected/c08cdbbd-d96b-4003-85bd-80811312a24d-kube-api-access-fvshw\") pod \"test-curl-7949-809753\" (UID: \"c08cdbbd-d96b-4003-85bd-80811312a24d\") " pod="opendatahub/test-curl-7949-809753"
Sep 09 13:39:05.629438 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:05.629407    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7949-809753"
Sep 09 13:39:05.896369 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:05.896347    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7949-809753"]
Sep 09 13:39:05.898873 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:39:05.898844    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc08cdbbd_d96b_4003_85bd_80811312a24d.slice/crio-452cf57b1616fcaede4cac1d142a2ad7ab15f921787698cba31bdb81affbe3d0 WatchSource:0}: Error finding container 452cf57b1616fcaede4cac1d142a2ad7ab15f921787698cba31bdb81affbe3d0: Status 404 returned error can't find the container with id 452cf57b1616fcaede4cac1d142a2ad7ab15f921787698cba31bdb81affbe3d0
Sep 09 13:39:06.593166 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:06.593126    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7949-809753" event={"ID":"c08cdbbd-d96b-4003-85bd-80811312a24d","Type":"ContainerStarted","Data":"452cf57b1616fcaede4cac1d142a2ad7ab15f921787698cba31bdb81affbe3d0"}
Sep 09 13:39:07.582690 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.582655    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"]
Sep 09 13:39:07.591890 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.591859    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:07.596705 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.596668    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a649773d-openshift-default-dockercfg-ksdwx\""
Sep 09 13:39:07.603969 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.603928    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"]
Sep 09 13:39:07.700337 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.700302    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-workload-socket\") pod \"e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") " pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:07.700534 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.700349    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istio-podinfo\") pod \"e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") " pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:07.700534 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.700427    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istio-envoy\") pod \"e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") " pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:07.700534 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.700452    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istio-data\") pod \"e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") " pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:07.700534 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.700480    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p58z\" (UniqueName: \"kubernetes.io/projected/76d4cffb-c769-4dfe-b2ab-63e57691feb8-kube-api-access-4p58z\") pod \"e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") " pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:07.700534 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.700515    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-credential-socket\") pod \"e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") " pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:07.700773 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.700573    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-workload-certs\") pod \"e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") " pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:07.700773 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.700629    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istiod-ca-cert\") pod \"e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") " pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:07.700773 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.700714    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istio-token\") pod \"e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") " pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:07.802306 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.802257    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istio-token\") pod \"e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") " pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:07.802499 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.802339    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-workload-socket\") pod \"e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") " pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:07.802499 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.802374    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istio-podinfo\") pod \"e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") " pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:07.802499 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.802420    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istio-envoy\") pod \"e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") " pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:07.802499 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.802443    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istio-data\") pod \"e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") " pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:07.802499 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.802474    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p58z\" (UniqueName: \"kubernetes.io/projected/76d4cffb-c769-4dfe-b2ab-63e57691feb8-kube-api-access-4p58z\") pod \"e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") " pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:07.802748 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.802506    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-credential-socket\") pod \"e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") " pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:07.802748 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.802533    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-workload-certs\") pod \"e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") " pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:07.802748 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.802572    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istiod-ca-cert\") pod \"e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") " pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:07.802935 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.802796    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-workload-socket\") pod \"e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") " pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:07.803119 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.803085    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-credential-socket\") pod \"e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") " pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:07.803384 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.803359    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istio-data\") pod \"e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") " pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:07.803517 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.803446    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-workload-certs\") pod \"e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") " pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:07.803589 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.803557    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istiod-ca-cert\") pod \"e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") " pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:07.805315 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.805288    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istio-podinfo\") pod \"e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") " pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:07.805540 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.805510    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istio-envoy\") pod \"e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") " pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:07.811333 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.811305    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istio-token\") pod \"e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") " pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:07.811451 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.811434    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p58z\" (UniqueName: \"kubernetes.io/projected/76d4cffb-c769-4dfe-b2ab-63e57691feb8-kube-api-access-4p58z\") pod \"e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") " pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:07.904656 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:07.904574    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:08.284285 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:08.284261    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"]
Sep 09 13:39:08.286601 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:39:08.286575    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76d4cffb_c769_4dfe_b2ab_63e57691feb8.slice/crio-59fef9903572a4bf2de9119affdf94f770e5ac495c7816e050335ebdb05947b3 WatchSource:0}: Error finding container 59fef9903572a4bf2de9119affdf94f770e5ac495c7816e050335ebdb05947b3: Status 404 returned error can't find the container with id 59fef9903572a4bf2de9119affdf94f770e5ac495c7816e050335ebdb05947b3
Sep 09 13:39:08.288780 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:08.288745    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:39:08.288900 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:08.288826    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:39:08.288900 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:08.288860    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:39:08.606167 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:08.606111    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7949-809753" event={"ID":"c08cdbbd-d96b-4003-85bd-80811312a24d","Type":"ContainerStarted","Data":"088642f72da074db1a3589779359c838f76d423b2464ddf523381a7c822070ea"}
Sep 09 13:39:08.607738 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:08.607712    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4" event={"ID":"76d4cffb-c769-4dfe-b2ab-63e57691feb8","Type":"ContainerStarted","Data":"fdb2c215e9ddf0c692e0b9b9c0ca6f81087d8ea6ed2ed6ccf7ccd483fd3f7ddc"}
Sep 09 13:39:08.607874 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:08.607762    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4" event={"ID":"76d4cffb-c769-4dfe-b2ab-63e57691feb8","Type":"ContainerStarted","Data":"59fef9903572a4bf2de9119affdf94f770e5ac495c7816e050335ebdb05947b3"}
Sep 09 13:39:08.624855 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:08.624783    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7949-809753" podStartSLOduration=1.453508787 podStartE2EDuration="3.624767677s" totalImagesPullingTime="2.17125889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:39:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:39:08.622528108 +0000 UTC m=+781.982377938" watchObservedRunningTime="2026-09-09 13:39:08.624767677 +0000 UTC m=+781.984617503"
Sep 09 13:39:08.659583 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:08.659527    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4" podStartSLOduration=1.6595121499999999 podStartE2EDuration="1.65951215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:39:08.659051417 +0000 UTC m=+782.018901254" watchObservedRunningTime="2026-09-09 13:39:08.65951215 +0000 UTC m=+782.019361977"
Sep 09 13:39:08.905431 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:08.905340    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:09.371629 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:09.371591    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7949-809753"]
Sep 09 13:39:09.375887 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:09.375861    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7949-809753"]
Sep 09 13:39:09.620915 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:39:09.620655    2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7949-809753" container="test-curl-7949-809753"
Sep 09 13:39:09.914386 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:09.914348    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:10.624726 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:10.624662    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7949-809753" podUID="c08cdbbd-d96b-4003-85bd-80811312a24d" containerName="test-curl-7949-809753" containerID="cri-o://088642f72da074db1a3589779359c838f76d423b2464ddf523381a7c822070ea" gracePeriod=2
Sep 09 13:39:10.625231 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:10.624806    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:10.625912 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:10.625892    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:10.627222 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:39:10.627201    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7949-809753\" is forbidden: User \"system:node:ip-10-0-129-84.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-84.ec2.internal' and this object" podUID="c08cdbbd-d96b-4003-85bd-80811312a24d" pod="opendatahub/test-curl-7949-809753"
Sep 09 13:39:10.675458 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:39:10.675414    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7949-809753\" is forbidden: User \"system:node:ip-10-0-129-84.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-84.ec2.internal' and this object" podUID="c08cdbbd-d96b-4003-85bd-80811312a24d" pod="opendatahub/test-curl-7949-809753"
Sep 09 13:39:10.800070 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:10.800031    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"]
Sep 09 13:39:10.800786 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:10.800761    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c08cdbbd-d96b-4003-85bd-80811312a24d" containerName="test-curl-7949-809753"
Sep 09 13:39:10.800944 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:10.800931    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c08cdbbd-d96b-4003-85bd-80811312a24d" containerName="test-curl-7949-809753"
Sep 09 13:39:10.801144 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:10.801130    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c08cdbbd-d96b-4003-85bd-80811312a24d" containerName="test-curl-7949-809753"
Sep 09 13:39:10.817370 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:10.817313    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:10.820207 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:10.820181    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-71c181-openshift-default-dockercfg-qhlf6\""
Sep 09 13:39:10.825830 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:10.825780    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"]
Sep 09 13:39:10.852858 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:39:10.852798    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7949-809753\" is forbidden: User \"system:node:ip-10-0-129-84.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-84.ec2.internal' and this object" podUID="c08cdbbd-d96b-4003-85bd-80811312a24d" pod="opendatahub/test-curl-7949-809753"
Sep 09 13:39:10.934394 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:10.934302    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-istio-data\") pod \"e2e-autores-71c181-openshift-default-5b75999664-9s7bz\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") " pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:10.934394 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:10.934362    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-credential-socket\") pod \"e2e-autores-71c181-openshift-default-5b75999664-9s7bz\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") " pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:10.934575 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:10.934404    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-workload-certs\") pod \"e2e-autores-71c181-openshift-default-5b75999664-9s7bz\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") " pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:10.934575 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:10.934519    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-istio-envoy\") pod \"e2e-autores-71c181-openshift-default-5b75999664-9s7bz\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") " pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:10.934575 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:10.934538    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5398979-4319-4212-a4f7-83436cc58135-istiod-ca-cert\") pod \"e2e-autores-71c181-openshift-default-5b75999664-9s7bz\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") " pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:10.934575 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:10.934553    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-workload-socket\") pod \"e2e-autores-71c181-openshift-default-5b75999664-9s7bz\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") " pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:10.934699 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:10.934581    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5398979-4319-4212-a4f7-83436cc58135-istio-token\") pod \"e2e-autores-71c181-openshift-default-5b75999664-9s7bz\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") " pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:10.934699 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:10.934597    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc9cs\" (UniqueName: \"kubernetes.io/projected/d5398979-4319-4212-a4f7-83436cc58135-kube-api-access-wc9cs\") pod \"e2e-autores-71c181-openshift-default-5b75999664-9s7bz\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") " pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:10.934699 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:10.934653    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5398979-4319-4212-a4f7-83436cc58135-istio-podinfo\") pod \"e2e-autores-71c181-openshift-default-5b75999664-9s7bz\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") " pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:11.035654 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:11.035613    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-istio-envoy\") pod \"e2e-autores-71c181-openshift-default-5b75999664-9s7bz\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") " pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:11.035654 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:11.035661    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5398979-4319-4212-a4f7-83436cc58135-istiod-ca-cert\") pod \"e2e-autores-71c181-openshift-default-5b75999664-9s7bz\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") " pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:11.035948 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:11.035690    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-workload-socket\") pod \"e2e-autores-71c181-openshift-default-5b75999664-9s7bz\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") " pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:11.035948 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:11.035725    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5398979-4319-4212-a4f7-83436cc58135-istio-token\") pod \"e2e-autores-71c181-openshift-default-5b75999664-9s7bz\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") " pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:11.035948 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:11.035750    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc9cs\" (UniqueName: \"kubernetes.io/projected/d5398979-4319-4212-a4f7-83436cc58135-kube-api-access-wc9cs\") pod \"e2e-autores-71c181-openshift-default-5b75999664-9s7bz\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") " pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:11.035948 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:11.035787    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5398979-4319-4212-a4f7-83436cc58135-istio-podinfo\") pod \"e2e-autores-71c181-openshift-default-5b75999664-9s7bz\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") " pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:11.036152 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:11.035951    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-istio-data\") pod \"e2e-autores-71c181-openshift-default-5b75999664-9s7bz\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") " pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:11.036152 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:11.036015    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-credential-socket\") pod \"e2e-autores-71c181-openshift-default-5b75999664-9s7bz\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") " pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:11.036152 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:11.036039    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-workload-certs\") pod \"e2e-autores-71c181-openshift-default-5b75999664-9s7bz\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") " pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:11.036428 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:11.036402    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-istio-data\") pod \"e2e-autores-71c181-openshift-default-5b75999664-9s7bz\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") " pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:11.036550 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:11.036428    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-workload-socket\") pod \"e2e-autores-71c181-openshift-default-5b75999664-9s7bz\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") " pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:11.036550 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:11.036481    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-credential-socket\") pod \"e2e-autores-71c181-openshift-default-5b75999664-9s7bz\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") " pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:11.036550 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:11.036522    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5398979-4319-4212-a4f7-83436cc58135-istiod-ca-cert\") pod \"e2e-autores-71c181-openshift-default-5b75999664-9s7bz\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") " pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:11.036737 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:11.036721    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-workload-certs\") pod \"e2e-autores-71c181-openshift-default-5b75999664-9s7bz\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") " pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:11.038136 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:11.038108    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-istio-envoy\") pod \"e2e-autores-71c181-openshift-default-5b75999664-9s7bz\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") " pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:11.038372 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:11.038354    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5398979-4319-4212-a4f7-83436cc58135-istio-podinfo\") pod \"e2e-autores-71c181-openshift-default-5b75999664-9s7bz\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") " pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:11.067569 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:11.067540    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc9cs\" (UniqueName: \"kubernetes.io/projected/d5398979-4319-4212-a4f7-83436cc58135-kube-api-access-wc9cs\") pod \"e2e-autores-71c181-openshift-default-5b75999664-9s7bz\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") " pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:11.068951 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:11.068917    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5398979-4319-4212-a4f7-83436cc58135-istio-token\") pod \"e2e-autores-71c181-openshift-default-5b75999664-9s7bz\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") " pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:11.130061 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:11.130022    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:11.427673 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:11.427645    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"]
Sep 09 13:39:11.429675 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:39:11.429644    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5398979_4319_4212_a4f7_83436cc58135.slice/crio-61a209743bc4c909fa0fb195e90fb1506ff54701f86c036dde80b238e4b55232 WatchSource:0}: Error finding container 61a209743bc4c909fa0fb195e90fb1506ff54701f86c036dde80b238e4b55232: Status 404 returned error can't find the container with id 61a209743bc4c909fa0fb195e90fb1506ff54701f86c036dde80b238e4b55232
Sep 09 13:39:11.431804 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:11.431771    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:39:11.431893 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:11.431878    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:39:11.431933 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:11.431922    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:39:11.632291 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:11.632183    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz" event={"ID":"d5398979-4319-4212-a4f7-83436cc58135","Type":"ContainerStarted","Data":"5cdb4203591f267ab9895e8e2dd48a020fdb1f4c1997b09bdd8e7581edff3451"}
Sep 09 13:39:11.632291 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:11.632251    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz" event={"ID":"d5398979-4319-4212-a4f7-83436cc58135","Type":"ContainerStarted","Data":"61a209743bc4c909fa0fb195e90fb1506ff54701f86c036dde80b238e4b55232"}
Sep 09 13:39:11.735085 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:11.734944    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz" podStartSLOduration=1.734919944 podStartE2EDuration="1.734919944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:39:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:39:11.729521992 +0000 UTC m=+785.089371818" watchObservedRunningTime="2026-09-09 13:39:11.734919944 +0000 UTC m=+785.094769772"
Sep 09 13:39:12.130472 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:12.130432    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:12.135765 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:12.135737    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:12.638069 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:12.638042    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:12.639014 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:12.638993    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:12.861203 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:12.861172    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7949-809753"
Sep 09 13:39:12.955097 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:12.955002    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvshw\" (UniqueName: \"kubernetes.io/projected/c08cdbbd-d96b-4003-85bd-80811312a24d-kube-api-access-fvshw\") pod \"c08cdbbd-d96b-4003-85bd-80811312a24d\" (UID: \"c08cdbbd-d96b-4003-85bd-80811312a24d\") "
Sep 09 13:39:12.957627 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:12.957289    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08cdbbd-d96b-4003-85bd-80811312a24d-kube-api-access-fvshw" (OuterVolumeSpecName: "kube-api-access-fvshw") pod "c08cdbbd-d96b-4003-85bd-80811312a24d" (UID: "c08cdbbd-d96b-4003-85bd-80811312a24d"). InnerVolumeSpecName "kube-api-access-fvshw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:39:13.056643 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:13.056598    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvshw\" (UniqueName: \"kubernetes.io/projected/c08cdbbd-d96b-4003-85bd-80811312a24d-kube-api-access-fvshw\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:39:13.150288 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:13.150253    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c08cdbbd-d96b-4003-85bd-80811312a24d" path="/var/lib/kubelet/pods/c08cdbbd-d96b-4003-85bd-80811312a24d/volumes"
Sep 09 13:39:13.643422 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:13.643385    2566 generic.go:356] "Generic (PLEG): container finished" podID="c08cdbbd-d96b-4003-85bd-80811312a24d" containerID="088642f72da074db1a3589779359c838f76d423b2464ddf523381a7c822070ea" exitCode=137
Sep 09 13:39:13.643906 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:13.643439    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7949-809753"
Sep 09 13:39:13.643906 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:13.643492    2566 scope.go:117] "RemoveContainer" containerID="088642f72da074db1a3589779359c838f76d423b2464ddf523381a7c822070ea"
Sep 09 13:39:13.651953 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:13.651928    2566 scope.go:117] "RemoveContainer" containerID="088642f72da074db1a3589779359c838f76d423b2464ddf523381a7c822070ea"
Sep 09 13:39:13.652242 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:39:13.652217    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"088642f72da074db1a3589779359c838f76d423b2464ddf523381a7c822070ea\": container with ID starting with 088642f72da074db1a3589779359c838f76d423b2464ddf523381a7c822070ea not found: ID does not exist" containerID="088642f72da074db1a3589779359c838f76d423b2464ddf523381a7c822070ea"
Sep 09 13:39:13.652354 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:13.652248    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"088642f72da074db1a3589779359c838f76d423b2464ddf523381a7c822070ea"} err="failed to get container status \"088642f72da074db1a3589779359c838f76d423b2464ddf523381a7c822070ea\": rpc error: code = NotFound desc = could not find container \"088642f72da074db1a3589779359c838f76d423b2464ddf523381a7c822070ea\": container with ID starting with 088642f72da074db1a3589779359c838f76d423b2464ddf523381a7c822070ea not found: ID does not exist"
Sep 09 13:39:16.463751 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:16.463701    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-71c181-9448f866c-4p6s9"]
Sep 09 13:39:16.470083 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:16.470048    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-71c181-9448f866c-4p6s9"
Sep 09 13:39:16.477415 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:16.477253    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-71c181\""
Sep 09 13:39:16.477513 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:16.477462    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-71c181-dockercfg-kqrt4\""
Sep 09 13:39:16.489766 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:16.489741    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-71c181-9448f866c-4p6s9"]
Sep 09 13:39:16.589985 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:16.589952    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfwlp\" (UniqueName: \"kubernetes.io/projected/6592a366-304b-4e42-bd75-3f26fc5901c5-kube-api-access-zfwlp\") pod \"payload-processing-e2e-autores-71c181-9448f866c-4p6s9\" (UID: \"6592a366-304b-4e42-bd75-3f26fc5901c5\") " pod="openshift-ingress/payload-processing-e2e-autores-71c181-9448f866c-4p6s9"
Sep 09 13:39:16.590157 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:16.590020    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6592a366-304b-4e42-bd75-3f26fc5901c5-plugins-config-volume\") pod \"payload-processing-e2e-autores-71c181-9448f866c-4p6s9\" (UID: \"6592a366-304b-4e42-bd75-3f26fc5901c5\") " pod="openshift-ingress/payload-processing-e2e-autores-71c181-9448f866c-4p6s9"
Sep 09 13:39:16.691078 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:16.691021    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfwlp\" (UniqueName: \"kubernetes.io/projected/6592a366-304b-4e42-bd75-3f26fc5901c5-kube-api-access-zfwlp\") pod \"payload-processing-e2e-autores-71c181-9448f866c-4p6s9\" (UID: \"6592a366-304b-4e42-bd75-3f26fc5901c5\") " pod="openshift-ingress/payload-processing-e2e-autores-71c181-9448f866c-4p6s9"
Sep 09 13:39:16.691235 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:16.691097    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6592a366-304b-4e42-bd75-3f26fc5901c5-plugins-config-volume\") pod \"payload-processing-e2e-autores-71c181-9448f866c-4p6s9\" (UID: \"6592a366-304b-4e42-bd75-3f26fc5901c5\") " pod="openshift-ingress/payload-processing-e2e-autores-71c181-9448f866c-4p6s9"
Sep 09 13:39:16.691771 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:16.691752    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6592a366-304b-4e42-bd75-3f26fc5901c5-plugins-config-volume\") pod \"payload-processing-e2e-autores-71c181-9448f866c-4p6s9\" (UID: \"6592a366-304b-4e42-bd75-3f26fc5901c5\") " pod="openshift-ingress/payload-processing-e2e-autores-71c181-9448f866c-4p6s9"
Sep 09 13:39:16.705039 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:16.705009    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfwlp\" (UniqueName: \"kubernetes.io/projected/6592a366-304b-4e42-bd75-3f26fc5901c5-kube-api-access-zfwlp\") pod \"payload-processing-e2e-autores-71c181-9448f866c-4p6s9\" (UID: \"6592a366-304b-4e42-bd75-3f26fc5901c5\") " pod="openshift-ingress/payload-processing-e2e-autores-71c181-9448f866c-4p6s9"
Sep 09 13:39:16.781084 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:16.780989    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-71c181-9448f866c-4p6s9"
Sep 09 13:39:17.068603 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:17.068520    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-71c181-9448f866c-4p6s9"]
Sep 09 13:39:17.082698 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:39:17.082656    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6592a366_304b_4e42_bd75_3f26fc5901c5.slice/crio-0c4a519189fcfb11f7a3a7035aa6130befbf1905955dfaf1fc7b133dfa2833fe WatchSource:0}: Error finding container 0c4a519189fcfb11f7a3a7035aa6130befbf1905955dfaf1fc7b133dfa2833fe: Status 404 returned error can't find the container with id 0c4a519189fcfb11f7a3a7035aa6130befbf1905955dfaf1fc7b133dfa2833fe
Sep 09 13:39:17.668016 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:17.667974    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-71c181-9448f866c-4p6s9" event={"ID":"6592a366-304b-4e42-bd75-3f26fc5901c5","Type":"ContainerStarted","Data":"0c4a519189fcfb11f7a3a7035aa6130befbf1905955dfaf1fc7b133dfa2833fe"}
Sep 09 13:39:18.357988 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:18.356516    2566 patch_prober.go:28] interesting pod/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": read tcp 10.132.0.2:47530->10.132.0.31:15021: read: connection reset by peer" start-of-body=
Sep 09 13:39:18.357988 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:18.356594    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh" podUID="c8d6623e-e731-4ace-8f3b-f44db09987c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": read tcp 10.132.0.2:47530->10.132.0.31:15021: read: connection reset by peer"
Sep 09 13:39:19.682010 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:19.681977    2566 generic.go:356] "Generic (PLEG): container finished" podID="c8d6623e-e731-4ace-8f3b-f44db09987c8" containerID="49d2710ad58674557cfd0a8c1cfe7569b379b0b15112bad6ea0939d197f5326c" exitCode=0
Sep 09 13:39:19.682452 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:19.682055    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh" event={"ID":"c8d6623e-e731-4ace-8f3b-f44db09987c8","Type":"ContainerDied","Data":"49d2710ad58674557cfd0a8c1cfe7569b379b0b15112bad6ea0939d197f5326c"}
Sep 09 13:39:19.682524 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:19.682505    2566 scope.go:117] "RemoveContainer" containerID="49d2710ad58674557cfd0a8c1cfe7569b379b0b15112bad6ea0939d197f5326c"
Sep 09 13:39:19.683793 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:19.683758    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:39:19.683923 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:19.683785    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-71c181-9448f866c-4p6s9" event={"ID":"6592a366-304b-4e42-bd75-3f26fc5901c5","Type":"ContainerStarted","Data":"3d1705930983350d8534a2079f833170ade64d0ba10e2f2a95a8734591763c24"}
Sep 09 13:39:19.683923 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:19.683864    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:39:19.683923 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:19.683903    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:39:19.684023 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:19.683946    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-71c181-9448f866c-4p6s9"
Sep 09 13:39:19.747203 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:19.747159    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-71c181-9448f866c-4p6s9" podStartSLOduration=1.54636854 podStartE2EDuration="3.747142182s" totalImagesPullingTime="2.200773642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:39:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:39:19.746909034 +0000 UTC m=+793.106758860" watchObservedRunningTime="2026-09-09 13:39:19.747142182 +0000 UTC m=+793.106992005"
Sep 09 13:39:20.691472 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:20.691427    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh" event={"ID":"c8d6623e-e731-4ace-8f3b-f44db09987c8","Type":"ContainerStarted","Data":"5ae16fd07f956588921fdf2088943b7dff60e7c04f5d564ecda0c2a81e5b2376"}
Sep 09 13:39:20.987245 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:20.987158    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:39:20.987444 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:20.987422    2566 patch_prober.go:28] interesting pod/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 09 13:39:20.987507 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:20.987487    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh" podUID="c8d6623e-e731-4ace-8f3b-f44db09987c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 09 13:39:21.987346 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:21.987296    2566 patch_prober.go:28] interesting pod/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 09 13:39:21.987882 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:21.987374    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh" podUID="c8d6623e-e731-4ace-8f3b-f44db09987c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 09 13:39:22.986863 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:22.986800    2566 patch_prober.go:28] interesting pod/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 09 13:39:22.987108 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:22.987083    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh" podUID="c8d6623e-e731-4ace-8f3b-f44db09987c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 09 13:39:23.987525 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:23.987482    2566 patch_prober.go:28] interesting pod/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 09 13:39:23.987960 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:23.987546    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh" podUID="c8d6623e-e731-4ace-8f3b-f44db09987c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 09 13:39:24.654477 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:24.654437    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"]
Sep 09 13:39:24.662989 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:24.662955    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"
Sep 09 13:39:24.668243 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:24.668211    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-bcc5a97b-kserve-self-signed-certs\""
Sep 09 13:39:24.668485 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:24.668453    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"]
Sep 09 13:39:24.779578 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:24.779543    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-tmp-dir\") pod \"e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq\" (UID: \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\") " pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"
Sep 09 13:39:24.779795 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:24.779599    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-tls-certs\") pod \"e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq\" (UID: \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\") " pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"
Sep 09 13:39:24.779795 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:24.779626    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-dshm\") pod \"e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq\" (UID: \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\") " pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"
Sep 09 13:39:24.779795 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:24.779668    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-model-cache\") pod \"e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq\" (UID: \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\") " pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"
Sep 09 13:39:24.779795 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:24.779691    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-home\") pod \"e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq\" (UID: \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\") " pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"
Sep 09 13:39:24.779795 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:24.779718    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtbx9\" (UniqueName: \"kubernetes.io/projected/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-kube-api-access-mtbx9\") pod \"e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq\" (UID: \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\") " pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"
Sep 09 13:39:24.881523 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:24.881473    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-tls-certs\") pod \"e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq\" (UID: \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\") " pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"
Sep 09 13:39:24.881719 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:24.881540    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-dshm\") pod \"e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq\" (UID: \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\") " pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"
Sep 09 13:39:24.881719 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:24.881592    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-model-cache\") pod \"e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq\" (UID: \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\") " pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"
Sep 09 13:39:24.881719 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:24.881630    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-home\") pod \"e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq\" (UID: \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\") " pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"
Sep 09 13:39:24.881719 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:24.881657    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtbx9\" (UniqueName: \"kubernetes.io/projected/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-kube-api-access-mtbx9\") pod \"e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq\" (UID: \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\") " pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"
Sep 09 13:39:24.882021 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:24.881803    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-tmp-dir\") pod \"e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq\" (UID: \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\") " pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"
Sep 09 13:39:24.882542 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:24.882512    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-tmp-dir\") pod \"e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq\" (UID: \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\") " pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"
Sep 09 13:39:24.882695 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:24.882639    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-model-cache\") pod \"e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq\" (UID: \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\") " pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"
Sep 09 13:39:24.882899 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:24.882880    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-home\") pod \"e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq\" (UID: \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\") " pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"
Sep 09 13:39:24.884221 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:24.884200    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-dshm\") pod \"e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq\" (UID: \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\") " pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"
Sep 09 13:39:24.884399 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:24.884379    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-tls-certs\") pod \"e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq\" (UID: \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\") " pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"
Sep 09 13:39:24.899265 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:24.899230    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtbx9\" (UniqueName: \"kubernetes.io/projected/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-kube-api-access-mtbx9\") pod \"e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq\" (UID: \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\") " pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"
Sep 09 13:39:24.976334 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:24.976230    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"
Sep 09 13:39:24.987499 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:24.987469    2566 patch_prober.go:28] interesting pod/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 09 13:39:24.987640 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:24.987526    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh" podUID="c8d6623e-e731-4ace-8f3b-f44db09987c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 09 13:39:25.327677 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:25.326318    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"]
Sep 09 13:39:25.354202 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:39:25.354128    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8f84e7e_bf8b_403a_9eab_92ae34fb9357.slice/crio-6611bd5ce5054a7d2f230c527b42ef220be01f36e673144bf3dd75a41e1be9a0 WatchSource:0}: Error finding container 6611bd5ce5054a7d2f230c527b42ef220be01f36e673144bf3dd75a41e1be9a0: Status 404 returned error can't find the container with id 6611bd5ce5054a7d2f230c527b42ef220be01f36e673144bf3dd75a41e1be9a0
Sep 09 13:39:25.735240 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:25.734284    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq" event={"ID":"e8f84e7e-bf8b-403a-9eab-92ae34fb9357","Type":"ContainerStarted","Data":"5152b5a30d3f2c344d55ac8fbfbe79aea02309d4cab2d979a644e880028bf1a1"}
Sep 09 13:39:25.735240 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:25.734341    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq" event={"ID":"e8f84e7e-bf8b-403a-9eab-92ae34fb9357","Type":"ContainerStarted","Data":"6611bd5ce5054a7d2f230c527b42ef220be01f36e673144bf3dd75a41e1be9a0"}
Sep 09 13:39:25.987531 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:25.987447    2566 patch_prober.go:28] interesting pod/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 09 13:39:25.987531 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:25.987509    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh" podUID="c8d6623e-e731-4ace-8f3b-f44db09987c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 09 13:39:26.992448 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:26.992413    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:39:26.993640 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:26.993613    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:39:26.998324 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:26.998291    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:39:27.020096 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:27.020026    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq" podStartSLOduration=3.019995273 podStartE2EDuration="3.019995273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:39:25.759997592 +0000 UTC m=+799.119847412" watchObservedRunningTime="2026-09-09 13:39:27.019995273 +0000 UTC m=+800.379845101"
Sep 09 13:39:28.121672 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.121632    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"]
Sep 09 13:39:28.129922 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.129887    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.133029 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.133002    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-093e38-openshift-default-dockercfg-ss5bm\""
Sep 09 13:39:28.152686 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.152651    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"]
Sep 09 13:39:28.223368 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.223319    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-istio-data\") pod \"e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") " pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.223368 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.223387    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ff22c80-e444-46f0-97d8-0cb8247f2231-istiod-ca-cert\") pod \"e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") " pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.223632 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.223439    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-workload-certs\") pod \"e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") " pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.223632 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.223461    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ff22c80-e444-46f0-97d8-0cb8247f2231-istio-token\") pod \"e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") " pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.223632 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.223532    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ff22c80-e444-46f0-97d8-0cb8247f2231-istio-podinfo\") pod \"e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") " pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.223790 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.223656    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-istio-envoy\") pod \"e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") " pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.223790 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.223711    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbsjd\" (UniqueName: \"kubernetes.io/projected/9ff22c80-e444-46f0-97d8-0cb8247f2231-kube-api-access-kbsjd\") pod \"e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") " pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.223790 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.223743    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-credential-socket\") pod \"e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") " pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.223790 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.223774    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-workload-socket\") pod \"e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") " pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.324847 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.324761    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-credential-socket\") pod \"e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") " pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.324847 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.324832    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-workload-socket\") pod \"e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") " pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.325105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.324875    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-istio-data\") pod \"e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") " pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.325105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.324919    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ff22c80-e444-46f0-97d8-0cb8247f2231-istiod-ca-cert\") pod \"e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") " pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.325105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.324960    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-workload-certs\") pod \"e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") " pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.325105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.324983    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ff22c80-e444-46f0-97d8-0cb8247f2231-istio-token\") pod \"e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") " pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.325105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.325015    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ff22c80-e444-46f0-97d8-0cb8247f2231-istio-podinfo\") pod \"e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") " pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.325342 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.325118    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-istio-envoy\") pod \"e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") " pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.325342 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.325163    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbsjd\" (UniqueName: \"kubernetes.io/projected/9ff22c80-e444-46f0-97d8-0cb8247f2231-kube-api-access-kbsjd\") pod \"e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") " pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.325342 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.325270    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-credential-socket\") pod \"e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") " pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.325477 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.325334    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-workload-socket\") pod \"e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") " pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.325525 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.325494    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-workload-certs\") pod \"e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") " pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.325742 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.325721    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-istio-data\") pod \"e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") " pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.326745 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.326718    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ff22c80-e444-46f0-97d8-0cb8247f2231-istiod-ca-cert\") pod \"e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") " pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.327984 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.327956    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-istio-envoy\") pod \"e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") " pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.328509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.328486    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ff22c80-e444-46f0-97d8-0cb8247f2231-istio-podinfo\") pod \"e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") " pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.349190 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.349151    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ff22c80-e444-46f0-97d8-0cb8247f2231-istio-token\") pod \"e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") " pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.353043 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.352995    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbsjd\" (UniqueName: \"kubernetes.io/projected/9ff22c80-e444-46f0-97d8-0cb8247f2231-kube-api-access-kbsjd\") pod \"e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") " pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.442116 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.441761    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:28.744873 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.744743    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"]
Sep 09 13:39:28.750633 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:39:28.750587    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ff22c80_e444_46f0_97d8_0cb8247f2231.slice/crio-67890444f6c0d343ea3d17077cb1ca106bb5f4ab05b438c9e1830e6db94d4646 WatchSource:0}: Error finding container 67890444f6c0d343ea3d17077cb1ca106bb5f4ab05b438c9e1830e6db94d4646: Status 404 returned error can't find the container with id 67890444f6c0d343ea3d17077cb1ca106bb5f4ab05b438c9e1830e6db94d4646
Sep 09 13:39:28.753434 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.753385    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:39:28.753555 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.753469    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:39:28.753555 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:28.753500    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:39:29.767863 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:29.767751    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl" event={"ID":"9ff22c80-e444-46f0-97d8-0cb8247f2231","Type":"ContainerStarted","Data":"a05e356d07ca65e07301901373c80c84624bd06460b0faaac10ab90d9962f3fe"}
Sep 09 13:39:29.767863 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:29.767833    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl" event={"ID":"9ff22c80-e444-46f0-97d8-0cb8247f2231","Type":"ContainerStarted","Data":"67890444f6c0d343ea3d17077cb1ca106bb5f4ab05b438c9e1830e6db94d4646"}
Sep 09 13:39:29.802583 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:29.802510    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl" podStartSLOduration=1.802484952 podStartE2EDuration="1.802484952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:39:29.798726052 +0000 UTC m=+803.158575899" watchObservedRunningTime="2026-09-09 13:39:29.802484952 +0000 UTC m=+803.162334779"
Sep 09 13:39:30.442495 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:30.442459    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:30.447890 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:30.447864    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:30.693966 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:30.693865    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-71c181-9448f866c-4p6s9"
Sep 09 13:39:30.773990 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:30.773956    2566 generic.go:356] "Generic (PLEG): container finished" podID="76d4cffb-c769-4dfe-b2ab-63e57691feb8" containerID="fdb2c215e9ddf0c692e0b9b9c0ca6f81087d8ea6ed2ed6ccf7ccd483fd3f7ddc" exitCode=0
Sep 09 13:39:30.774466 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:30.774036    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4" event={"ID":"76d4cffb-c769-4dfe-b2ab-63e57691feb8","Type":"ContainerDied","Data":"fdb2c215e9ddf0c692e0b9b9c0ca6f81087d8ea6ed2ed6ccf7ccd483fd3f7ddc"}
Sep 09 13:39:30.774466 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:30.774416    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:30.774787 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:30.774660    2566 scope.go:117] "RemoveContainer" containerID="fdb2c215e9ddf0c692e0b9b9c0ca6f81087d8ea6ed2ed6ccf7ccd483fd3f7ddc"
Sep 09 13:39:30.775689 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:30.775572    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:30.776067 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:30.776033    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:39:30.776154 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:30.776134    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:39:30.776208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:30.776192    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:39:31.780468 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:31.780436    2566 generic.go:356] "Generic (PLEG): container finished" podID="d5398979-4319-4212-a4f7-83436cc58135" containerID="5cdb4203591f267ab9895e8e2dd48a020fdb1f4c1997b09bdd8e7581edff3451" exitCode=0
Sep 09 13:39:31.781012 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:31.780505    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz" event={"ID":"d5398979-4319-4212-a4f7-83436cc58135","Type":"ContainerDied","Data":"5cdb4203591f267ab9895e8e2dd48a020fdb1f4c1997b09bdd8e7581edff3451"}
Sep 09 13:39:31.781078 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:31.781057    2566 scope.go:117] "RemoveContainer" containerID="5cdb4203591f267ab9895e8e2dd48a020fdb1f4c1997b09bdd8e7581edff3451"
Sep 09 13:39:31.782272 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:31.782185    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:39:31.782358 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:31.782315    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:39:31.782358 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:31.782350    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:39:31.782803 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:31.782781    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4" event={"ID":"76d4cffb-c769-4dfe-b2ab-63e57691feb8","Type":"ContainerStarted","Data":"571a29465882bc409a4be4b9134aa2efc0a29d4c7b0d83d38ad729e443f14333"}
Sep 09 13:39:31.905515 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:31.905051    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:31.905515 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:31.905190    2566 patch_prober.go:28] interesting pod/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 09 13:39:31.905515 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:31.905238    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4" podUID="76d4cffb-c769-4dfe-b2ab-63e57691feb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 09 13:39:32.792713 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:32.791328    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz" event={"ID":"d5398979-4319-4212-a4f7-83436cc58135","Type":"ContainerStarted","Data":"e373e1097f289368529b6365ca4a242c9fa3b9cec7b399bb59388e6cafa31cc4"}
Sep 09 13:39:32.905754 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:32.905697    2566 patch_prober.go:28] interesting pod/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 09 13:39:32.905988 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:32.905788    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4" podUID="76d4cffb-c769-4dfe-b2ab-63e57691feb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 09 13:39:33.131297 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:33.131249    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:33.131493 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:33.131391    2566 patch_prober.go:28] interesting pod/e2e-autores-71c181-openshift-default-5b75999664-9s7bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Sep 09 13:39:33.131493 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:33.131433    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz" podUID="d5398979-4319-4212-a4f7-83436cc58135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Sep 09 13:39:33.905646 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:33.905604    2566 patch_prober.go:28] interesting pod/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 09 13:39:33.906235 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:33.906199    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4" podUID="76d4cffb-c769-4dfe-b2ab-63e57691feb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 09 13:39:34.131304 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:34.131261    2566 patch_prober.go:28] interesting pod/e2e-autores-71c181-openshift-default-5b75999664-9s7bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Sep 09 13:39:34.131601 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:34.131568    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz" podUID="d5398979-4319-4212-a4f7-83436cc58135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Sep 09 13:39:34.905235 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:34.905191    2566 patch_prober.go:28] interesting pod/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 09 13:39:34.905551 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:34.905517    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4" podUID="76d4cffb-c769-4dfe-b2ab-63e57691feb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 09 13:39:34.976785 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:34.976753    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"
Sep 09 13:39:34.977200 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:34.976839    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"
Sep 09 13:39:34.990836 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:34.990781    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"
Sep 09 13:39:35.130577 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:35.130533    2566 patch_prober.go:28] interesting pod/e2e-autores-71c181-openshift-default-5b75999664-9s7bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Sep 09 13:39:35.130894 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:35.130857    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz" podUID="d5398979-4319-4212-a4f7-83436cc58135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Sep 09 13:39:35.855225 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:35.855186    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"
Sep 09 13:39:35.905362 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:35.905318    2566 patch_prober.go:28] interesting pod/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 09 13:39:35.905570 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:35.905389    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4" podUID="76d4cffb-c769-4dfe-b2ab-63e57691feb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 09 13:39:36.131380 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:36.131268    2566 patch_prober.go:28] interesting pod/e2e-autores-71c181-openshift-default-5b75999664-9s7bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Sep 09 13:39:36.131380 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:36.131348    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz" podUID="d5398979-4319-4212-a4f7-83436cc58135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Sep 09 13:39:36.905422 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:36.905375    2566 patch_prober.go:28] interesting pod/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 09 13:39:36.905766 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:36.905728    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4" podUID="76d4cffb-c769-4dfe-b2ab-63e57691feb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 09 13:39:37.131380 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:37.131338    2566 patch_prober.go:28] interesting pod/e2e-autores-71c181-openshift-default-5b75999664-9s7bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Sep 09 13:39:37.131562 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:37.131414    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz" podUID="d5398979-4319-4212-a4f7-83436cc58135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Sep 09 13:39:37.914386 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:37.914345    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:37.914804 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:37.914775    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:37.915678 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:37.915655    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:38.130878 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:38.130805    2566 patch_prober.go:28] interesting pod/e2e-autores-71c181-openshift-default-5b75999664-9s7bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Sep 09 13:39:38.131052 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:38.130913    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz" podUID="d5398979-4319-4212-a4f7-83436cc58135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Sep 09 13:39:39.135605 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:39.135574    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:39.136035 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:39.135920    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:39.136671 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:39.136655    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:39:40.543918 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:40.543882    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a649773d-9lpdb"]
Sep 09 13:39:40.546245 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:40.546222    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a649773d-9lpdb"
Sep 09 13:39:40.548677 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:40.548655    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 13:39:40.548834 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:40.548726    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lh487\""
Sep 09 13:39:40.556758 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:40.556729    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a649773d-9lpdb"]
Sep 09 13:39:40.561917 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:40.561888    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86073ed5-d744-43c5-8ff3-a330f617632a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a649773d-9lpdb\" (UID: \"86073ed5-d744-43c5-8ff3-a330f617632a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a649773d-9lpdb"
Sep 09 13:39:40.663083 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:40.663046    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86073ed5-d744-43c5-8ff3-a330f617632a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a649773d-9lpdb\" (UID: \"86073ed5-d744-43c5-8ff3-a330f617632a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a649773d-9lpdb"
Sep 09 13:39:40.663686 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:40.663665    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86073ed5-d744-43c5-8ff3-a330f617632a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a649773d-9lpdb\" (UID: \"86073ed5-d744-43c5-8ff3-a330f617632a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a649773d-9lpdb"
Sep 09 13:39:40.857023 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:40.856990    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a649773d-9lpdb"
Sep 09 13:39:41.125524 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:41.125444    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a649773d-9lpdb"]
Sep 09 13:39:41.127911 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:39:41.127884    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86073ed5_d744_43c5_8ff3_a330f617632a.slice/crio-242e0b29b7f30a2ca576fc62d609e26d64a60bcf2c0fea2c0eb97eb2093cd880 WatchSource:0}: Error finding container 242e0b29b7f30a2ca576fc62d609e26d64a60bcf2c0fea2c0eb97eb2093cd880: Status 404 returned error can't find the container with id 242e0b29b7f30a2ca576fc62d609e26d64a60bcf2c0fea2c0eb97eb2093cd880
Sep 09 13:39:41.854029 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:41.853990    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a649773d-9lpdb" event={"ID":"86073ed5-d744-43c5-8ff3-a330f617632a","Type":"ContainerStarted","Data":"242e0b29b7f30a2ca576fc62d609e26d64a60bcf2c0fea2c0eb97eb2093cd880"}
Sep 09 13:39:43.867946 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:43.867911    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a649773d-9lpdb" event={"ID":"86073ed5-d744-43c5-8ff3-a330f617632a","Type":"ContainerStarted","Data":"11c5ebf99c1e1306c635501ccdaee41e32628be8c06f6194a06aabab7e66ac4e"}
Sep 09 13:39:43.884959 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:43.884911    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a649773d-9lpdb" podStartSLOduration=1.594584488 podStartE2EDuration="3.884894577s" totalImagesPullingTime="2.290310089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:39:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:39:43.88371284 +0000 UTC m=+817.243562667" watchObservedRunningTime="2026-09-09 13:39:43.884894577 +0000 UTC m=+817.244744403"
Sep 09 13:39:44.874250 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:44.874214    2566 generic.go:356] "Generic (PLEG): container finished" podID="86073ed5-d744-43c5-8ff3-a330f617632a" containerID="11c5ebf99c1e1306c635501ccdaee41e32628be8c06f6194a06aabab7e66ac4e" exitCode=0
Sep 09 13:39:44.874701 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:44.874298    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a649773d-9lpdb" event={"ID":"86073ed5-d744-43c5-8ff3-a330f617632a","Type":"ContainerDied","Data":"11c5ebf99c1e1306c635501ccdaee41e32628be8c06f6194a06aabab7e66ac4e"}
Sep 09 13:39:46.072230 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:46.072203    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a649773d-9lpdb"
Sep 09 13:39:46.125352 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:46.125318    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86073ed5-d744-43c5-8ff3-a330f617632a-maas-api-service-ca\") pod \"86073ed5-d744-43c5-8ff3-a330f617632a\" (UID: \"86073ed5-d744-43c5-8ff3-a330f617632a\") "
Sep 09 13:39:46.125668 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:46.125642    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86073ed5-d744-43c5-8ff3-a330f617632a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "86073ed5-d744-43c5-8ff3-a330f617632a" (UID: "86073ed5-d744-43c5-8ff3-a330f617632a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:39:46.227006 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:46.226910    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86073ed5-d744-43c5-8ff3-a330f617632a-maas-api-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:39:46.886141 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:46.886111    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a649773d-9lpdb"
Sep 09 13:39:46.886330 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:46.886152    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a649773d-9lpdb" event={"ID":"86073ed5-d744-43c5-8ff3-a330f617632a","Type":"ContainerDied","Data":"242e0b29b7f30a2ca576fc62d609e26d64a60bcf2c0fea2c0eb97eb2093cd880"}
Sep 09 13:39:46.886330 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:46.886200    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="242e0b29b7f30a2ca576fc62d609e26d64a60bcf2c0fea2c0eb97eb2093cd880"
Sep 09 13:39:52.321996 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:52.321953    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a649773d-9lpdb"]
Sep 09 13:39:52.328522 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:52.328486    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a649773d-9lpdb"]
Sep 09 13:39:52.455468 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:52.455431    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"]
Sep 09 13:39:52.456104 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:52.456050    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4" podUID="76d4cffb-c769-4dfe-b2ab-63e57691feb8" containerName="istio-proxy" containerID="cri-o://571a29465882bc409a4be4b9134aa2efc0a29d4c7b0d83d38ad729e443f14333" gracePeriod=30
Sep 09 13:39:52.917386 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:52.917348    2566 patch_prober.go:28] interesting pod/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 13:39:52.917556 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:52.917441    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4" podUID="76d4cffb-c769-4dfe-b2ab-63e57691feb8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 13:39:53.150070 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:53.150042    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86073ed5-d744-43c5-8ff3-a330f617632a" path="/var/lib/kubelet/pods/86073ed5-d744-43c5-8ff3-a330f617632a/volumes"
Sep 09 13:39:53.930461 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:53.930425    2566 generic.go:356] "Generic (PLEG): container finished" podID="9ff22c80-e444-46f0-97d8-0cb8247f2231" containerID="a05e356d07ca65e07301901373c80c84624bd06460b0faaac10ab90d9962f3fe" exitCode=0
Sep 09 13:39:53.930897 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:53.930499    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl" event={"ID":"9ff22c80-e444-46f0-97d8-0cb8247f2231","Type":"ContainerDied","Data":"a05e356d07ca65e07301901373c80c84624bd06460b0faaac10ab90d9962f3fe"}
Sep 09 13:39:53.930951 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:53.930937    2566 scope.go:117] "RemoveContainer" containerID="a05e356d07ca65e07301901373c80c84624bd06460b0faaac10ab90d9962f3fe"
Sep 09 13:39:53.931997 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:53.931967    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:39:53.932079 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:53.932036    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:39:53.932079 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:53.932067    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:39:54.937179 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:54.937146    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl" event={"ID":"9ff22c80-e444-46f0-97d8-0cb8247f2231","Type":"ContainerStarted","Data":"e736a806891a50d3e4d34db269d7a178d696a6161d53fcd22ef7aecd19b94c36"}
Sep 09 13:39:55.442908 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:55.442868    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:39:55.443144 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:55.443121    2566 patch_prober.go:28] interesting pod/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 09 13:39:55.443200 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:55.443173    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl" podUID="9ff22c80-e444-46f0-97d8-0cb8247f2231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 09 13:39:56.442254 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:56.442217    2566 patch_prober.go:28] interesting pod/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 09 13:39:56.442652 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:56.442283    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl" podUID="9ff22c80-e444-46f0-97d8-0cb8247f2231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 09 13:39:57.443081 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.443048    2566 patch_prober.go:28] interesting pod/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 09 13:39:57.443477 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.443131    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl" podUID="9ff22c80-e444-46f0-97d8-0cb8247f2231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 09 13:39:57.542396 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.542356    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g"]
Sep 09 13:39:57.542754 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.542742    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86073ed5-d744-43c5-8ff3-a330f617632a" containerName="revoke-keys"
Sep 09 13:39:57.542797 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.542756    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="86073ed5-d744-43c5-8ff3-a330f617632a" containerName="revoke-keys"
Sep 09 13:39:57.542850 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.542829    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="86073ed5-d744-43c5-8ff3-a330f617632a" containerName="revoke-keys"
Sep 09 13:39:57.546366 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.546345    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g"
Sep 09 13:39:57.548686 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.548660    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-219dfd41\""
Sep 09 13:39:57.557945 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.557913    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g"]
Sep 09 13:39:57.645219 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.645164    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ab9563-5d90-4b5b-bd5b-3d6da3e438a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g\" (UID: \"94ab9563-5d90-4b5b-bd5b-3d6da3e438a3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g"
Sep 09 13:39:57.645415 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.645319    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5s72\" (UniqueName: \"kubernetes.io/projected/94ab9563-5d90-4b5b-bd5b-3d6da3e438a3-kube-api-access-f5s72\") pod \"payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g\" (UID: \"94ab9563-5d90-4b5b-bd5b-3d6da3e438a3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g"
Sep 09 13:39:57.746126 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.746029    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5s72\" (UniqueName: \"kubernetes.io/projected/94ab9563-5d90-4b5b-bd5b-3d6da3e438a3-kube-api-access-f5s72\") pod \"payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g\" (UID: \"94ab9563-5d90-4b5b-bd5b-3d6da3e438a3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g"
Sep 09 13:39:57.746284 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.746162    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ab9563-5d90-4b5b-bd5b-3d6da3e438a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g\" (UID: \"94ab9563-5d90-4b5b-bd5b-3d6da3e438a3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g"
Sep 09 13:39:57.747086 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.747062    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ab9563-5d90-4b5b-bd5b-3d6da3e438a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g\" (UID: \"94ab9563-5d90-4b5b-bd5b-3d6da3e438a3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g"
Sep 09 13:39:57.756299 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.756261    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5s72\" (UniqueName: \"kubernetes.io/projected/94ab9563-5d90-4b5b-bd5b-3d6da3e438a3-kube-api-access-f5s72\") pod \"payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g\" (UID: \"94ab9563-5d90-4b5b-bd5b-3d6da3e438a3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g"
Sep 09 13:39:57.786706 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.786679    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:57.860542 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.860504    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g"
Sep 09 13:39:57.948536 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.948499    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istio-podinfo\") pod \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") "
Sep 09 13:39:57.948673 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.948578    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istio-data\") pod \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") "
Sep 09 13:39:57.948673 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.948660    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-workload-certs\") pod \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") "
Sep 09 13:39:57.948790 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.948719    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-workload-socket\") pod \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") "
Sep 09 13:39:57.948790 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.948777    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-credential-socket\") pod \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") "
Sep 09 13:39:57.948932 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.948824    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istio-envoy\") pod \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") "
Sep 09 13:39:57.948932 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.948864    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4p58z\" (UniqueName: \"kubernetes.io/projected/76d4cffb-c769-4dfe-b2ab-63e57691feb8-kube-api-access-4p58z\") pod \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") "
Sep 09 13:39:57.948932 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.948905    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istiod-ca-cert\") pod \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") "
Sep 09 13:39:57.948932 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.948928    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istio-token\") pod \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\" (UID: \"76d4cffb-c769-4dfe-b2ab-63e57691feb8\") "
Sep 09 13:39:57.950010 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.949376    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "76d4cffb-c769-4dfe-b2ab-63e57691feb8" (UID: "76d4cffb-c769-4dfe-b2ab-63e57691feb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:39:57.950010 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.949547    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "76d4cffb-c769-4dfe-b2ab-63e57691feb8" (UID: "76d4cffb-c769-4dfe-b2ab-63e57691feb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:39:57.950010 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.949915    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "76d4cffb-c769-4dfe-b2ab-63e57691feb8" (UID: "76d4cffb-c769-4dfe-b2ab-63e57691feb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:39:57.950010 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.949938    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "76d4cffb-c769-4dfe-b2ab-63e57691feb8" (UID: "76d4cffb-c769-4dfe-b2ab-63e57691feb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:39:57.950773 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.950161    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istio-data" (OuterVolumeSpecName: "istio-data") pod "76d4cffb-c769-4dfe-b2ab-63e57691feb8" (UID: "76d4cffb-c769-4dfe-b2ab-63e57691feb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:39:57.956835 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.956772    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istio-token" (OuterVolumeSpecName: "istio-token") pod "76d4cffb-c769-4dfe-b2ab-63e57691feb8" (UID: "76d4cffb-c769-4dfe-b2ab-63e57691feb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:39:57.956835 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.956826    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "76d4cffb-c769-4dfe-b2ab-63e57691feb8" (UID: "76d4cffb-c769-4dfe-b2ab-63e57691feb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 13:39:57.957018 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.956801    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "76d4cffb-c769-4dfe-b2ab-63e57691feb8" (UID: "76d4cffb-c769-4dfe-b2ab-63e57691feb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:39:57.957018 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.956802    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76d4cffb-c769-4dfe-b2ab-63e57691feb8-kube-api-access-4p58z" (OuterVolumeSpecName: "kube-api-access-4p58z") pod "76d4cffb-c769-4dfe-b2ab-63e57691feb8" (UID: "76d4cffb-c769-4dfe-b2ab-63e57691feb8"). InnerVolumeSpecName "kube-api-access-4p58z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:39:57.957759 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.957685    2566 generic.go:356] "Generic (PLEG): container finished" podID="76d4cffb-c769-4dfe-b2ab-63e57691feb8" containerID="571a29465882bc409a4be4b9134aa2efc0a29d4c7b0d83d38ad729e443f14333" exitCode=0
Sep 09 13:39:57.957759 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.957730    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"
Sep 09 13:39:57.958009 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.957759    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4" event={"ID":"76d4cffb-c769-4dfe-b2ab-63e57691feb8","Type":"ContainerDied","Data":"571a29465882bc409a4be4b9134aa2efc0a29d4c7b0d83d38ad729e443f14333"}
Sep 09 13:39:57.958009 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.957827    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4" event={"ID":"76d4cffb-c769-4dfe-b2ab-63e57691feb8","Type":"ContainerDied","Data":"59fef9903572a4bf2de9119affdf94f770e5ac495c7816e050335ebdb05947b3"}
Sep 09 13:39:57.958009 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.957856    2566 scope.go:117] "RemoveContainer" containerID="571a29465882bc409a4be4b9134aa2efc0a29d4c7b0d83d38ad729e443f14333"
Sep 09 13:39:57.968356 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.968332    2566 scope.go:117] "RemoveContainer" containerID="fdb2c215e9ddf0c692e0b9b9c0ca6f81087d8ea6ed2ed6ccf7ccd483fd3f7ddc"
Sep 09 13:39:57.977560 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.977531    2566 scope.go:117] "RemoveContainer" containerID="571a29465882bc409a4be4b9134aa2efc0a29d4c7b0d83d38ad729e443f14333"
Sep 09 13:39:57.977906 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:39:57.977884    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"571a29465882bc409a4be4b9134aa2efc0a29d4c7b0d83d38ad729e443f14333\": container with ID starting with 571a29465882bc409a4be4b9134aa2efc0a29d4c7b0d83d38ad729e443f14333 not found: ID does not exist" containerID="571a29465882bc409a4be4b9134aa2efc0a29d4c7b0d83d38ad729e443f14333"
Sep 09 13:39:57.977983 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.977917    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"571a29465882bc409a4be4b9134aa2efc0a29d4c7b0d83d38ad729e443f14333"} err="failed to get container status \"571a29465882bc409a4be4b9134aa2efc0a29d4c7b0d83d38ad729e443f14333\": rpc error: code = NotFound desc = could not find container \"571a29465882bc409a4be4b9134aa2efc0a29d4c7b0d83d38ad729e443f14333\": container with ID starting with 571a29465882bc409a4be4b9134aa2efc0a29d4c7b0d83d38ad729e443f14333 not found: ID does not exist"
Sep 09 13:39:57.977983 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.977937    2566 scope.go:117] "RemoveContainer" containerID="fdb2c215e9ddf0c692e0b9b9c0ca6f81087d8ea6ed2ed6ccf7ccd483fd3f7ddc"
Sep 09 13:39:57.978286 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:39:57.978233    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdb2c215e9ddf0c692e0b9b9c0ca6f81087d8ea6ed2ed6ccf7ccd483fd3f7ddc\": container with ID starting with fdb2c215e9ddf0c692e0b9b9c0ca6f81087d8ea6ed2ed6ccf7ccd483fd3f7ddc not found: ID does not exist" containerID="fdb2c215e9ddf0c692e0b9b9c0ca6f81087d8ea6ed2ed6ccf7ccd483fd3f7ddc"
Sep 09 13:39:57.978414 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.978293    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdb2c215e9ddf0c692e0b9b9c0ca6f81087d8ea6ed2ed6ccf7ccd483fd3f7ddc"} err="failed to get container status \"fdb2c215e9ddf0c692e0b9b9c0ca6f81087d8ea6ed2ed6ccf7ccd483fd3f7ddc\": rpc error: code = NotFound desc = could not find container \"fdb2c215e9ddf0c692e0b9b9c0ca6f81087d8ea6ed2ed6ccf7ccd483fd3f7ddc\": container with ID starting with fdb2c215e9ddf0c692e0b9b9c0ca6f81087d8ea6ed2ed6ccf7ccd483fd3f7ddc not found: ID does not exist"
Sep 09 13:39:57.980677 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.980657    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"]
Sep 09 13:39:57.985882 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:57.985861    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a649773d-openshift-default-7ccdffd898-prvp4"]
Sep 09 13:39:58.050247 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:58.050156    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:39:58.050247 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:58.050186    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:39:58.050247 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:58.050194    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:39:58.050247 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:58.050201    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:39:58.050247 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:58.050209    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:39:58.050247 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:58.050216    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:39:58.050247 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:58.050224    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:39:58.050247 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:58.050235    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d4cffb-c769-4dfe-b2ab-63e57691feb8-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:39:58.050247 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:58.050244    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p58z\" (UniqueName: \"kubernetes.io/projected/76d4cffb-c769-4dfe-b2ab-63e57691feb8-kube-api-access-4p58z\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:39:58.159046 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:58.159014    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g"]
Sep 09 13:39:58.160459 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:39:58.160430    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ab9563_5d90_4b5b_bd5b_3d6da3e438a3.slice/crio-616d0b4d9a33c46c306fa8edfd34eed0417c56c3ae6da2ba05a2f0f2b71772c1 WatchSource:0}: Error finding container 616d0b4d9a33c46c306fa8edfd34eed0417c56c3ae6da2ba05a2f0f2b71772c1: Status 404 returned error can't find the container with id 616d0b4d9a33c46c306fa8edfd34eed0417c56c3ae6da2ba05a2f0f2b71772c1
Sep 09 13:39:58.442677 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:58.442638    2566 patch_prober.go:28] interesting pod/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 09 13:39:58.442879 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:58.442695    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl" podUID="9ff22c80-e444-46f0-97d8-0cb8247f2231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 09 13:39:58.965370 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:58.965334    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g" event={"ID":"94ab9563-5d90-4b5b-bd5b-3d6da3e438a3","Type":"ContainerStarted","Data":"20d361fd2cce93a3bde982cba70b19942a4c0e0184a6e4d5138f2ab10dfcd3e2"}
Sep 09 13:39:58.965780 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:58.965392    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g" event={"ID":"94ab9563-5d90-4b5b-bd5b-3d6da3e438a3","Type":"ContainerStarted","Data":"616d0b4d9a33c46c306fa8edfd34eed0417c56c3ae6da2ba05a2f0f2b71772c1"}
Sep 09 13:39:58.965780 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:58.965425    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g"
Sep 09 13:39:58.986037 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:58.985983    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g" podStartSLOduration=1.98596459 podStartE2EDuration="1.98596459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:39:58.984753742 +0000 UTC m=+832.344603579" watchObservedRunningTime="2026-09-09 13:39:58.98596459 +0000 UTC m=+832.345814416"
Sep 09 13:39:59.150831 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:59.150793    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76d4cffb-c769-4dfe-b2ab-63e57691feb8" path="/var/lib/kubelet/pods/76d4cffb-c769-4dfe-b2ab-63e57691feb8/volumes"
Sep 09 13:39:59.443162 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:59.443121    2566 patch_prober.go:28] interesting pod/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 09 13:39:59.443348 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:39:59.443192    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl" podUID="9ff22c80-e444-46f0-97d8-0cb8247f2231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 09 13:40:00.447389 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:00.447351    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:40:00.447834 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:00.447772    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:40:00.448644 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:00.448619    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:40:09.973027 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:09.972992    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g"
Sep 09 13:40:16.764638 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:16.764600    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"]
Sep 09 13:40:16.765037 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:16.765021    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76d4cffb-c769-4dfe-b2ab-63e57691feb8" containerName="istio-proxy"
Sep 09 13:40:16.765085 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:16.765039    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="76d4cffb-c769-4dfe-b2ab-63e57691feb8" containerName="istio-proxy"
Sep 09 13:40:16.765125 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:16.765116    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="76d4cffb-c769-4dfe-b2ab-63e57691feb8" containerName="istio-proxy"
Sep 09 13:40:16.765159 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:16.765134    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="76d4cffb-c769-4dfe-b2ab-63e57691feb8" containerName="istio-proxy"
Sep 09 13:40:16.769460 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:16.769439    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"
Sep 09 13:40:16.771622 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:16.771601    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-bcc5a97b-kserve-self-signed-certs\""
Sep 09 13:40:16.782012 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:16.781983    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"]
Sep 09 13:40:16.936778 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:16.936731    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-model-cache\") pod \"e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8\" (UID: \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\") " pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"
Sep 09 13:40:16.937009 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:16.936793    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7qj7\" (UniqueName: \"kubernetes.io/projected/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-kube-api-access-l7qj7\") pod \"e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8\" (UID: \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\") " pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"
Sep 09 13:40:16.937009 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:16.936932    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-tls-certs\") pod \"e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8\" (UID: \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\") " pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"
Sep 09 13:40:16.937009 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:16.936982    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-home\") pod \"e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8\" (UID: \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\") " pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"
Sep 09 13:40:16.937134 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:16.937012    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-dshm\") pod \"e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8\" (UID: \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\") " pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"
Sep 09 13:40:16.937134 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:16.937063    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-tmp-dir\") pod \"e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8\" (UID: \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\") " pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"
Sep 09 13:40:17.038356 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:17.038244    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-tls-certs\") pod \"e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8\" (UID: \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\") " pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"
Sep 09 13:40:17.038356 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:17.038316    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-home\") pod \"e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8\" (UID: \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\") " pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"
Sep 09 13:40:17.038356 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:17.038353    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-dshm\") pod \"e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8\" (UID: \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\") " pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"
Sep 09 13:40:17.038635 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:17.038397    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-tmp-dir\") pod \"e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8\" (UID: \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\") " pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"
Sep 09 13:40:17.038635 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:17.038467    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-model-cache\") pod \"e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8\" (UID: \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\") " pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"
Sep 09 13:40:17.038635 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:17.038505    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7qj7\" (UniqueName: \"kubernetes.io/projected/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-kube-api-access-l7qj7\") pod \"e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8\" (UID: \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\") " pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"
Sep 09 13:40:17.038796 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:17.038758    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-home\") pod \"e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8\" (UID: \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\") " pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"
Sep 09 13:40:17.038885 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:17.038798    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-tmp-dir\") pod \"e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8\" (UID: \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\") " pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"
Sep 09 13:40:17.038953 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:17.038930    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-model-cache\") pod \"e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8\" (UID: \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\") " pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"
Sep 09 13:40:17.040752 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:17.040735    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-dshm\") pod \"e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8\" (UID: \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\") " pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"
Sep 09 13:40:17.041041 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:17.041025    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-tls-certs\") pod \"e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8\" (UID: \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\") " pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"
Sep 09 13:40:17.048247 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:17.048219    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7qj7\" (UniqueName: \"kubernetes.io/projected/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-kube-api-access-l7qj7\") pod \"e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8\" (UID: \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\") " pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"
Sep 09 13:40:17.079028 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:17.078990    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"
Sep 09 13:40:17.363562 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:17.363438    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"]
Sep 09 13:40:17.366737 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:40:17.366707    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3c6d694_f3aa_4e21_ad3d_bfad107f2f31.slice/crio-76f0511040dc4b1c7ed382b2f28f1c60c7ab144e7a7be4d31981e824527f206f WatchSource:0}: Error finding container 76f0511040dc4b1c7ed382b2f28f1c60c7ab144e7a7be4d31981e824527f206f: Status 404 returned error can't find the container with id 76f0511040dc4b1c7ed382b2f28f1c60c7ab144e7a7be4d31981e824527f206f
Sep 09 13:40:18.059726 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:18.059678    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8" event={"ID":"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31","Type":"ContainerStarted","Data":"0ce677a7da8455b827cd0e54207293e170201ebf1628a96c6972173c670d0353"}
Sep 09 13:40:18.059726 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:18.059732    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8" event={"ID":"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31","Type":"ContainerStarted","Data":"76f0511040dc4b1c7ed382b2f28f1c60c7ab144e7a7be4d31981e824527f206f"}
Sep 09 13:40:18.082326 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:18.082268    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8" podStartSLOduration=2.082249535 podStartE2EDuration="2.082249535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:40:18.080542862 +0000 UTC m=+851.440392688" watchObservedRunningTime="2026-09-09 13:40:18.082249535 +0000 UTC m=+851.442099360"
Sep 09 13:40:21.409412 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:21.409372    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-219dfd41-4nszp"]
Sep 09 13:40:21.409838 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:21.409745    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76d4cffb-c769-4dfe-b2ab-63e57691feb8" containerName="istio-proxy"
Sep 09 13:40:21.409838 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:21.409765    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="76d4cffb-c769-4dfe-b2ab-63e57691feb8" containerName="istio-proxy"
Sep 09 13:40:21.412991 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:21.412972    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-219dfd41-4nszp"
Sep 09 13:40:21.415370 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:21.415347    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lh487\""
Sep 09 13:40:21.415460 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:21.415372    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 13:40:21.421747 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:21.421722    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-219dfd41-4nszp"]
Sep 09 13:40:21.479710 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:21.479660    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95931938-9551-45e0-8712-7502808e9741-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-219dfd41-4nszp\" (UID: \"95931938-9551-45e0-8712-7502808e9741\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-219dfd41-4nszp"
Sep 09 13:40:21.580937 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:21.580896    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95931938-9551-45e0-8712-7502808e9741-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-219dfd41-4nszp\" (UID: \"95931938-9551-45e0-8712-7502808e9741\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-219dfd41-4nszp"
Sep 09 13:40:21.581478 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:21.581456    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95931938-9551-45e0-8712-7502808e9741-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-219dfd41-4nszp\" (UID: \"95931938-9551-45e0-8712-7502808e9741\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-219dfd41-4nszp"
Sep 09 13:40:21.726536 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:21.726435    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-219dfd41-4nszp"
Sep 09 13:40:22.004075 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:22.004040    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-219dfd41-4nszp"]
Sep 09 13:40:22.006649 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:40:22.006620    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95931938_9551_45e0_8712_7502808e9741.slice/crio-23b27af84191cfcfe1e2557ce13501fbae8cb18646b09abcf1d9d843bbf508ca WatchSource:0}: Error finding container 23b27af84191cfcfe1e2557ce13501fbae8cb18646b09abcf1d9d843bbf508ca: Status 404 returned error can't find the container with id 23b27af84191cfcfe1e2557ce13501fbae8cb18646b09abcf1d9d843bbf508ca
Sep 09 13:40:22.080662 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:22.080624    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-219dfd41-4nszp" event={"ID":"95931938-9551-45e0-8712-7502808e9741","Type":"ContainerStarted","Data":"23b27af84191cfcfe1e2557ce13501fbae8cb18646b09abcf1d9d843bbf508ca"}
Sep 09 13:40:23.086911 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:23.086875    2566 generic.go:356] "Generic (PLEG): container finished" podID="95931938-9551-45e0-8712-7502808e9741" containerID="740a79d962f7ba6f3aebeb4fa8abed59ec9b7ac5889cbd19568d860973026206" exitCode=0
Sep 09 13:40:23.087273 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:23.086972    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-219dfd41-4nszp" event={"ID":"95931938-9551-45e0-8712-7502808e9741","Type":"ContainerDied","Data":"740a79d962f7ba6f3aebeb4fa8abed59ec9b7ac5889cbd19568d860973026206"}
Sep 09 13:40:24.236423 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:24.236396    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-219dfd41-4nszp"
Sep 09 13:40:24.304973 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:24.304933    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95931938-9551-45e0-8712-7502808e9741-maas-api-service-ca\") pod \"95931938-9551-45e0-8712-7502808e9741\" (UID: \"95931938-9551-45e0-8712-7502808e9741\") "
Sep 09 13:40:24.305308 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:24.305287    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95931938-9551-45e0-8712-7502808e9741-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "95931938-9551-45e0-8712-7502808e9741" (UID: "95931938-9551-45e0-8712-7502808e9741"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:40:24.406485 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:24.406395    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95931938-9551-45e0-8712-7502808e9741-maas-api-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:40:25.099238 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:25.099204    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-219dfd41-4nszp" event={"ID":"95931938-9551-45e0-8712-7502808e9741","Type":"ContainerDied","Data":"23b27af84191cfcfe1e2557ce13501fbae8cb18646b09abcf1d9d843bbf508ca"}
Sep 09 13:40:25.099238 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:25.099237    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-219dfd41-4nszp"
Sep 09 13:40:25.099453 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:25.099256    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23b27af84191cfcfe1e2557ce13501fbae8cb18646b09abcf1d9d843bbf508ca"
Sep 09 13:40:27.080138 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:27.080099    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"
Sep 09 13:40:27.080540 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:27.080184    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"
Sep 09 13:40:27.092964 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:27.092939    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"
Sep 09 13:40:27.126205 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:27.126176    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"
Sep 09 13:40:28.098782 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:28.098747    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g"]
Sep 09 13:40:28.099532 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:28.099497    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g" podUID="94ab9563-5d90-4b5b-bd5b-3d6da3e438a3" containerName="payload-pre-processing" containerID="cri-o://20d361fd2cce93a3bde982cba70b19942a4c0e0184a6e4d5138f2ab10dfcd3e2" gracePeriod=30
Sep 09 13:40:28.275850 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:28.275784    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-219dfd41-4nszp"]
Sep 09 13:40:28.279043 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:28.279006    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-219dfd41-4nszp"]
Sep 09 13:40:28.396477 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:28.395662    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g"
Sep 09 13:40:28.447256 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:28.447217    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5s72\" (UniqueName: \"kubernetes.io/projected/94ab9563-5d90-4b5b-bd5b-3d6da3e438a3-kube-api-access-f5s72\") pod \"94ab9563-5d90-4b5b-bd5b-3d6da3e438a3\" (UID: \"94ab9563-5d90-4b5b-bd5b-3d6da3e438a3\") "
Sep 09 13:40:28.447256 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:28.447260    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ab9563-5d90-4b5b-bd5b-3d6da3e438a3-plugins-config-volume\") pod \"94ab9563-5d90-4b5b-bd5b-3d6da3e438a3\" (UID: \"94ab9563-5d90-4b5b-bd5b-3d6da3e438a3\") "
Sep 09 13:40:28.447689 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:28.447651    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94ab9563-5d90-4b5b-bd5b-3d6da3e438a3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "94ab9563-5d90-4b5b-bd5b-3d6da3e438a3" (UID: "94ab9563-5d90-4b5b-bd5b-3d6da3e438a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:40:28.449369 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:28.449343    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ab9563-5d90-4b5b-bd5b-3d6da3e438a3-kube-api-access-f5s72" (OuterVolumeSpecName: "kube-api-access-f5s72") pod "94ab9563-5d90-4b5b-bd5b-3d6da3e438a3" (UID: "94ab9563-5d90-4b5b-bd5b-3d6da3e438a3"). InnerVolumeSpecName "kube-api-access-f5s72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:40:28.548993 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:28.548955    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ab9563-5d90-4b5b-bd5b-3d6da3e438a3-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:40:28.548993 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:28.548985    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5s72\" (UniqueName: \"kubernetes.io/projected/94ab9563-5d90-4b5b-bd5b-3d6da3e438a3-kube-api-access-f5s72\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:40:29.124336 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:29.124295    2566 generic.go:356] "Generic (PLEG): container finished" podID="94ab9563-5d90-4b5b-bd5b-3d6da3e438a3" containerID="20d361fd2cce93a3bde982cba70b19942a4c0e0184a6e4d5138f2ab10dfcd3e2" exitCode=0
Sep 09 13:40:29.124766 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:29.124355    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g"
Sep 09 13:40:29.124766 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:29.124385    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g" event={"ID":"94ab9563-5d90-4b5b-bd5b-3d6da3e438a3","Type":"ContainerDied","Data":"20d361fd2cce93a3bde982cba70b19942a4c0e0184a6e4d5138f2ab10dfcd3e2"}
Sep 09 13:40:29.124766 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:29.124485    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g" event={"ID":"94ab9563-5d90-4b5b-bd5b-3d6da3e438a3","Type":"ContainerDied","Data":"616d0b4d9a33c46c306fa8edfd34eed0417c56c3ae6da2ba05a2f0f2b71772c1"}
Sep 09 13:40:29.124766 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:29.124516    2566 scope.go:117] "RemoveContainer" containerID="20d361fd2cce93a3bde982cba70b19942a4c0e0184a6e4d5138f2ab10dfcd3e2"
Sep 09 13:40:29.135059 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:29.135038    2566 scope.go:117] "RemoveContainer" containerID="20d361fd2cce93a3bde982cba70b19942a4c0e0184a6e4d5138f2ab10dfcd3e2"
Sep 09 13:40:29.135424 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:40:29.135399    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20d361fd2cce93a3bde982cba70b19942a4c0e0184a6e4d5138f2ab10dfcd3e2\": container with ID starting with 20d361fd2cce93a3bde982cba70b19942a4c0e0184a6e4d5138f2ab10dfcd3e2 not found: ID does not exist" containerID="20d361fd2cce93a3bde982cba70b19942a4c0e0184a6e4d5138f2ab10dfcd3e2"
Sep 09 13:40:29.135473 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:29.135435    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d361fd2cce93a3bde982cba70b19942a4c0e0184a6e4d5138f2ab10dfcd3e2"} err="failed to get container status \"20d361fd2cce93a3bde982cba70b19942a4c0e0184a6e4d5138f2ab10dfcd3e2\": rpc error: code = NotFound desc = could not find container \"20d361fd2cce93a3bde982cba70b19942a4c0e0184a6e4d5138f2ab10dfcd3e2\": container with ID starting with 20d361fd2cce93a3bde982cba70b19942a4c0e0184a6e4d5138f2ab10dfcd3e2 not found: ID does not exist"
Sep 09 13:40:29.153507 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:29.153466    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95931938-9551-45e0-8712-7502808e9741" path="/var/lib/kubelet/pods/95931938-9551-45e0-8712-7502808e9741/volumes"
Sep 09 13:40:29.153886 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:29.153865    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g"]
Sep 09 13:40:29.164396 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:29.164364    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-219dfd41-6dbd9d6574-np79g"]
Sep 09 13:40:31.150117 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:31.150083    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94ab9563-5d90-4b5b-bd5b-3d6da3e438a3" path="/var/lib/kubelet/pods/94ab9563-5d90-4b5b-bd5b-3d6da3e438a3/volumes"
Sep 09 13:40:43.555683 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.555646    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"]
Sep 09 13:40:43.556069 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.556014    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95931938-9551-45e0-8712-7502808e9741" containerName="revoke-keys"
Sep 09 13:40:43.556069 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.556029    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="95931938-9551-45e0-8712-7502808e9741" containerName="revoke-keys"
Sep 09 13:40:43.556069 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.556047    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94ab9563-5d90-4b5b-bd5b-3d6da3e438a3" containerName="payload-pre-processing"
Sep 09 13:40:43.556069 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.556053    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="94ab9563-5d90-4b5b-bd5b-3d6da3e438a3" containerName="payload-pre-processing"
Sep 09 13:40:43.556200 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.556123    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="94ab9563-5d90-4b5b-bd5b-3d6da3e438a3" containerName="payload-pre-processing"
Sep 09 13:40:43.556200 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.556133    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="95931938-9551-45e0-8712-7502808e9741" containerName="revoke-keys"
Sep 09 13:40:43.558995 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.558979    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"
Sep 09 13:40:43.562479 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.562457    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-71c181\"/\"kube-root-ca.crt\""
Sep 09 13:40:43.562479 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.562468    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-71c181\"/\"openshift-service-ca.crt\""
Sep 09 13:40:43.562640 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.562481    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-71c181\"/\"autores-model-71c181-kserve-self-signed-certs\""
Sep 09 13:40:43.562640 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.562502    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-71c181\"/\"default-dockercfg-h6dqc\""
Sep 09 13:40:43.568928 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.568903    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"]
Sep 09 13:40:43.703226 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.703184    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt6p2\" (UniqueName: \"kubernetes.io/projected/865ae236-d377-4e05-a2e9-bbdd15856966-kube-api-access-bt6p2\") pod \"autores-model-71c181-kserve-699d5bb559-kpxw9\" (UID: \"865ae236-d377-4e05-a2e9-bbdd15856966\") " pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"
Sep 09 13:40:43.703226 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.703225    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/865ae236-d377-4e05-a2e9-bbdd15856966-home\") pod \"autores-model-71c181-kserve-699d5bb559-kpxw9\" (UID: \"865ae236-d377-4e05-a2e9-bbdd15856966\") " pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"
Sep 09 13:40:43.703441 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.703328    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/865ae236-d377-4e05-a2e9-bbdd15856966-tmp-dir\") pod \"autores-model-71c181-kserve-699d5bb559-kpxw9\" (UID: \"865ae236-d377-4e05-a2e9-bbdd15856966\") " pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"
Sep 09 13:40:43.703441 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.703403    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/865ae236-d377-4e05-a2e9-bbdd15856966-dshm\") pod \"autores-model-71c181-kserve-699d5bb559-kpxw9\" (UID: \"865ae236-d377-4e05-a2e9-bbdd15856966\") " pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"
Sep 09 13:40:43.703441 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.703421    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/865ae236-d377-4e05-a2e9-bbdd15856966-model-cache\") pod \"autores-model-71c181-kserve-699d5bb559-kpxw9\" (UID: \"865ae236-d377-4e05-a2e9-bbdd15856966\") " pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"
Sep 09 13:40:43.703441 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.703438    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/865ae236-d377-4e05-a2e9-bbdd15856966-tls-certs\") pod \"autores-model-71c181-kserve-699d5bb559-kpxw9\" (UID: \"865ae236-d377-4e05-a2e9-bbdd15856966\") " pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"
Sep 09 13:40:43.804826 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.804776    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/865ae236-d377-4e05-a2e9-bbdd15856966-tmp-dir\") pod \"autores-model-71c181-kserve-699d5bb559-kpxw9\" (UID: \"865ae236-d377-4e05-a2e9-bbdd15856966\") " pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"
Sep 09 13:40:43.804994 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.804929    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/865ae236-d377-4e05-a2e9-bbdd15856966-dshm\") pod \"autores-model-71c181-kserve-699d5bb559-kpxw9\" (UID: \"865ae236-d377-4e05-a2e9-bbdd15856966\") " pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"
Sep 09 13:40:43.804994 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.804947    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/865ae236-d377-4e05-a2e9-bbdd15856966-model-cache\") pod \"autores-model-71c181-kserve-699d5bb559-kpxw9\" (UID: \"865ae236-d377-4e05-a2e9-bbdd15856966\") " pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"
Sep 09 13:40:43.804994 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.804964    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/865ae236-d377-4e05-a2e9-bbdd15856966-tls-certs\") pod \"autores-model-71c181-kserve-699d5bb559-kpxw9\" (UID: \"865ae236-d377-4e05-a2e9-bbdd15856966\") " pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"
Sep 09 13:40:43.805141 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.805022    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt6p2\" (UniqueName: \"kubernetes.io/projected/865ae236-d377-4e05-a2e9-bbdd15856966-kube-api-access-bt6p2\") pod \"autores-model-71c181-kserve-699d5bb559-kpxw9\" (UID: \"865ae236-d377-4e05-a2e9-bbdd15856966\") " pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"
Sep 09 13:40:43.805141 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.805041    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/865ae236-d377-4e05-a2e9-bbdd15856966-home\") pod \"autores-model-71c181-kserve-699d5bb559-kpxw9\" (UID: \"865ae236-d377-4e05-a2e9-bbdd15856966\") " pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"
Sep 09 13:40:43.805250 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.805195    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/865ae236-d377-4e05-a2e9-bbdd15856966-tmp-dir\") pod \"autores-model-71c181-kserve-699d5bb559-kpxw9\" (UID: \"865ae236-d377-4e05-a2e9-bbdd15856966\") " pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"
Sep 09 13:40:43.805369 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.805346    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/865ae236-d377-4e05-a2e9-bbdd15856966-model-cache\") pod \"autores-model-71c181-kserve-699d5bb559-kpxw9\" (UID: \"865ae236-d377-4e05-a2e9-bbdd15856966\") " pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"
Sep 09 13:40:43.805469 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.805419    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/865ae236-d377-4e05-a2e9-bbdd15856966-home\") pod \"autores-model-71c181-kserve-699d5bb559-kpxw9\" (UID: \"865ae236-d377-4e05-a2e9-bbdd15856966\") " pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"
Sep 09 13:40:43.807242 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.807188    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/865ae236-d377-4e05-a2e9-bbdd15856966-dshm\") pod \"autores-model-71c181-kserve-699d5bb559-kpxw9\" (UID: \"865ae236-d377-4e05-a2e9-bbdd15856966\") " pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"
Sep 09 13:40:43.807503 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.807482    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/865ae236-d377-4e05-a2e9-bbdd15856966-tls-certs\") pod \"autores-model-71c181-kserve-699d5bb559-kpxw9\" (UID: \"865ae236-d377-4e05-a2e9-bbdd15856966\") " pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"
Sep 09 13:40:43.813724 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.813702    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt6p2\" (UniqueName: \"kubernetes.io/projected/865ae236-d377-4e05-a2e9-bbdd15856966-kube-api-access-bt6p2\") pod \"autores-model-71c181-kserve-699d5bb559-kpxw9\" (UID: \"865ae236-d377-4e05-a2e9-bbdd15856966\") " pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"
Sep 09 13:40:43.869478 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:43.869443    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"
Sep 09 13:40:44.143221 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:44.143189    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"]
Sep 09 13:40:44.196767 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:44.196730    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9" event={"ID":"865ae236-d377-4e05-a2e9-bbdd15856966","Type":"ContainerStarted","Data":"d917ea36b89f098e6ead7e07f5273bc022eee0f666b021493744864f0357c88f"}
Sep 09 13:40:45.202490 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:45.202453    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9" event={"ID":"865ae236-d377-4e05-a2e9-bbdd15856966","Type":"ContainerStarted","Data":"2c77ea0f4f88ff721fe43444f314ee075e9fac64cfc9a0bd1b13e218b2cfbfa2"}
Sep 09 13:40:45.224726 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:45.224671    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9" podStartSLOduration=2.224655736 podStartE2EDuration="2.224655736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:40:45.22300209 +0000 UTC m=+878.582851921" watchObservedRunningTime="2026-09-09 13:40:45.224655736 +0000 UTC m=+878.584505562"
Sep 09 13:40:53.870302 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:53.870259    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"
Sep 09 13:40:53.870850 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:53.870326    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"
Sep 09 13:40:53.883525 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:53.883489    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"
Sep 09 13:40:54.257410 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:40:54.257330    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"
Sep 09 13:41:00.300120 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:00.300041    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9bbeff1-4nhd2"]
Sep 09 13:41:00.304197 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:00.304178    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9bbeff1-4nhd2"
Sep 09 13:41:00.306519 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:00.306496    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lh487\""
Sep 09 13:41:00.306647 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:00.306538    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 13:41:00.311160 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:00.311025    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9bbeff1-4nhd2"]
Sep 09 13:41:00.469173 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:00.469134    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e437e1ab-ff24-49c7-b6d2-3e8484d42587-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a9bbeff1-4nhd2\" (UID: \"e437e1ab-ff24-49c7-b6d2-3e8484d42587\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9bbeff1-4nhd2"
Sep 09 13:41:00.569835 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:00.569736    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e437e1ab-ff24-49c7-b6d2-3e8484d42587-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a9bbeff1-4nhd2\" (UID: \"e437e1ab-ff24-49c7-b6d2-3e8484d42587\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9bbeff1-4nhd2"
Sep 09 13:41:00.570434 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:00.570414    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e437e1ab-ff24-49c7-b6d2-3e8484d42587-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a9bbeff1-4nhd2\" (UID: \"e437e1ab-ff24-49c7-b6d2-3e8484d42587\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9bbeff1-4nhd2"
Sep 09 13:41:00.640028 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:00.639990    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9bbeff1-4nhd2"
Sep 09 13:41:00.913294 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:00.913268    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9bbeff1-4nhd2"]
Sep 09 13:41:00.914693 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:41:00.914669    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode437e1ab_ff24_49c7_b6d2_3e8484d42587.slice/crio-2d042ab35e993c401cc7ec60fafcd93cf52e860a9322c593bb1dd6f7330f88fa WatchSource:0}: Error finding container 2d042ab35e993c401cc7ec60fafcd93cf52e860a9322c593bb1dd6f7330f88fa: Status 404 returned error can't find the container with id 2d042ab35e993c401cc7ec60fafcd93cf52e860a9322c593bb1dd6f7330f88fa
Sep 09 13:41:01.284163 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:01.284121    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9bbeff1-4nhd2" event={"ID":"e437e1ab-ff24-49c7-b6d2-3e8484d42587","Type":"ContainerStarted","Data":"31ae7ec6bc1a409beae67a65d0ebfd18f994595a47977e0a01e3ee7b89ed49cc"}
Sep 09 13:41:01.284889 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:01.284185    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9bbeff1-4nhd2" event={"ID":"e437e1ab-ff24-49c7-b6d2-3e8484d42587","Type":"ContainerStarted","Data":"2d042ab35e993c401cc7ec60fafcd93cf52e860a9322c593bb1dd6f7330f88fa"}
Sep 09 13:41:02.291603 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:02.291567    2566 generic.go:356] "Generic (PLEG): container finished" podID="e437e1ab-ff24-49c7-b6d2-3e8484d42587" containerID="31ae7ec6bc1a409beae67a65d0ebfd18f994595a47977e0a01e3ee7b89ed49cc" exitCode=0
Sep 09 13:41:02.292009 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:02.291621    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9bbeff1-4nhd2" event={"ID":"e437e1ab-ff24-49c7-b6d2-3e8484d42587","Type":"ContainerDied","Data":"31ae7ec6bc1a409beae67a65d0ebfd18f994595a47977e0a01e3ee7b89ed49cc"}
Sep 09 13:41:03.430729 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:03.430696    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9bbeff1-4nhd2"
Sep 09 13:41:03.500402 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:03.500363    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e437e1ab-ff24-49c7-b6d2-3e8484d42587-maas-api-service-ca\") pod \"e437e1ab-ff24-49c7-b6d2-3e8484d42587\" (UID: \"e437e1ab-ff24-49c7-b6d2-3e8484d42587\") "
Sep 09 13:41:03.500769 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:03.500742    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e437e1ab-ff24-49c7-b6d2-3e8484d42587-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e437e1ab-ff24-49c7-b6d2-3e8484d42587" (UID: "e437e1ab-ff24-49c7-b6d2-3e8484d42587"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:41:03.601898 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:03.601857    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e437e1ab-ff24-49c7-b6d2-3e8484d42587-maas-api-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:41:04.302087 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:04.302061    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9bbeff1-4nhd2"
Sep 09 13:41:04.302282 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:04.302055    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9bbeff1-4nhd2" event={"ID":"e437e1ab-ff24-49c7-b6d2-3e8484d42587","Type":"ContainerDied","Data":"2d042ab35e993c401cc7ec60fafcd93cf52e860a9322c593bb1dd6f7330f88fa"}
Sep 09 13:41:04.302282 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:04.302194    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d042ab35e993c401cc7ec60fafcd93cf52e860a9322c593bb1dd6f7330f88fa"
Sep 09 13:41:06.546147 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:06.546102    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9bbeff1-4nhd2"]
Sep 09 13:41:06.547990 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:06.547969    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a9bbeff1-4nhd2"]
Sep 09 13:41:07.134620 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:07.134588    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ftm96_af902e8b-d693-4341-9979-03cf69548cfb/console-operator/2.log"
Sep 09 13:41:07.145193 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:07.145163    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ftm96_af902e8b-d693-4341-9979-03cf69548cfb/console-operator/2.log"
Sep 09 13:41:07.151353 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:07.151321    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e437e1ab-ff24-49c7-b6d2-3e8484d42587" path="/var/lib/kubelet/pods/e437e1ab-ff24-49c7-b6d2-3e8484d42587/volumes"
Sep 09 13:41:15.948625 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:15.948578    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"]
Sep 09 13:41:15.949150 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:15.949129    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e437e1ab-ff24-49c7-b6d2-3e8484d42587" containerName="revoke-keys"
Sep 09 13:41:15.949226 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:15.949153    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e437e1ab-ff24-49c7-b6d2-3e8484d42587" containerName="revoke-keys"
Sep 09 13:41:15.949306 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:15.949293    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e437e1ab-ff24-49c7-b6d2-3e8484d42587" containerName="revoke-keys"
Sep 09 13:41:15.953682 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:15.953659    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"
Sep 09 13:41:15.958099 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:15.958073    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4f59a0\"/\"kube-root-ca.crt\""
Sep 09 13:41:15.958283 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:15.958107    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4f59a0\"/\"openshift-service-ca.crt\""
Sep 09 13:41:15.958283 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:15.958081    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4f59a0\"/\"default-dockercfg-fncct\""
Sep 09 13:41:15.958283 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:15.958081    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4f59a0\"/\"test-model-4f59a0-kserve-self-signed-certs\""
Sep 09 13:41:15.964725 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:15.964700    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"]
Sep 09 13:41:16.027398 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:16.027360    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/585e4180-f184-462e-8325-100634a9524c-dshm\") pod \"test-model-4f59a0-kserve-57bf467579-ldjfn\" (UID: \"585e4180-f184-462e-8325-100634a9524c\") " pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"
Sep 09 13:41:16.027398 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:16.027400    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/585e4180-f184-462e-8325-100634a9524c-model-cache\") pod \"test-model-4f59a0-kserve-57bf467579-ldjfn\" (UID: \"585e4180-f184-462e-8325-100634a9524c\") " pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"
Sep 09 13:41:16.027654 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:16.027420    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcjrk\" (UniqueName: \"kubernetes.io/projected/585e4180-f184-462e-8325-100634a9524c-kube-api-access-xcjrk\") pod \"test-model-4f59a0-kserve-57bf467579-ldjfn\" (UID: \"585e4180-f184-462e-8325-100634a9524c\") " pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"
Sep 09 13:41:16.027654 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:16.027453    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/585e4180-f184-462e-8325-100634a9524c-tmp-dir\") pod \"test-model-4f59a0-kserve-57bf467579-ldjfn\" (UID: \"585e4180-f184-462e-8325-100634a9524c\") " pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"
Sep 09 13:41:16.027654 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:16.027556    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/585e4180-f184-462e-8325-100634a9524c-tls-certs\") pod \"test-model-4f59a0-kserve-57bf467579-ldjfn\" (UID: \"585e4180-f184-462e-8325-100634a9524c\") " pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"
Sep 09 13:41:16.027654 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:16.027572    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/585e4180-f184-462e-8325-100634a9524c-home\") pod \"test-model-4f59a0-kserve-57bf467579-ldjfn\" (UID: \"585e4180-f184-462e-8325-100634a9524c\") " pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"
Sep 09 13:41:16.128898 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:16.128858    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/585e4180-f184-462e-8325-100634a9524c-dshm\") pod \"test-model-4f59a0-kserve-57bf467579-ldjfn\" (UID: \"585e4180-f184-462e-8325-100634a9524c\") " pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"
Sep 09 13:41:16.128898 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:16.128899    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/585e4180-f184-462e-8325-100634a9524c-model-cache\") pod \"test-model-4f59a0-kserve-57bf467579-ldjfn\" (UID: \"585e4180-f184-462e-8325-100634a9524c\") " pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"
Sep 09 13:41:16.129168 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:16.128920    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcjrk\" (UniqueName: \"kubernetes.io/projected/585e4180-f184-462e-8325-100634a9524c-kube-api-access-xcjrk\") pod \"test-model-4f59a0-kserve-57bf467579-ldjfn\" (UID: \"585e4180-f184-462e-8325-100634a9524c\") " pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"
Sep 09 13:41:16.129168 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:16.128939    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/585e4180-f184-462e-8325-100634a9524c-tmp-dir\") pod \"test-model-4f59a0-kserve-57bf467579-ldjfn\" (UID: \"585e4180-f184-462e-8325-100634a9524c\") " pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"
Sep 09 13:41:16.129168 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:16.129070    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/585e4180-f184-462e-8325-100634a9524c-tls-certs\") pod \"test-model-4f59a0-kserve-57bf467579-ldjfn\" (UID: \"585e4180-f184-462e-8325-100634a9524c\") " pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"
Sep 09 13:41:16.129168 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:16.129094    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/585e4180-f184-462e-8325-100634a9524c-home\") pod \"test-model-4f59a0-kserve-57bf467579-ldjfn\" (UID: \"585e4180-f184-462e-8325-100634a9524c\") " pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"
Sep 09 13:41:16.129428 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:16.129265    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/585e4180-f184-462e-8325-100634a9524c-model-cache\") pod \"test-model-4f59a0-kserve-57bf467579-ldjfn\" (UID: \"585e4180-f184-462e-8325-100634a9524c\") " pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"
Sep 09 13:41:16.129428 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:16.129405    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/585e4180-f184-462e-8325-100634a9524c-tmp-dir\") pod \"test-model-4f59a0-kserve-57bf467579-ldjfn\" (UID: \"585e4180-f184-462e-8325-100634a9524c\") " pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"
Sep 09 13:41:16.129428 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:16.129422    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/585e4180-f184-462e-8325-100634a9524c-home\") pod \"test-model-4f59a0-kserve-57bf467579-ldjfn\" (UID: \"585e4180-f184-462e-8325-100634a9524c\") " pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"
Sep 09 13:41:16.131211 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:16.131189    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/585e4180-f184-462e-8325-100634a9524c-dshm\") pod \"test-model-4f59a0-kserve-57bf467579-ldjfn\" (UID: \"585e4180-f184-462e-8325-100634a9524c\") " pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"
Sep 09 13:41:16.131463 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:16.131445    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/585e4180-f184-462e-8325-100634a9524c-tls-certs\") pod \"test-model-4f59a0-kserve-57bf467579-ldjfn\" (UID: \"585e4180-f184-462e-8325-100634a9524c\") " pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"
Sep 09 13:41:16.139250 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:16.139214    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcjrk\" (UniqueName: \"kubernetes.io/projected/585e4180-f184-462e-8325-100634a9524c-kube-api-access-xcjrk\") pod \"test-model-4f59a0-kserve-57bf467579-ldjfn\" (UID: \"585e4180-f184-462e-8325-100634a9524c\") " pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"
Sep 09 13:41:16.264594 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:16.264502    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"
Sep 09 13:41:16.541856 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:16.541716    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"]
Sep 09 13:41:16.544584 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:41:16.544555    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod585e4180_f184_462e_8325_100634a9524c.slice/crio-a6c4c099d49662146a4612f5f21e9979e3f38dc928500ad04097968f05423af0 WatchSource:0}: Error finding container a6c4c099d49662146a4612f5f21e9979e3f38dc928500ad04097968f05423af0: Status 404 returned error can't find the container with id a6c4c099d49662146a4612f5f21e9979e3f38dc928500ad04097968f05423af0
Sep 09 13:41:17.085511 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:17.085477    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71c181-wnn6c"]
Sep 09 13:41:17.088930 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:17.088908    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71c181-wnn6c"
Sep 09 13:41:17.091250 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:17.091227    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 13:41:17.091250 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:17.091240    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lh487\""
Sep 09 13:41:17.096200 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:17.096170    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71c181-wnn6c"]
Sep 09 13:41:17.142045 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:17.142013    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18332fe-99f6-49ab-a578-10ecbf274bbb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-71c181-wnn6c\" (UID: \"b18332fe-99f6-49ab-a578-10ecbf274bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71c181-wnn6c"
Sep 09 13:41:17.242976 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:17.242930    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18332fe-99f6-49ab-a578-10ecbf274bbb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-71c181-wnn6c\" (UID: \"b18332fe-99f6-49ab-a578-10ecbf274bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71c181-wnn6c"
Sep 09 13:41:17.243642 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:17.243617    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18332fe-99f6-49ab-a578-10ecbf274bbb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-71c181-wnn6c\" (UID: \"b18332fe-99f6-49ab-a578-10ecbf274bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71c181-wnn6c"
Sep 09 13:41:17.370631 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:17.370509    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn" event={"ID":"585e4180-f184-462e-8325-100634a9524c","Type":"ContainerStarted","Data":"8a225d605960898ab23e45127db7f0b8fa73a605700d09f8458c578cf3aafb4f"}
Sep 09 13:41:17.370631 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:17.370575    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn" event={"ID":"585e4180-f184-462e-8325-100634a9524c","Type":"ContainerStarted","Data":"a6c4c099d49662146a4612f5f21e9979e3f38dc928500ad04097968f05423af0"}
Sep 09 13:41:17.388443 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:17.388393    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn" podStartSLOduration=2.388374794 podStartE2EDuration="2.388374794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:41:17.386983044 +0000 UTC m=+910.746832871" watchObservedRunningTime="2026-09-09 13:41:17.388374794 +0000 UTC m=+910.748224619"
Sep 09 13:41:17.401105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:17.401070    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71c181-wnn6c"
Sep 09 13:41:17.702432 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:17.702340    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71c181-wnn6c"]
Sep 09 13:41:17.705202 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:41:17.705172    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb18332fe_99f6_49ab_a578_10ecbf274bbb.slice/crio-71d4ee20d1563acc89e1331b516a68585f09d27fecad2820b6fcad0ae4d8229c WatchSource:0}: Error finding container 71d4ee20d1563acc89e1331b516a68585f09d27fecad2820b6fcad0ae4d8229c: Status 404 returned error can't find the container with id 71d4ee20d1563acc89e1331b516a68585f09d27fecad2820b6fcad0ae4d8229c
Sep 09 13:41:18.377256 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:18.377217    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71c181-wnn6c" event={"ID":"b18332fe-99f6-49ab-a578-10ecbf274bbb","Type":"ContainerStarted","Data":"4e7b009861d40b868c57e291fb5c766bd268b99a621592e2628380d93e2073aa"}
Sep 09 13:41:18.377696 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:18.377270    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71c181-wnn6c" event={"ID":"b18332fe-99f6-49ab-a578-10ecbf274bbb","Type":"ContainerStarted","Data":"71d4ee20d1563acc89e1331b516a68585f09d27fecad2820b6fcad0ae4d8229c"}
Sep 09 13:41:18.394549 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:18.394480    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71c181-wnn6c" podStartSLOduration=1.394458927 podStartE2EDuration="1.394458927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:41:18.393547946 +0000 UTC m=+911.753397773" watchObservedRunningTime="2026-09-09 13:41:18.394458927 +0000 UTC m=+911.754308755"
Sep 09 13:41:19.383604 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:19.383566    2566 generic.go:356] "Generic (PLEG): container finished" podID="b18332fe-99f6-49ab-a578-10ecbf274bbb" containerID="4e7b009861d40b868c57e291fb5c766bd268b99a621592e2628380d93e2073aa" exitCode=0
Sep 09 13:41:19.384006 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:19.383648    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71c181-wnn6c" event={"ID":"b18332fe-99f6-49ab-a578-10ecbf274bbb","Type":"ContainerDied","Data":"4e7b009861d40b868c57e291fb5c766bd268b99a621592e2628380d93e2073aa"}
Sep 09 13:41:19.650276 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:19.650195    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"]
Sep 09 13:41:20.531679 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:20.531648    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71c181-wnn6c"
Sep 09 13:41:20.557094 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:20.557059    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"]
Sep 09 13:41:20.684062 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:20.683961    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18332fe-99f6-49ab-a578-10ecbf274bbb-maas-api-service-ca\") pod \"b18332fe-99f6-49ab-a578-10ecbf274bbb\" (UID: \"b18332fe-99f6-49ab-a578-10ecbf274bbb\") "
Sep 09 13:41:20.684404 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:20.684378    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b18332fe-99f6-49ab-a578-10ecbf274bbb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b18332fe-99f6-49ab-a578-10ecbf274bbb" (UID: "b18332fe-99f6-49ab-a578-10ecbf274bbb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:41:20.785261 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:20.785220    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18332fe-99f6-49ab-a578-10ecbf274bbb-maas-api-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:41:21.394765 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:21.394737    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71c181-wnn6c"
Sep 09 13:41:21.394963 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:21.394764    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71c181-wnn6c" event={"ID":"b18332fe-99f6-49ab-a578-10ecbf274bbb","Type":"ContainerDied","Data":"71d4ee20d1563acc89e1331b516a68585f09d27fecad2820b6fcad0ae4d8229c"}
Sep 09 13:41:21.394963 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:21.394831    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71d4ee20d1563acc89e1331b516a68585f09d27fecad2820b6fcad0ae4d8229c"
Sep 09 13:41:23.248658 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:23.248618    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"]
Sep 09 13:41:26.265093 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:26.265053    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"
Sep 09 13:41:26.265093 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:26.265103    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"
Sep 09 13:41:26.278145 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:26.278110    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"
Sep 09 13:41:26.435916 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:26.435887    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"
Sep 09 13:41:27.926151 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:27.926117    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-71c181-9448f866c-4p6s9"]
Sep 09 13:41:27.926624 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:27.926402    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-71c181-9448f866c-4p6s9" podUID="6592a366-304b-4e42-bd75-3f26fc5901c5" containerName="payload-processing" containerID="cri-o://3d1705930983350d8534a2079f833170ade64d0ba10e2f2a95a8734591763c24" gracePeriod=30
Sep 09 13:41:28.167512 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:28.167473    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71c181-wnn6c"]
Sep 09 13:41:28.169259 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:28.169238    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71c181-wnn6c"]
Sep 09 13:41:28.189032 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:28.188962    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-71c181-9448f866c-4p6s9"
Sep 09 13:41:28.359515 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:28.359478    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfwlp\" (UniqueName: \"kubernetes.io/projected/6592a366-304b-4e42-bd75-3f26fc5901c5-kube-api-access-zfwlp\") pod \"6592a366-304b-4e42-bd75-3f26fc5901c5\" (UID: \"6592a366-304b-4e42-bd75-3f26fc5901c5\") "
Sep 09 13:41:28.359732 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:28.359537    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6592a366-304b-4e42-bd75-3f26fc5901c5-plugins-config-volume\") pod \"6592a366-304b-4e42-bd75-3f26fc5901c5\" (UID: \"6592a366-304b-4e42-bd75-3f26fc5901c5\") "
Sep 09 13:41:28.359953 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:28.359925    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6592a366-304b-4e42-bd75-3f26fc5901c5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6592a366-304b-4e42-bd75-3f26fc5901c5" (UID: "6592a366-304b-4e42-bd75-3f26fc5901c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:41:28.361703 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:28.361665    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6592a366-304b-4e42-bd75-3f26fc5901c5-kube-api-access-zfwlp" (OuterVolumeSpecName: "kube-api-access-zfwlp") pod "6592a366-304b-4e42-bd75-3f26fc5901c5" (UID: "6592a366-304b-4e42-bd75-3f26fc5901c5"). InnerVolumeSpecName "kube-api-access-zfwlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:41:28.435633 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:28.435599    2566 generic.go:356] "Generic (PLEG): container finished" podID="6592a366-304b-4e42-bd75-3f26fc5901c5" containerID="3d1705930983350d8534a2079f833170ade64d0ba10e2f2a95a8734591763c24" exitCode=0
Sep 09 13:41:28.435829 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:28.435656    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-71c181-9448f866c-4p6s9"
Sep 09 13:41:28.435829 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:28.435688    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-71c181-9448f866c-4p6s9" event={"ID":"6592a366-304b-4e42-bd75-3f26fc5901c5","Type":"ContainerDied","Data":"3d1705930983350d8534a2079f833170ade64d0ba10e2f2a95a8734591763c24"}
Sep 09 13:41:28.435829 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:28.435775    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-71c181-9448f866c-4p6s9" event={"ID":"6592a366-304b-4e42-bd75-3f26fc5901c5","Type":"ContainerDied","Data":"0c4a519189fcfb11f7a3a7035aa6130befbf1905955dfaf1fc7b133dfa2833fe"}
Sep 09 13:41:28.435829 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:28.435805    2566 scope.go:117] "RemoveContainer" containerID="3d1705930983350d8534a2079f833170ade64d0ba10e2f2a95a8734591763c24"
Sep 09 13:41:28.447908 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:28.447883    2566 scope.go:117] "RemoveContainer" containerID="3d1705930983350d8534a2079f833170ade64d0ba10e2f2a95a8734591763c24"
Sep 09 13:41:28.448282 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:41:28.448259    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d1705930983350d8534a2079f833170ade64d0ba10e2f2a95a8734591763c24\": container with ID starting with 3d1705930983350d8534a2079f833170ade64d0ba10e2f2a95a8734591763c24 not found: ID does not exist" containerID="3d1705930983350d8534a2079f833170ade64d0ba10e2f2a95a8734591763c24"
Sep 09 13:41:28.448333 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:28.448292    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d1705930983350d8534a2079f833170ade64d0ba10e2f2a95a8734591763c24"} err="failed to get container status \"3d1705930983350d8534a2079f833170ade64d0ba10e2f2a95a8734591763c24\": rpc error: code = NotFound desc = could not find container \"3d1705930983350d8534a2079f833170ade64d0ba10e2f2a95a8734591763c24\": container with ID starting with 3d1705930983350d8534a2079f833170ade64d0ba10e2f2a95a8734591763c24 not found: ID does not exist"
Sep 09 13:41:28.460133 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:28.460094    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-71c181-9448f866c-4p6s9"]
Sep 09 13:41:28.460700 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:28.460675    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfwlp\" (UniqueName: \"kubernetes.io/projected/6592a366-304b-4e42-bd75-3f26fc5901c5-kube-api-access-zfwlp\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:41:28.460838 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:28.460701    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6592a366-304b-4e42-bd75-3f26fc5901c5-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:41:28.467262 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:28.467234    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-71c181-9448f866c-4p6s9"]
Sep 09 13:41:29.151390 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:29.151351    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6592a366-304b-4e42-bd75-3f26fc5901c5" path="/var/lib/kubelet/pods/6592a366-304b-4e42-bd75-3f26fc5901c5/volumes"
Sep 09 13:41:29.151876 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:29.151858    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b18332fe-99f6-49ab-a578-10ecbf274bbb" path="/var/lib/kubelet/pods/b18332fe-99f6-49ab-a578-10ecbf274bbb/volumes"
Sep 09 13:41:34.670966 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:34.670923    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq" podUID="e8f84e7e-bf8b-403a-9eab-92ae34fb9357" containerName="main" containerID="cri-o://5152b5a30d3f2c344d55ac8fbfbe79aea02309d4cab2d979a644e880028bf1a1" gracePeriod=45
Sep 09 13:41:34.933347 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:34.933275    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"
Sep 09 13:41:35.129458 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.129418    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-home\") pod \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\" (UID: \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\") "
Sep 09 13:41:35.129680 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.129493    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-tmp-dir\") pod \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\" (UID: \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\") "
Sep 09 13:41:35.129680 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.129554    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-dshm\") pod \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\" (UID: \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\") "
Sep 09 13:41:35.129680 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.129618    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtbx9\" (UniqueName: \"kubernetes.io/projected/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-kube-api-access-mtbx9\") pod \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\" (UID: \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\") "
Sep 09 13:41:35.129884 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.129682    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-model-cache\") pod \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\" (UID: \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\") "
Sep 09 13:41:35.129884 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.129709    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-tls-certs\") pod \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\" (UID: \"e8f84e7e-bf8b-403a-9eab-92ae34fb9357\") "
Sep 09 13:41:35.129884 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.129761    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-home" (OuterVolumeSpecName: "home") pod "e8f84e7e-bf8b-403a-9eab-92ae34fb9357" (UID: "e8f84e7e-bf8b-403a-9eab-92ae34fb9357"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:41:35.129884 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.129766    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e8f84e7e-bf8b-403a-9eab-92ae34fb9357" (UID: "e8f84e7e-bf8b-403a-9eab-92ae34fb9357"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:41:35.130118 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.130031    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-model-cache" (OuterVolumeSpecName: "model-cache") pod "e8f84e7e-bf8b-403a-9eab-92ae34fb9357" (UID: "e8f84e7e-bf8b-403a-9eab-92ae34fb9357"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:41:35.130118 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.130113    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:41:35.130244 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.130130    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:41:35.130244 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.130140    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:41:35.131945 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.131829    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-dshm" (OuterVolumeSpecName: "dshm") pod "e8f84e7e-bf8b-403a-9eab-92ae34fb9357" (UID: "e8f84e7e-bf8b-403a-9eab-92ae34fb9357"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:41:35.132060 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.131897    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8f84e7e-bf8b-403a-9eab-92ae34fb9357" (UID: "e8f84e7e-bf8b-403a-9eab-92ae34fb9357"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:41:35.132060 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.131910    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-kube-api-access-mtbx9" (OuterVolumeSpecName: "kube-api-access-mtbx9") pod "e8f84e7e-bf8b-403a-9eab-92ae34fb9357" (UID: "e8f84e7e-bf8b-403a-9eab-92ae34fb9357"). InnerVolumeSpecName "kube-api-access-mtbx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:41:35.231631 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.231545    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:41:35.231631 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.231574    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:41:35.231631 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.231588    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtbx9\" (UniqueName: \"kubernetes.io/projected/e8f84e7e-bf8b-403a-9eab-92ae34fb9357-kube-api-access-mtbx9\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:41:35.479369 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.479337    2566 generic.go:356] "Generic (PLEG): container finished" podID="e8f84e7e-bf8b-403a-9eab-92ae34fb9357" containerID="5152b5a30d3f2c344d55ac8fbfbe79aea02309d4cab2d979a644e880028bf1a1" exitCode=0
Sep 09 13:41:35.479524 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.479376    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"
Sep 09 13:41:35.479524 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.479418    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq" event={"ID":"e8f84e7e-bf8b-403a-9eab-92ae34fb9357","Type":"ContainerDied","Data":"5152b5a30d3f2c344d55ac8fbfbe79aea02309d4cab2d979a644e880028bf1a1"}
Sep 09 13:41:35.479524 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.479458    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq" event={"ID":"e8f84e7e-bf8b-403a-9eab-92ae34fb9357","Type":"ContainerDied","Data":"6611bd5ce5054a7d2f230c527b42ef220be01f36e673144bf3dd75a41e1be9a0"}
Sep 09 13:41:35.479524 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.479475    2566 scope.go:117] "RemoveContainer" containerID="5152b5a30d3f2c344d55ac8fbfbe79aea02309d4cab2d979a644e880028bf1a1"
Sep 09 13:41:35.488135 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.488119    2566 scope.go:117] "RemoveContainer" containerID="5152b5a30d3f2c344d55ac8fbfbe79aea02309d4cab2d979a644e880028bf1a1"
Sep 09 13:41:35.488417 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:41:35.488402    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5152b5a30d3f2c344d55ac8fbfbe79aea02309d4cab2d979a644e880028bf1a1\": container with ID starting with 5152b5a30d3f2c344d55ac8fbfbe79aea02309d4cab2d979a644e880028bf1a1 not found: ID does not exist" containerID="5152b5a30d3f2c344d55ac8fbfbe79aea02309d4cab2d979a644e880028bf1a1"
Sep 09 13:41:35.488461 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.488426    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5152b5a30d3f2c344d55ac8fbfbe79aea02309d4cab2d979a644e880028bf1a1"} err="failed to get container status \"5152b5a30d3f2c344d55ac8fbfbe79aea02309d4cab2d979a644e880028bf1a1\": rpc error: code = NotFound desc = could not find container \"5152b5a30d3f2c344d55ac8fbfbe79aea02309d4cab2d979a644e880028bf1a1\": container with ID starting with 5152b5a30d3f2c344d55ac8fbfbe79aea02309d4cab2d979a644e880028bf1a1 not found: ID does not exist"
Sep 09 13:41:35.496558 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.496527    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"]
Sep 09 13:41:35.500866 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.500839    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-bcc5a97b-kserve-6bb8b69cd5-566gq"]
Sep 09 13:41:35.577938 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.577900    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8" podUID="b3c6d694-f3aa-4e21-ad3d-bfad107f2f31" containerName="main" containerID="cri-o://0ce677a7da8455b827cd0e54207293e170201ebf1628a96c6972173c670d0353" gracePeriod=45
Sep 09 13:41:35.673685 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.673648    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ecd209d-xnrmx"]
Sep 09 13:41:35.674096 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.674004    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8f84e7e-bf8b-403a-9eab-92ae34fb9357" containerName="main"
Sep 09 13:41:35.674096 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.674015    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8f84e7e-bf8b-403a-9eab-92ae34fb9357" containerName="main"
Sep 09 13:41:35.674096 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.674028    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b18332fe-99f6-49ab-a578-10ecbf274bbb" containerName="revoke-keys"
Sep 09 13:41:35.674096 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.674034    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b18332fe-99f6-49ab-a578-10ecbf274bbb" containerName="revoke-keys"
Sep 09 13:41:35.674096 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.674049    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6592a366-304b-4e42-bd75-3f26fc5901c5" containerName="payload-processing"
Sep 09 13:41:35.674096 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.674056    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6592a366-304b-4e42-bd75-3f26fc5901c5" containerName="payload-processing"
Sep 09 13:41:35.674318 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.674113    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8f84e7e-bf8b-403a-9eab-92ae34fb9357" containerName="main"
Sep 09 13:41:35.674318 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.674120    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b18332fe-99f6-49ab-a578-10ecbf274bbb" containerName="revoke-keys"
Sep 09 13:41:35.674318 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.674127    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6592a366-304b-4e42-bd75-3f26fc5901c5" containerName="payload-processing"
Sep 09 13:41:35.677080 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.677061    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ecd209d-xnrmx"
Sep 09 13:41:35.679627 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.679589    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lh487\""
Sep 09 13:41:35.679895 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.679878    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 13:41:35.687575 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.687550    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ecd209d-xnrmx"]
Sep 09 13:41:35.838280 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.838246    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16c5d225-6c48-4b39-af15-495e9d7a9b4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4ecd209d-xnrmx\" (UID: \"16c5d225-6c48-4b39-af15-495e9d7a9b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ecd209d-xnrmx"
Sep 09 13:41:35.843096 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.843076    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"
Sep 09 13:41:35.939231 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.939195    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-tls-certs\") pod \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\" (UID: \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\") "
Sep 09 13:41:35.939411 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.939273    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-tmp-dir\") pod \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\" (UID: \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\") "
Sep 09 13:41:35.939481 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.939430    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-dshm\") pod \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\" (UID: \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\") "
Sep 09 13:41:35.939544 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.939492    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b3c6d694-f3aa-4e21-ad3d-bfad107f2f31" (UID: "b3c6d694-f3aa-4e21-ad3d-bfad107f2f31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:41:35.939544 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.939508    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-home\") pod \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\" (UID: \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\") "
Sep 09 13:41:35.939544 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.939539    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-model-cache\") pod \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\" (UID: \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\") "
Sep 09 13:41:35.939708 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.939571    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7qj7\" (UniqueName: \"kubernetes.io/projected/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-kube-api-access-l7qj7\") pod \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\" (UID: \"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31\") "
Sep 09 13:41:35.939708 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.939662    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-home" (OuterVolumeSpecName: "home") pod "b3c6d694-f3aa-4e21-ad3d-bfad107f2f31" (UID: "b3c6d694-f3aa-4e21-ad3d-bfad107f2f31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:41:35.939892 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.939735    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-model-cache" (OuterVolumeSpecName: "model-cache") pod "b3c6d694-f3aa-4e21-ad3d-bfad107f2f31" (UID: "b3c6d694-f3aa-4e21-ad3d-bfad107f2f31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:41:35.939892 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.939858    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16c5d225-6c48-4b39-af15-495e9d7a9b4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4ecd209d-xnrmx\" (UID: \"16c5d225-6c48-4b39-af15-495e9d7a9b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ecd209d-xnrmx"
Sep 09 13:41:35.940003 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.939976    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:41:35.940003 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.939991    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:41:35.940107 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.940005    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:41:35.940499 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.940476    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16c5d225-6c48-4b39-af15-495e9d7a9b4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4ecd209d-xnrmx\" (UID: \"16c5d225-6c48-4b39-af15-495e9d7a9b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ecd209d-xnrmx"
Sep 09 13:41:35.941496 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.941471    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b3c6d694-f3aa-4e21-ad3d-bfad107f2f31" (UID: "b3c6d694-f3aa-4e21-ad3d-bfad107f2f31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:41:35.941579 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.941551    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-dshm" (OuterVolumeSpecName: "dshm") pod "b3c6d694-f3aa-4e21-ad3d-bfad107f2f31" (UID: "b3c6d694-f3aa-4e21-ad3d-bfad107f2f31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:41:35.941618 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.941604    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-kube-api-access-l7qj7" (OuterVolumeSpecName: "kube-api-access-l7qj7") pod "b3c6d694-f3aa-4e21-ad3d-bfad107f2f31" (UID: "b3c6d694-f3aa-4e21-ad3d-bfad107f2f31"). InnerVolumeSpecName "kube-api-access-l7qj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:41:35.999054 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:35.999014    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ecd209d-xnrmx"
Sep 09 13:41:36.040574 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:36.040540    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:41:36.040574 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:36.040569    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:41:36.040574 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:36.040578    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7qj7\" (UniqueName: \"kubernetes.io/projected/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31-kube-api-access-l7qj7\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:41:36.273530 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:36.273499    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ecd209d-xnrmx"]
Sep 09 13:41:36.275980 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:41:36.275954    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16c5d225_6c48_4b39_af15_495e9d7a9b4b.slice/crio-bc9ca6f1b5296655b6f5a2f6b7dcb6e727d9a6e8a23b171c651805324d675712 WatchSource:0}: Error finding container bc9ca6f1b5296655b6f5a2f6b7dcb6e727d9a6e8a23b171c651805324d675712: Status 404 returned error can't find the container with id bc9ca6f1b5296655b6f5a2f6b7dcb6e727d9a6e8a23b171c651805324d675712
Sep 09 13:41:36.487869 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:36.487834    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ecd209d-xnrmx" event={"ID":"16c5d225-6c48-4b39-af15-495e9d7a9b4b","Type":"ContainerStarted","Data":"bc9ca6f1b5296655b6f5a2f6b7dcb6e727d9a6e8a23b171c651805324d675712"}
Sep 09 13:41:36.489120 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:36.489096    2566 generic.go:356] "Generic (PLEG): container finished" podID="b3c6d694-f3aa-4e21-ad3d-bfad107f2f31" containerID="0ce677a7da8455b827cd0e54207293e170201ebf1628a96c6972173c670d0353" exitCode=0
Sep 09 13:41:36.489250 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:36.489137    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"
Sep 09 13:41:36.489250 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:36.489154    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8" event={"ID":"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31","Type":"ContainerDied","Data":"0ce677a7da8455b827cd0e54207293e170201ebf1628a96c6972173c670d0353"}
Sep 09 13:41:36.489250 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:36.489189    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8" event={"ID":"b3c6d694-f3aa-4e21-ad3d-bfad107f2f31","Type":"ContainerDied","Data":"76f0511040dc4b1c7ed382b2f28f1c60c7ab144e7a7be4d31981e824527f206f"}
Sep 09 13:41:36.489250 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:36.489215    2566 scope.go:117] "RemoveContainer" containerID="0ce677a7da8455b827cd0e54207293e170201ebf1628a96c6972173c670d0353"
Sep 09 13:41:36.528470 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:36.528388    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"]
Sep 09 13:41:36.537513 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:36.537482    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-bcc5a97b-kserve-ccbb4ff95-649f8"]
Sep 09 13:41:36.589381 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:36.589359    2566 scope.go:117] "RemoveContainer" containerID="0ce677a7da8455b827cd0e54207293e170201ebf1628a96c6972173c670d0353"
Sep 09 13:41:36.589757 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:41:36.589729    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ce677a7da8455b827cd0e54207293e170201ebf1628a96c6972173c670d0353\": container with ID starting with 0ce677a7da8455b827cd0e54207293e170201ebf1628a96c6972173c670d0353 not found: ID does not exist" containerID="0ce677a7da8455b827cd0e54207293e170201ebf1628a96c6972173c670d0353"
Sep 09 13:41:36.589846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:36.589763    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ce677a7da8455b827cd0e54207293e170201ebf1628a96c6972173c670d0353"} err="failed to get container status \"0ce677a7da8455b827cd0e54207293e170201ebf1628a96c6972173c670d0353\": rpc error: code = NotFound desc = could not find container \"0ce677a7da8455b827cd0e54207293e170201ebf1628a96c6972173c670d0353\": container with ID starting with 0ce677a7da8455b827cd0e54207293e170201ebf1628a96c6972173c670d0353 not found: ID does not exist"
Sep 09 13:41:37.151578 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:37.151545    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3c6d694-f3aa-4e21-ad3d-bfad107f2f31" path="/var/lib/kubelet/pods/b3c6d694-f3aa-4e21-ad3d-bfad107f2f31/volumes"
Sep 09 13:41:37.151973 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:37.151959    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8f84e7e-bf8b-403a-9eab-92ae34fb9357" path="/var/lib/kubelet/pods/e8f84e7e-bf8b-403a-9eab-92ae34fb9357/volumes"
Sep 09 13:41:37.495376 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:37.495336    2566 generic.go:356] "Generic (PLEG): container finished" podID="16c5d225-6c48-4b39-af15-495e9d7a9b4b" containerID="8f979c80f95fbd8f31bf0da6e1251e361d82f7f7153825376ec40ae2ded217e0" exitCode=0
Sep 09 13:41:37.495546 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:37.495413    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ecd209d-xnrmx" event={"ID":"16c5d225-6c48-4b39-af15-495e9d7a9b4b","Type":"ContainerDied","Data":"8f979c80f95fbd8f31bf0da6e1251e361d82f7f7153825376ec40ae2ded217e0"}
Sep 09 13:41:38.269125 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:38.269081    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9" podUID="865ae236-d377-4e05-a2e9-bbdd15856966" containerName="main" containerID="cri-o://2c77ea0f4f88ff721fe43444f314ee075e9fac64cfc9a0bd1b13e218b2cfbfa2" gracePeriod=45
Sep 09 13:41:38.505792 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:38.505758    2566 generic.go:356] "Generic (PLEG): container finished" podID="865ae236-d377-4e05-a2e9-bbdd15856966" containerID="2c77ea0f4f88ff721fe43444f314ee075e9fac64cfc9a0bd1b13e218b2cfbfa2" exitCode=0
Sep 09 13:41:38.505963 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:38.505856    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9" event={"ID":"865ae236-d377-4e05-a2e9-bbdd15856966","Type":"ContainerDied","Data":"2c77ea0f4f88ff721fe43444f314ee075e9fac64cfc9a0bd1b13e218b2cfbfa2"}
Sep 09 13:41:38.530603 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:38.530541    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"
Sep 09 13:41:38.560854 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:38.560787    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/865ae236-d377-4e05-a2e9-bbdd15856966-tls-certs\") pod \"865ae236-d377-4e05-a2e9-bbdd15856966\" (UID: \"865ae236-d377-4e05-a2e9-bbdd15856966\") "
Sep 09 13:41:38.561255 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:38.561145    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/865ae236-d377-4e05-a2e9-bbdd15856966-home\") pod \"865ae236-d377-4e05-a2e9-bbdd15856966\" (UID: \"865ae236-d377-4e05-a2e9-bbdd15856966\") "
Sep 09 13:41:38.561255 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:38.561229    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/865ae236-d377-4e05-a2e9-bbdd15856966-tmp-dir\") pod \"865ae236-d377-4e05-a2e9-bbdd15856966\" (UID: \"865ae236-d377-4e05-a2e9-bbdd15856966\") "
Sep 09 13:41:38.561406 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:38.561310    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/865ae236-d377-4e05-a2e9-bbdd15856966-model-cache\") pod \"865ae236-d377-4e05-a2e9-bbdd15856966\" (UID: \"865ae236-d377-4e05-a2e9-bbdd15856966\") "
Sep 09 13:41:38.561406 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:38.561341    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bt6p2\" (UniqueName: \"kubernetes.io/projected/865ae236-d377-4e05-a2e9-bbdd15856966-kube-api-access-bt6p2\") pod \"865ae236-d377-4e05-a2e9-bbdd15856966\" (UID: \"865ae236-d377-4e05-a2e9-bbdd15856966\") "
Sep 09 13:41:38.561406 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:38.561367    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/865ae236-d377-4e05-a2e9-bbdd15856966-dshm\") pod \"865ae236-d377-4e05-a2e9-bbdd15856966\" (UID: \"865ae236-d377-4e05-a2e9-bbdd15856966\") "
Sep 09 13:41:38.561574 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:38.561424    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865ae236-d377-4e05-a2e9-bbdd15856966-home" (OuterVolumeSpecName: "home") pod "865ae236-d377-4e05-a2e9-bbdd15856966" (UID: "865ae236-d377-4e05-a2e9-bbdd15856966"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:41:38.561974 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:38.561642    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865ae236-d377-4e05-a2e9-bbdd15856966-model-cache" (OuterVolumeSpecName: "model-cache") pod "865ae236-d377-4e05-a2e9-bbdd15856966" (UID: "865ae236-d377-4e05-a2e9-bbdd15856966"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:41:38.561974 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:38.561764    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/865ae236-d377-4e05-a2e9-bbdd15856966-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:41:38.561974 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:38.561784    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/865ae236-d377-4e05-a2e9-bbdd15856966-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:41:38.561974 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:38.561937    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865ae236-d377-4e05-a2e9-bbdd15856966-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "865ae236-d377-4e05-a2e9-bbdd15856966" (UID: "865ae236-d377-4e05-a2e9-bbdd15856966"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:41:38.563937 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:38.563908    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/865ae236-d377-4e05-a2e9-bbdd15856966-kube-api-access-bt6p2" (OuterVolumeSpecName: "kube-api-access-bt6p2") pod "865ae236-d377-4e05-a2e9-bbdd15856966" (UID: "865ae236-d377-4e05-a2e9-bbdd15856966"). InnerVolumeSpecName "kube-api-access-bt6p2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:41:38.564031 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:38.563949    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/865ae236-d377-4e05-a2e9-bbdd15856966-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "865ae236-d377-4e05-a2e9-bbdd15856966" (UID: "865ae236-d377-4e05-a2e9-bbdd15856966"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:41:38.564251 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:38.564222    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865ae236-d377-4e05-a2e9-bbdd15856966-dshm" (OuterVolumeSpecName: "dshm") pod "865ae236-d377-4e05-a2e9-bbdd15856966" (UID: "865ae236-d377-4e05-a2e9-bbdd15856966"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:41:38.639442 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:38.639414    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ecd209d-xnrmx"
Sep 09 13:41:38.663166 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:38.663130    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16c5d225-6c48-4b39-af15-495e9d7a9b4b-maas-api-service-ca\") pod \"16c5d225-6c48-4b39-af15-495e9d7a9b4b\" (UID: \"16c5d225-6c48-4b39-af15-495e9d7a9b4b\") "
Sep 09 13:41:38.663526 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:38.663495    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16c5d225-6c48-4b39-af15-495e9d7a9b4b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "16c5d225-6c48-4b39-af15-495e9d7a9b4b" (UID: "16c5d225-6c48-4b39-af15-495e9d7a9b4b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:41:38.663740 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:38.663591    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/865ae236-d377-4e05-a2e9-bbdd15856966-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:41:38.663740 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:38.663612    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16c5d225-6c48-4b39-af15-495e9d7a9b4b-maas-api-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:41:38.663740 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:38.663627    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt6p2\" (UniqueName: \"kubernetes.io/projected/865ae236-d377-4e05-a2e9-bbdd15856966-kube-api-access-bt6p2\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:41:38.663740 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:38.663641    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/865ae236-d377-4e05-a2e9-bbdd15856966-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:41:38.663740 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:38.663656    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/865ae236-d377-4e05-a2e9-bbdd15856966-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:41:39.512403 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:39.512311    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"
Sep 09 13:41:39.512403 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:39.512344    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9" event={"ID":"865ae236-d377-4e05-a2e9-bbdd15856966","Type":"ContainerDied","Data":"d917ea36b89f098e6ead7e07f5273bc022eee0f666b021493744864f0357c88f"}
Sep 09 13:41:39.512403 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:39.512396    2566 scope.go:117] "RemoveContainer" containerID="2c77ea0f4f88ff721fe43444f314ee075e9fac64cfc9a0bd1b13e218b2cfbfa2"
Sep 09 13:41:39.513951 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:39.513872    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ecd209d-xnrmx"
Sep 09 13:41:39.513951 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:39.513882    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ecd209d-xnrmx" event={"ID":"16c5d225-6c48-4b39-af15-495e9d7a9b4b","Type":"ContainerDied","Data":"bc9ca6f1b5296655b6f5a2f6b7dcb6e727d9a6e8a23b171c651805324d675712"}
Sep 09 13:41:39.513951 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:39.513922    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc9ca6f1b5296655b6f5a2f6b7dcb6e727d9a6e8a23b171c651805324d675712"
Sep 09 13:41:39.529388 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:39.529364    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"]
Sep 09 13:41:39.535186 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:39.535160    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-71c181/autores-model-71c181-kserve-699d5bb559-kpxw9"]
Sep 09 13:41:41.149913 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:41.149876    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="865ae236-d377-4e05-a2e9-bbdd15856966" path="/var/lib/kubelet/pods/865ae236-d377-4e05-a2e9-bbdd15856966/volumes"
Sep 09 13:41:43.107147 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:43.107111    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ecd209d-xnrmx"]
Sep 09 13:41:43.110232 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:43.110205    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4ecd209d-xnrmx"]
Sep 09 13:41:43.149390 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:43.149357    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16c5d225-6c48-4b39-af15-495e9d7a9b4b" path="/var/lib/kubelet/pods/16c5d225-6c48-4b39-af15-495e9d7a9b4b/volumes"
Sep 09 13:41:50.191721 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.191685    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"]
Sep 09 13:41:50.192213 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.192078    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3c6d694-f3aa-4e21-ad3d-bfad107f2f31" containerName="main"
Sep 09 13:41:50.192213 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.192089    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3c6d694-f3aa-4e21-ad3d-bfad107f2f31" containerName="main"
Sep 09 13:41:50.192213 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.192107    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16c5d225-6c48-4b39-af15-495e9d7a9b4b" containerName="revoke-keys"
Sep 09 13:41:50.192213 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.192113    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="16c5d225-6c48-4b39-af15-495e9d7a9b4b" containerName="revoke-keys"
Sep 09 13:41:50.192213 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.192121    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="865ae236-d377-4e05-a2e9-bbdd15856966" containerName="main"
Sep 09 13:41:50.192213 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.192126    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="865ae236-d377-4e05-a2e9-bbdd15856966" containerName="main"
Sep 09 13:41:50.192213 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.192181    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="865ae236-d377-4e05-a2e9-bbdd15856966" containerName="main"
Sep 09 13:41:50.192213 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.192191    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3c6d694-f3aa-4e21-ad3d-bfad107f2f31" containerName="main"
Sep 09 13:41:50.192213 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.192197    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="16c5d225-6c48-4b39-af15-495e9d7a9b4b" containerName="revoke-keys"
Sep 09 13:41:50.196682 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.196665    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"
Sep 09 13:41:50.199124 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.199095    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-093e38\"/\"openshift-service-ca.crt\""
Sep 09 13:41:50.199266 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.199140    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-093e38\"/\"test-model-093e38-kserve-self-signed-certs\""
Sep 09 13:41:50.199698 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.199680    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-093e38\"/\"kube-root-ca.crt\""
Sep 09 13:41:50.199873 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.199855    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-093e38\"/\"default-dockercfg-ctx8c\""
Sep 09 13:41:50.205499 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.205477    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"]
Sep 09 13:41:50.274628 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.274590    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64b9df97-cfd1-4916-8ebb-14f29c9452f5-dshm\") pod \"test-model-093e38-kserve-776878b8d5-f89zc\" (UID: \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\") " pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"
Sep 09 13:41:50.274628 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.274632    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64b9df97-cfd1-4916-8ebb-14f29c9452f5-home\") pod \"test-model-093e38-kserve-776878b8d5-f89zc\" (UID: \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\") " pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"
Sep 09 13:41:50.274898 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.274710    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64b9df97-cfd1-4916-8ebb-14f29c9452f5-tmp-dir\") pod \"test-model-093e38-kserve-776878b8d5-f89zc\" (UID: \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\") " pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"
Sep 09 13:41:50.274898 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.274747    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64b9df97-cfd1-4916-8ebb-14f29c9452f5-model-cache\") pod \"test-model-093e38-kserve-776878b8d5-f89zc\" (UID: \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\") " pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"
Sep 09 13:41:50.274898 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.274766    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvnfn\" (UniqueName: \"kubernetes.io/projected/64b9df97-cfd1-4916-8ebb-14f29c9452f5-kube-api-access-hvnfn\") pod \"test-model-093e38-kserve-776878b8d5-f89zc\" (UID: \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\") " pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"
Sep 09 13:41:50.274898 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.274841    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64b9df97-cfd1-4916-8ebb-14f29c9452f5-tls-certs\") pod \"test-model-093e38-kserve-776878b8d5-f89zc\" (UID: \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\") " pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"
Sep 09 13:41:50.375778 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.375733    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64b9df97-cfd1-4916-8ebb-14f29c9452f5-dshm\") pod \"test-model-093e38-kserve-776878b8d5-f89zc\" (UID: \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\") " pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"
Sep 09 13:41:50.375778 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.375779    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64b9df97-cfd1-4916-8ebb-14f29c9452f5-home\") pod \"test-model-093e38-kserve-776878b8d5-f89zc\" (UID: \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\") " pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"
Sep 09 13:41:50.376086 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.375844    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64b9df97-cfd1-4916-8ebb-14f29c9452f5-tmp-dir\") pod \"test-model-093e38-kserve-776878b8d5-f89zc\" (UID: \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\") " pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"
Sep 09 13:41:50.376086 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.375872    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64b9df97-cfd1-4916-8ebb-14f29c9452f5-model-cache\") pod \"test-model-093e38-kserve-776878b8d5-f89zc\" (UID: \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\") " pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"
Sep 09 13:41:50.376086 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.376001    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvnfn\" (UniqueName: \"kubernetes.io/projected/64b9df97-cfd1-4916-8ebb-14f29c9452f5-kube-api-access-hvnfn\") pod \"test-model-093e38-kserve-776878b8d5-f89zc\" (UID: \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\") " pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"
Sep 09 13:41:50.376086 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.376061    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64b9df97-cfd1-4916-8ebb-14f29c9452f5-tls-certs\") pod \"test-model-093e38-kserve-776878b8d5-f89zc\" (UID: \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\") " pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"
Sep 09 13:41:50.376325 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.376303    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64b9df97-cfd1-4916-8ebb-14f29c9452f5-model-cache\") pod \"test-model-093e38-kserve-776878b8d5-f89zc\" (UID: \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\") " pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"
Sep 09 13:41:50.376387 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.376330    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64b9df97-cfd1-4916-8ebb-14f29c9452f5-home\") pod \"test-model-093e38-kserve-776878b8d5-f89zc\" (UID: \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\") " pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"
Sep 09 13:41:50.376387 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.376357    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64b9df97-cfd1-4916-8ebb-14f29c9452f5-tmp-dir\") pod \"test-model-093e38-kserve-776878b8d5-f89zc\" (UID: \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\") " pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"
Sep 09 13:41:50.378152 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.378133    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64b9df97-cfd1-4916-8ebb-14f29c9452f5-dshm\") pod \"test-model-093e38-kserve-776878b8d5-f89zc\" (UID: \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\") " pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"
Sep 09 13:41:50.378463 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.378445    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64b9df97-cfd1-4916-8ebb-14f29c9452f5-tls-certs\") pod \"test-model-093e38-kserve-776878b8d5-f89zc\" (UID: \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\") " pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"
Sep 09 13:41:50.384588 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.384560    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvnfn\" (UniqueName: \"kubernetes.io/projected/64b9df97-cfd1-4916-8ebb-14f29c9452f5-kube-api-access-hvnfn\") pod \"test-model-093e38-kserve-776878b8d5-f89zc\" (UID: \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\") " pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"
Sep 09 13:41:50.507019 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.506929    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"
Sep 09 13:41:50.778453 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:50.778426    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"]
Sep 09 13:41:50.780030 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:41:50.780004    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64b9df97_cfd1_4916_8ebb_14f29c9452f5.slice/crio-50877bf38962c10885cfe51f1c3eee92d50dc8f9477ac156dc06fff47c790bb3 WatchSource:0}: Error finding container 50877bf38962c10885cfe51f1c3eee92d50dc8f9477ac156dc06fff47c790bb3: Status 404 returned error can't find the container with id 50877bf38962c10885cfe51f1c3eee92d50dc8f9477ac156dc06fff47c790bb3
Sep 09 13:41:51.579726 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:51.579684    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc" event={"ID":"64b9df97-cfd1-4916-8ebb-14f29c9452f5","Type":"ContainerStarted","Data":"2b479fa744515dd41fe92fcfac6af1ef379ebad7aea8617d411c21b4ac328bb2"}
Sep 09 13:41:51.580203 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:51.579733    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc" event={"ID":"64b9df97-cfd1-4916-8ebb-14f29c9452f5","Type":"ContainerStarted","Data":"50877bf38962c10885cfe51f1c3eee92d50dc8f9477ac156dc06fff47c790bb3"}
Sep 09 13:41:51.600407 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:41:51.600358    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc" podStartSLOduration=1.600341998 podStartE2EDuration="1.600341998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:41:51.599098007 +0000 UTC m=+944.958947833" watchObservedRunningTime="2026-09-09 13:41:51.600341998 +0000 UTC m=+944.960191824"
Sep 09 13:42:00.507888 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:00.507843    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"
Sep 09 13:42:00.508402 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:00.507901    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"
Sep 09 13:42:00.521044 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:00.521021    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"
Sep 09 13:42:00.637346 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:00.637310    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"
Sep 09 13:42:11.172487 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:11.172448    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"]
Sep 09 13:42:11.179768 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:11.179684    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz" podUID="d5398979-4319-4212-a4f7-83436cc58135" containerName="istio-proxy" containerID="cri-o://e373e1097f289368529b6365ca4a242c9fa3b9cec7b399bb59388e6cafa31cc4" gracePeriod=30
Sep 09 13:42:16.432689 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.432654    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:42:16.523247 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.523168    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-workload-certs\") pod \"d5398979-4319-4212-a4f7-83436cc58135\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") "
Sep 09 13:42:16.523247 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.523206    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-istio-data\") pod \"d5398979-4319-4212-a4f7-83436cc58135\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") "
Sep 09 13:42:16.523247 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.523240    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5398979-4319-4212-a4f7-83436cc58135-istio-token\") pod \"d5398979-4319-4212-a4f7-83436cc58135\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") "
Sep 09 13:42:16.523508 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.523270    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5398979-4319-4212-a4f7-83436cc58135-istiod-ca-cert\") pod \"d5398979-4319-4212-a4f7-83436cc58135\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") "
Sep 09 13:42:16.523508 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.523304    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-workload-socket\") pod \"d5398979-4319-4212-a4f7-83436cc58135\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") "
Sep 09 13:42:16.523508 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.523335    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5398979-4319-4212-a4f7-83436cc58135-istio-podinfo\") pod \"d5398979-4319-4212-a4f7-83436cc58135\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") "
Sep 09 13:42:16.523508 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.523433    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-istio-envoy\") pod \"d5398979-4319-4212-a4f7-83436cc58135\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") "
Sep 09 13:42:16.523508 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.523488    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-credential-socket\") pod \"d5398979-4319-4212-a4f7-83436cc58135\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") "
Sep 09 13:42:16.523508 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.523499    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d5398979-4319-4212-a4f7-83436cc58135" (UID: "d5398979-4319-4212-a4f7-83436cc58135"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:42:16.523825 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.523524    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wc9cs\" (UniqueName: \"kubernetes.io/projected/d5398979-4319-4212-a4f7-83436cc58135-kube-api-access-wc9cs\") pod \"d5398979-4319-4212-a4f7-83436cc58135\" (UID: \"d5398979-4319-4212-a4f7-83436cc58135\") "
Sep 09 13:42:16.523825 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.523578    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-istio-data" (OuterVolumeSpecName: "istio-data") pod "d5398979-4319-4212-a4f7-83436cc58135" (UID: "d5398979-4319-4212-a4f7-83436cc58135"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:42:16.523825 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.523781    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d5398979-4319-4212-a4f7-83436cc58135" (UID: "d5398979-4319-4212-a4f7-83436cc58135"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:42:16.524002 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.523869    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5398979-4319-4212-a4f7-83436cc58135-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d5398979-4319-4212-a4f7-83436cc58135" (UID: "d5398979-4319-4212-a4f7-83436cc58135"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:42:16.524002 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.523897    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:42:16.524002 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.523918    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:42:16.524002 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.523932    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:42:16.524209 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.524040    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d5398979-4319-4212-a4f7-83436cc58135" (UID: "d5398979-4319-4212-a4f7-83436cc58135"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:42:16.525704 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.525677    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5398979-4319-4212-a4f7-83436cc58135-istio-token" (OuterVolumeSpecName: "istio-token") pod "d5398979-4319-4212-a4f7-83436cc58135" (UID: "d5398979-4319-4212-a4f7-83436cc58135"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:42:16.525913 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.525885    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5398979-4319-4212-a4f7-83436cc58135-kube-api-access-wc9cs" (OuterVolumeSpecName: "kube-api-access-wc9cs") pod "d5398979-4319-4212-a4f7-83436cc58135" (UID: "d5398979-4319-4212-a4f7-83436cc58135"). InnerVolumeSpecName "kube-api-access-wc9cs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:42:16.526012 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.525903    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d5398979-4319-4212-a4f7-83436cc58135" (UID: "d5398979-4319-4212-a4f7-83436cc58135"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:42:16.526145 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.526122    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5398979-4319-4212-a4f7-83436cc58135-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d5398979-4319-4212-a4f7-83436cc58135" (UID: "d5398979-4319-4212-a4f7-83436cc58135"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 13:42:16.625142 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.625087    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:42:16.625142 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.625121    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc9cs\" (UniqueName: \"kubernetes.io/projected/d5398979-4319-4212-a4f7-83436cc58135-kube-api-access-wc9cs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:42:16.625142 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.625135    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5398979-4319-4212-a4f7-83436cc58135-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:42:16.625142 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.625148    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5398979-4319-4212-a4f7-83436cc58135-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:42:16.625493 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.625160    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5398979-4319-4212-a4f7-83436cc58135-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:42:16.625493 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.625173    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5398979-4319-4212-a4f7-83436cc58135-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:42:16.708721 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.708690    2566 generic.go:356] "Generic (PLEG): container finished" podID="d5398979-4319-4212-a4f7-83436cc58135" containerID="e373e1097f289368529b6365ca4a242c9fa3b9cec7b399bb59388e6cafa31cc4" exitCode=0
Sep 09 13:42:16.708915 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.708739    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"
Sep 09 13:42:16.708915 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.708771    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz" event={"ID":"d5398979-4319-4212-a4f7-83436cc58135","Type":"ContainerDied","Data":"e373e1097f289368529b6365ca4a242c9fa3b9cec7b399bb59388e6cafa31cc4"}
Sep 09 13:42:16.708915 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.708840    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz" event={"ID":"d5398979-4319-4212-a4f7-83436cc58135","Type":"ContainerDied","Data":"61a209743bc4c909fa0fb195e90fb1506ff54701f86c036dde80b238e4b55232"}
Sep 09 13:42:16.708915 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.708869    2566 scope.go:117] "RemoveContainer" containerID="e373e1097f289368529b6365ca4a242c9fa3b9cec7b399bb59388e6cafa31cc4"
Sep 09 13:42:16.718278 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.718262    2566 scope.go:117] "RemoveContainer" containerID="5cdb4203591f267ab9895e8e2dd48a020fdb1f4c1997b09bdd8e7581edff3451"
Sep 09 13:42:16.723705 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.723677    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6"]
Sep 09 13:42:16.724112 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.724096    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5398979-4319-4212-a4f7-83436cc58135" containerName="istio-proxy"
Sep 09 13:42:16.724206 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.724114    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5398979-4319-4212-a4f7-83436cc58135" containerName="istio-proxy"
Sep 09 13:42:16.724264 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.724204    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5398979-4319-4212-a4f7-83436cc58135" containerName="istio-proxy"
Sep 09 13:42:16.724264 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.724216    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5398979-4319-4212-a4f7-83436cc58135" containerName="istio-proxy"
Sep 09 13:42:16.729844 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.729802    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6"
Sep 09 13:42:16.730577 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.730556    2566 scope.go:117] "RemoveContainer" containerID="e373e1097f289368529b6365ca4a242c9fa3b9cec7b399bb59388e6cafa31cc4"
Sep 09 13:42:16.731297 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:42:16.731268    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e373e1097f289368529b6365ca4a242c9fa3b9cec7b399bb59388e6cafa31cc4\": container with ID starting with e373e1097f289368529b6365ca4a242c9fa3b9cec7b399bb59388e6cafa31cc4 not found: ID does not exist" containerID="e373e1097f289368529b6365ca4a242c9fa3b9cec7b399bb59388e6cafa31cc4"
Sep 09 13:42:16.731432 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.731411    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e373e1097f289368529b6365ca4a242c9fa3b9cec7b399bb59388e6cafa31cc4"} err="failed to get container status \"e373e1097f289368529b6365ca4a242c9fa3b9cec7b399bb59388e6cafa31cc4\": rpc error: code = NotFound desc = could not find container \"e373e1097f289368529b6365ca4a242c9fa3b9cec7b399bb59388e6cafa31cc4\": container with ID starting with e373e1097f289368529b6365ca4a242c9fa3b9cec7b399bb59388e6cafa31cc4 not found: ID does not exist"
Sep 09 13:42:16.731522 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.731512    2566 scope.go:117] "RemoveContainer" containerID="5cdb4203591f267ab9895e8e2dd48a020fdb1f4c1997b09bdd8e7581edff3451"
Sep 09 13:42:16.731981 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:42:16.731945    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cdb4203591f267ab9895e8e2dd48a020fdb1f4c1997b09bdd8e7581edff3451\": container with ID starting with 5cdb4203591f267ab9895e8e2dd48a020fdb1f4c1997b09bdd8e7581edff3451 not found: ID does not exist" containerID="5cdb4203591f267ab9895e8e2dd48a020fdb1f4c1997b09bdd8e7581edff3451"
Sep 09 13:42:16.732139 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.732109    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cdb4203591f267ab9895e8e2dd48a020fdb1f4c1997b09bdd8e7581edff3451"} err="failed to get container status \"5cdb4203591f267ab9895e8e2dd48a020fdb1f4c1997b09bdd8e7581edff3451\": rpc error: code = NotFound desc = could not find container \"5cdb4203591f267ab9895e8e2dd48a020fdb1f4c1997b09bdd8e7581edff3451\": container with ID starting with 5cdb4203591f267ab9895e8e2dd48a020fdb1f4c1997b09bdd8e7581edff3451 not found: ID does not exist"
Sep 09 13:42:16.733277 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.733257    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-c5f375\""
Sep 09 13:42:16.733366 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.733283    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-c5f375-dockercfg-kt9tx\""
Sep 09 13:42:16.735650 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.735631    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"]
Sep 09 13:42:16.737706 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.737686    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6"]
Sep 09 13:42:16.743201 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.743177    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-71c181-openshift-default-5b75999664-9s7bz"]
Sep 09 13:42:16.832877 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.832838    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzxml\" (UniqueName: \"kubernetes.io/projected/a6cc7b00-86ad-4f3f-8e9f-9a62a736554b-kube-api-access-vzxml\") pod \"payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6\" (UID: \"a6cc7b00-86ad-4f3f-8e9f-9a62a736554b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6"
Sep 09 13:42:16.833057 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.832972    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6cc7b00-86ad-4f3f-8e9f-9a62a736554b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6\" (UID: \"a6cc7b00-86ad-4f3f-8e9f-9a62a736554b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6"
Sep 09 13:42:16.934455 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.934415    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6cc7b00-86ad-4f3f-8e9f-9a62a736554b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6\" (UID: \"a6cc7b00-86ad-4f3f-8e9f-9a62a736554b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6"
Sep 09 13:42:16.934668 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.934532    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzxml\" (UniqueName: \"kubernetes.io/projected/a6cc7b00-86ad-4f3f-8e9f-9a62a736554b-kube-api-access-vzxml\") pod \"payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6\" (UID: \"a6cc7b00-86ad-4f3f-8e9f-9a62a736554b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6"
Sep 09 13:42:16.935274 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.935249    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6cc7b00-86ad-4f3f-8e9f-9a62a736554b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6\" (UID: \"a6cc7b00-86ad-4f3f-8e9f-9a62a736554b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6"
Sep 09 13:42:16.942955 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:16.942933    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzxml\" (UniqueName: \"kubernetes.io/projected/a6cc7b00-86ad-4f3f-8e9f-9a62a736554b-kube-api-access-vzxml\") pod \"payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6\" (UID: \"a6cc7b00-86ad-4f3f-8e9f-9a62a736554b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6"
Sep 09 13:42:17.042231 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:17.042190    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6"
Sep 09 13:42:17.150048 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:17.150017    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5398979-4319-4212-a4f7-83436cc58135" path="/var/lib/kubelet/pods/d5398979-4319-4212-a4f7-83436cc58135/volumes"
Sep 09 13:42:17.315208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:17.315175    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6"]
Sep 09 13:42:17.317275 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:42:17.317249    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6cc7b00_86ad_4f3f_8e9f_9a62a736554b.slice/crio-0e054e08103726de990ae7889c965e3729bc7da4f78b7347dd8213b6d9c64c9c WatchSource:0}: Error finding container 0e054e08103726de990ae7889c965e3729bc7da4f78b7347dd8213b6d9c64c9c: Status 404 returned error can't find the container with id 0e054e08103726de990ae7889c965e3729bc7da4f78b7347dd8213b6d9c64c9c
Sep 09 13:42:17.717009 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:17.716913    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6" event={"ID":"a6cc7b00-86ad-4f3f-8e9f-9a62a736554b","Type":"ContainerStarted","Data":"28bb6355685d93f49dbdff5ef835818bbbe17c47a7a4d64c08bad6135fda72f5"}
Sep 09 13:42:17.717009 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:17.716973    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6" event={"ID":"a6cc7b00-86ad-4f3f-8e9f-9a62a736554b","Type":"ContainerStarted","Data":"0e054e08103726de990ae7889c965e3729bc7da4f78b7347dd8213b6d9c64c9c"}
Sep 09 13:42:17.717409 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:17.717048    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6"
Sep 09 13:42:17.738177 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:17.738118    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6" podStartSLOduration=1.738098317 podStartE2EDuration="1.738098317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:42:17.735575828 +0000 UTC m=+971.095425653" watchObservedRunningTime="2026-09-09 13:42:17.738098317 +0000 UTC m=+971.097948146"
Sep 09 13:42:28.724704 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:42:28.724664    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6"
Sep 09 13:43:07.071682 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.071650    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"]
Sep 09 13:43:07.072202 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.072102    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5398979-4319-4212-a4f7-83436cc58135" containerName="istio-proxy"
Sep 09 13:43:07.072202 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.072116    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5398979-4319-4212-a4f7-83436cc58135" containerName="istio-proxy"
Sep 09 13:43:07.074953 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.074937    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"
Sep 09 13:43:07.077899 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.077874    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c5f375\"/\"kube-root-ca.crt\""
Sep 09 13:43:07.078042 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.077937    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c5f375\"/\"explicit-model-c5f375-kserve-self-signed-certs\""
Sep 09 13:43:07.078042 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.077966    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c5f375\"/\"openshift-service-ca.crt\""
Sep 09 13:43:07.078147 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.078048    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c5f375\"/\"default-dockercfg-b9kht\""
Sep 09 13:43:07.085377 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.085355    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"]
Sep 09 13:43:07.111467 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.111434    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9305809d-30ff-4c5c-b588-7aa128af0c12-dshm\") pod \"explicit-model-c5f375-kserve-68d85d959-ht6rr\" (UID: \"9305809d-30ff-4c5c-b588-7aa128af0c12\") " pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"
Sep 09 13:43:07.111611 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.111557    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9305809d-30ff-4c5c-b588-7aa128af0c12-home\") pod \"explicit-model-c5f375-kserve-68d85d959-ht6rr\" (UID: \"9305809d-30ff-4c5c-b588-7aa128af0c12\") " pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"
Sep 09 13:43:07.111611 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.111592    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9305809d-30ff-4c5c-b588-7aa128af0c12-model-cache\") pod \"explicit-model-c5f375-kserve-68d85d959-ht6rr\" (UID: \"9305809d-30ff-4c5c-b588-7aa128af0c12\") " pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"
Sep 09 13:43:07.111690 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.111675    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9305809d-30ff-4c5c-b588-7aa128af0c12-tls-certs\") pod \"explicit-model-c5f375-kserve-68d85d959-ht6rr\" (UID: \"9305809d-30ff-4c5c-b588-7aa128af0c12\") " pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"
Sep 09 13:43:07.111725 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.111701    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjdb6\" (UniqueName: \"kubernetes.io/projected/9305809d-30ff-4c5c-b588-7aa128af0c12-kube-api-access-jjdb6\") pod \"explicit-model-c5f375-kserve-68d85d959-ht6rr\" (UID: \"9305809d-30ff-4c5c-b588-7aa128af0c12\") " pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"
Sep 09 13:43:07.111779 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.111764    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9305809d-30ff-4c5c-b588-7aa128af0c12-tmp-dir\") pod \"explicit-model-c5f375-kserve-68d85d959-ht6rr\" (UID: \"9305809d-30ff-4c5c-b588-7aa128af0c12\") " pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"
Sep 09 13:43:07.212844 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.212798    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9305809d-30ff-4c5c-b588-7aa128af0c12-home\") pod \"explicit-model-c5f375-kserve-68d85d959-ht6rr\" (UID: \"9305809d-30ff-4c5c-b588-7aa128af0c12\") " pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"
Sep 09 13:43:07.213015 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.212854    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9305809d-30ff-4c5c-b588-7aa128af0c12-model-cache\") pod \"explicit-model-c5f375-kserve-68d85d959-ht6rr\" (UID: \"9305809d-30ff-4c5c-b588-7aa128af0c12\") " pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"
Sep 09 13:43:07.213015 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.212910    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9305809d-30ff-4c5c-b588-7aa128af0c12-tls-certs\") pod \"explicit-model-c5f375-kserve-68d85d959-ht6rr\" (UID: \"9305809d-30ff-4c5c-b588-7aa128af0c12\") " pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"
Sep 09 13:43:07.213152 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.213021    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjdb6\" (UniqueName: \"kubernetes.io/projected/9305809d-30ff-4c5c-b588-7aa128af0c12-kube-api-access-jjdb6\") pod \"explicit-model-c5f375-kserve-68d85d959-ht6rr\" (UID: \"9305809d-30ff-4c5c-b588-7aa128af0c12\") " pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"
Sep 09 13:43:07.213152 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.213080    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9305809d-30ff-4c5c-b588-7aa128af0c12-tmp-dir\") pod \"explicit-model-c5f375-kserve-68d85d959-ht6rr\" (UID: \"9305809d-30ff-4c5c-b588-7aa128af0c12\") " pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"
Sep 09 13:43:07.213152 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.213121    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9305809d-30ff-4c5c-b588-7aa128af0c12-home\") pod \"explicit-model-c5f375-kserve-68d85d959-ht6rr\" (UID: \"9305809d-30ff-4c5c-b588-7aa128af0c12\") " pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"
Sep 09 13:43:07.213152 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.213125    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9305809d-30ff-4c5c-b588-7aa128af0c12-dshm\") pod \"explicit-model-c5f375-kserve-68d85d959-ht6rr\" (UID: \"9305809d-30ff-4c5c-b588-7aa128af0c12\") " pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"
Sep 09 13:43:07.213366 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.213206    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9305809d-30ff-4c5c-b588-7aa128af0c12-model-cache\") pod \"explicit-model-c5f375-kserve-68d85d959-ht6rr\" (UID: \"9305809d-30ff-4c5c-b588-7aa128af0c12\") " pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"
Sep 09 13:43:07.213422 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.213394    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9305809d-30ff-4c5c-b588-7aa128af0c12-tmp-dir\") pod \"explicit-model-c5f375-kserve-68d85d959-ht6rr\" (UID: \"9305809d-30ff-4c5c-b588-7aa128af0c12\") " pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"
Sep 09 13:43:07.215222 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.215201    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9305809d-30ff-4c5c-b588-7aa128af0c12-dshm\") pod \"explicit-model-c5f375-kserve-68d85d959-ht6rr\" (UID: \"9305809d-30ff-4c5c-b588-7aa128af0c12\") " pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"
Sep 09 13:43:07.215452 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.215438    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9305809d-30ff-4c5c-b588-7aa128af0c12-tls-certs\") pod \"explicit-model-c5f375-kserve-68d85d959-ht6rr\" (UID: \"9305809d-30ff-4c5c-b588-7aa128af0c12\") " pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"
Sep 09 13:43:07.222884 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.221364    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjdb6\" (UniqueName: \"kubernetes.io/projected/9305809d-30ff-4c5c-b588-7aa128af0c12-kube-api-access-jjdb6\") pod \"explicit-model-c5f375-kserve-68d85d959-ht6rr\" (UID: \"9305809d-30ff-4c5c-b588-7aa128af0c12\") " pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"
Sep 09 13:43:07.385093 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.385054    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"
Sep 09 13:43:07.661698 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.661615    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"]
Sep 09 13:43:07.664906 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:43:07.664875    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9305809d_30ff_4c5c_b588_7aa128af0c12.slice/crio-df44290a3d2d14551c75bb0b3f8eebaf852a25bbdb3e61d2594481d65ab04429 WatchSource:0}: Error finding container df44290a3d2d14551c75bb0b3f8eebaf852a25bbdb3e61d2594481d65ab04429: Status 404 returned error can't find the container with id df44290a3d2d14551c75bb0b3f8eebaf852a25bbdb3e61d2594481d65ab04429
Sep 09 13:43:07.968950 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.968855    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr" event={"ID":"9305809d-30ff-4c5c-b588-7aa128af0c12","Type":"ContainerStarted","Data":"0d3a06aa8d879743a3f96775eed6bba7e9811ae3fdd987e07f234c9d752a6173"}
Sep 09 13:43:07.968950 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.968896    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr" event={"ID":"9305809d-30ff-4c5c-b588-7aa128af0c12","Type":"ContainerStarted","Data":"df44290a3d2d14551c75bb0b3f8eebaf852a25bbdb3e61d2594481d65ab04429"}
Sep 09 13:43:07.987707 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:07.987647    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr" podStartSLOduration=0.987631966 podStartE2EDuration="987.631966ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:43:07.985260739 +0000 UTC m=+1021.345110565" watchObservedRunningTime="2026-09-09 13:43:07.987631966 +0000 UTC m=+1021.347481793"
Sep 09 13:43:09.440092 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:09.440057    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"]
Sep 09 13:43:17.386109 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:17.386069    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"
Sep 09 13:43:17.386620 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:17.386127    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"
Sep 09 13:43:17.398748 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:17.398718    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"
Sep 09 13:43:18.027585 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:18.027555    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"
Sep 09 13:43:19.292132 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.292098    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"]
Sep 09 13:43:19.295590 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.295575    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.298141 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.298118    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-905b058d-openshift-default-dockercfg-9ptfm\""
Sep 09 13:43:19.320465 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.320445    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"]
Sep 09 13:43:19.431879 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.431837    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-workload-socket\") pod \"e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") " pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.432064 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.431987    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxrmf\" (UniqueName: \"kubernetes.io/projected/2323ec19-7ff6-470a-b880-6cde1ce7104f-kube-api-access-lxrmf\") pod \"e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") " pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.432064 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.432020    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-istio-data\") pod \"e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") " pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.432064 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.432049    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-credential-socket\") pod \"e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") " pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.432248 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.432099    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-workload-certs\") pod \"e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") " pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.432248 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.432144    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2323ec19-7ff6-470a-b880-6cde1ce7104f-istio-token\") pod \"e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") " pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.432248 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.432177    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2323ec19-7ff6-470a-b880-6cde1ce7104f-istio-podinfo\") pod \"e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") " pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.432248 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.432224    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-istio-envoy\") pod \"e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") " pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.432406 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.432253    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2323ec19-7ff6-470a-b880-6cde1ce7104f-istiod-ca-cert\") pod \"e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") " pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.532805 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.532770    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxrmf\" (UniqueName: \"kubernetes.io/projected/2323ec19-7ff6-470a-b880-6cde1ce7104f-kube-api-access-lxrmf\") pod \"e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") " pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.533045 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.532850    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-istio-data\") pod \"e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") " pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.533045 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.532887    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-credential-socket\") pod \"e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") " pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.533045 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.532908    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-workload-certs\") pod \"e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") " pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.533045 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.532932    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2323ec19-7ff6-470a-b880-6cde1ce7104f-istio-token\") pod \"e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") " pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.533045 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.532970    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2323ec19-7ff6-470a-b880-6cde1ce7104f-istio-podinfo\") pod \"e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") " pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.533045 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.532998    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-istio-envoy\") pod \"e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") " pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.533045 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.533026    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2323ec19-7ff6-470a-b880-6cde1ce7104f-istiod-ca-cert\") pod \"e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") " pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.533397 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.533130    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-workload-socket\") pod \"e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") " pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.533397 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.533351    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-credential-socket\") pod \"e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") " pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.533397 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.533373    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-workload-certs\") pod \"e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") " pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.533568 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.533440    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-istio-data\") pod \"e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") " pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.533631 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.533608    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-workload-socket\") pod \"e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") " pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.533879 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.533859    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2323ec19-7ff6-470a-b880-6cde1ce7104f-istiod-ca-cert\") pod \"e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") " pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.535331 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.535313    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-istio-envoy\") pod \"e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") " pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.535482 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.535464    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2323ec19-7ff6-470a-b880-6cde1ce7104f-istio-podinfo\") pod \"e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") " pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.541353 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.541331    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2323ec19-7ff6-470a-b880-6cde1ce7104f-istio-token\") pod \"e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") " pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.541735 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.541717    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxrmf\" (UniqueName: \"kubernetes.io/projected/2323ec19-7ff6-470a-b880-6cde1ce7104f-kube-api-access-lxrmf\") pod \"e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") " pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.606404 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.606373    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:19.883718 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.883693    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"]
Sep 09 13:43:19.885906 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:43:19.885879    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2323ec19_7ff6_470a_b880_6cde1ce7104f.slice/crio-fbaa48043d1ad911c80a9c5e1453374e3a473ce470ffa0cf179a09522d821f37 WatchSource:0}: Error finding container fbaa48043d1ad911c80a9c5e1453374e3a473ce470ffa0cf179a09522d821f37: Status 404 returned error can't find the container with id fbaa48043d1ad911c80a9c5e1453374e3a473ce470ffa0cf179a09522d821f37
Sep 09 13:43:19.888522 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.888490    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:43:19.888605 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.888554    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:43:19.888605 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:19.888583    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:43:20.028795 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:20.028757    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww" event={"ID":"2323ec19-7ff6-470a-b880-6cde1ce7104f","Type":"ContainerStarted","Data":"5671894e44006add60f7115d25f05941b92c1b38c93e4e7cee21e03e99e5ded3"}
Sep 09 13:43:20.028980 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:20.028844    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww" event={"ID":"2323ec19-7ff6-470a-b880-6cde1ce7104f","Type":"ContainerStarted","Data":"fbaa48043d1ad911c80a9c5e1453374e3a473ce470ffa0cf179a09522d821f37"}
Sep 09 13:43:20.056067 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:20.056007    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww" podStartSLOduration=1.055985761 podStartE2EDuration="1.055985761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:43:20.052066869 +0000 UTC m=+1033.411916695" watchObservedRunningTime="2026-09-09 13:43:20.055985761 +0000 UTC m=+1033.415835588"
Sep 09 13:43:20.607546 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:20.607507    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:20.612527 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:20.612503    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:21.035917 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:21.035829    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:21.036825 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:21.036796    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:24.459945 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:24.459892    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn" podUID="585e4180-f184-462e-8325-100634a9524c" containerName="main" containerID="cri-o://8a225d605960898ab23e45127db7f0b8fa73a605700d09f8458c578cf3aafb4f" gracePeriod=45
Sep 09 13:43:24.708134 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:24.708109    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"
Sep 09 13:43:24.788490 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:24.788396    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcjrk\" (UniqueName: \"kubernetes.io/projected/585e4180-f184-462e-8325-100634a9524c-kube-api-access-xcjrk\") pod \"585e4180-f184-462e-8325-100634a9524c\" (UID: \"585e4180-f184-462e-8325-100634a9524c\") "
Sep 09 13:43:24.788490 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:24.788462    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/585e4180-f184-462e-8325-100634a9524c-tmp-dir\") pod \"585e4180-f184-462e-8325-100634a9524c\" (UID: \"585e4180-f184-462e-8325-100634a9524c\") "
Sep 09 13:43:24.788490 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:24.788490    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/585e4180-f184-462e-8325-100634a9524c-home\") pod \"585e4180-f184-462e-8325-100634a9524c\" (UID: \"585e4180-f184-462e-8325-100634a9524c\") "
Sep 09 13:43:24.788781 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:24.788541    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/585e4180-f184-462e-8325-100634a9524c-dshm\") pod \"585e4180-f184-462e-8325-100634a9524c\" (UID: \"585e4180-f184-462e-8325-100634a9524c\") "
Sep 09 13:43:24.788781 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:24.788561    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/585e4180-f184-462e-8325-100634a9524c-tls-certs\") pod \"585e4180-f184-462e-8325-100634a9524c\" (UID: \"585e4180-f184-462e-8325-100634a9524c\") "
Sep 09 13:43:24.788781 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:24.788593    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/585e4180-f184-462e-8325-100634a9524c-model-cache\") pod \"585e4180-f184-462e-8325-100634a9524c\" (UID: \"585e4180-f184-462e-8325-100634a9524c\") "
Sep 09 13:43:24.788781 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:24.788725    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/585e4180-f184-462e-8325-100634a9524c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "585e4180-f184-462e-8325-100634a9524c" (UID: "585e4180-f184-462e-8325-100634a9524c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:43:24.789017 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:24.788904    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/585e4180-f184-462e-8325-100634a9524c-home" (OuterVolumeSpecName: "home") pod "585e4180-f184-462e-8325-100634a9524c" (UID: "585e4180-f184-462e-8325-100634a9524c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:43:24.789017 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:24.788938    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/585e4180-f184-462e-8325-100634a9524c-model-cache" (OuterVolumeSpecName: "model-cache") pod "585e4180-f184-462e-8325-100634a9524c" (UID: "585e4180-f184-462e-8325-100634a9524c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:43:24.789017 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:24.789007    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/585e4180-f184-462e-8325-100634a9524c-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:24.789140 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:24.789024    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/585e4180-f184-462e-8325-100634a9524c-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:24.789140 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:24.789037    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/585e4180-f184-462e-8325-100634a9524c-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:24.790846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:24.790793    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/585e4180-f184-462e-8325-100634a9524c-kube-api-access-xcjrk" (OuterVolumeSpecName: "kube-api-access-xcjrk") pod "585e4180-f184-462e-8325-100634a9524c" (UID: "585e4180-f184-462e-8325-100634a9524c"). InnerVolumeSpecName "kube-api-access-xcjrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:43:24.790846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:24.790836    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/585e4180-f184-462e-8325-100634a9524c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "585e4180-f184-462e-8325-100634a9524c" (UID: "585e4180-f184-462e-8325-100634a9524c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:43:24.790992 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:24.790866    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/585e4180-f184-462e-8325-100634a9524c-dshm" (OuterVolumeSpecName: "dshm") pod "585e4180-f184-462e-8325-100634a9524c" (UID: "585e4180-f184-462e-8325-100634a9524c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:43:24.890344 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:24.890309    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/585e4180-f184-462e-8325-100634a9524c-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:24.890344 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:24.890338    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/585e4180-f184-462e-8325-100634a9524c-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:24.890539 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:24.890353    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcjrk\" (UniqueName: \"kubernetes.io/projected/585e4180-f184-462e-8325-100634a9524c-kube-api-access-xcjrk\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:25.059192 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:25.059101    2566 generic.go:356] "Generic (PLEG): container finished" podID="585e4180-f184-462e-8325-100634a9524c" containerID="8a225d605960898ab23e45127db7f0b8fa73a605700d09f8458c578cf3aafb4f" exitCode=0
Sep 09 13:43:25.059192 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:25.059147    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"
Sep 09 13:43:25.059383 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:25.059185    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn" event={"ID":"585e4180-f184-462e-8325-100634a9524c","Type":"ContainerDied","Data":"8a225d605960898ab23e45127db7f0b8fa73a605700d09f8458c578cf3aafb4f"}
Sep 09 13:43:25.059383 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:25.059231    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn" event={"ID":"585e4180-f184-462e-8325-100634a9524c","Type":"ContainerDied","Data":"a6c4c099d49662146a4612f5f21e9979e3f38dc928500ad04097968f05423af0"}
Sep 09 13:43:25.059383 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:25.059249    2566 scope.go:117] "RemoveContainer" containerID="8a225d605960898ab23e45127db7f0b8fa73a605700d09f8458c578cf3aafb4f"
Sep 09 13:43:25.068097 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:25.068053    2566 scope.go:117] "RemoveContainer" containerID="8a225d605960898ab23e45127db7f0b8fa73a605700d09f8458c578cf3aafb4f"
Sep 09 13:43:25.068331 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:43:25.068310    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a225d605960898ab23e45127db7f0b8fa73a605700d09f8458c578cf3aafb4f\": container with ID starting with 8a225d605960898ab23e45127db7f0b8fa73a605700d09f8458c578cf3aafb4f not found: ID does not exist" containerID="8a225d605960898ab23e45127db7f0b8fa73a605700d09f8458c578cf3aafb4f"
Sep 09 13:43:25.068386 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:25.068339    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a225d605960898ab23e45127db7f0b8fa73a605700d09f8458c578cf3aafb4f"} err="failed to get container status \"8a225d605960898ab23e45127db7f0b8fa73a605700d09f8458c578cf3aafb4f\": rpc error: code = NotFound desc = could not find container \"8a225d605960898ab23e45127db7f0b8fa73a605700d09f8458c578cf3aafb4f\": container with ID starting with 8a225d605960898ab23e45127db7f0b8fa73a605700d09f8458c578cf3aafb4f not found: ID does not exist"
Sep 09 13:43:25.079531 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:25.079503    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"]
Sep 09 13:43:25.085245 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:25.085224    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-4f59a0/test-model-4f59a0-kserve-57bf467579-ldjfn"]
Sep 09 13:43:25.149566 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:25.149538    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="585e4180-f184-462e-8325-100634a9524c" path="/var/lib/kubelet/pods/585e4180-f184-462e-8325-100634a9524c/volumes"
Sep 09 13:43:30.812017 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:30.811976    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"]
Sep 09 13:43:30.920949 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:30.920916    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f59a0-cp45c"]
Sep 09 13:43:30.921295 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:30.921284    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="585e4180-f184-462e-8325-100634a9524c" containerName="main"
Sep 09 13:43:30.921340 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:30.921297    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="585e4180-f184-462e-8325-100634a9524c" containerName="main"
Sep 09 13:43:30.921375 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:30.921355    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="585e4180-f184-462e-8325-100634a9524c" containerName="main"
Sep 09 13:43:30.925733 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:30.925714    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f59a0-cp45c"
Sep 09 13:43:30.927743 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:30.927712    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lh487\""
Sep 09 13:43:30.927893 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:30.927863    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 13:43:30.931866 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:30.931842    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f59a0-cp45c"]
Sep 09 13:43:31.045942 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:31.045909    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0914355c-07e5-4492-9d15-ef1feed2decc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4f59a0-cp45c\" (UID: \"0914355c-07e5-4492-9d15-ef1feed2decc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f59a0-cp45c"
Sep 09 13:43:31.146401 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:31.146370    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0914355c-07e5-4492-9d15-ef1feed2decc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4f59a0-cp45c\" (UID: \"0914355c-07e5-4492-9d15-ef1feed2decc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f59a0-cp45c"
Sep 09 13:43:31.147003 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:31.146984    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0914355c-07e5-4492-9d15-ef1feed2decc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4f59a0-cp45c\" (UID: \"0914355c-07e5-4492-9d15-ef1feed2decc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f59a0-cp45c"
Sep 09 13:43:31.238181 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:31.238147    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f59a0-cp45c"
Sep 09 13:43:31.514607 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:31.514578    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f59a0-cp45c"]
Sep 09 13:43:31.516404 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:43:31.516380    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0914355c_07e5_4492_9d15_ef1feed2decc.slice/crio-8d33ebc5e2ba4af50e71200d7fa1ad6baf85cd52bc50c0fae6ae85a7d668ac78 WatchSource:0}: Error finding container 8d33ebc5e2ba4af50e71200d7fa1ad6baf85cd52bc50c0fae6ae85a7d668ac78: Status 404 returned error can't find the container with id 8d33ebc5e2ba4af50e71200d7fa1ad6baf85cd52bc50c0fae6ae85a7d668ac78
Sep 09 13:43:31.968214 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:31.968181    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"]
Sep 09 13:43:32.076386 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:32.076294    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c5f375-wdjbr"]
Sep 09 13:43:32.079844 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:32.079798    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c5f375-wdjbr"
Sep 09 13:43:32.089024 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:32.088997    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c5f375-wdjbr"]
Sep 09 13:43:32.098452 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:32.098419    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f59a0-cp45c" event={"ID":"0914355c-07e5-4492-9d15-ef1feed2decc","Type":"ContainerStarted","Data":"e0a1b88a858abb2c2d3ddaa9c7e455567a50eb636fe77e2a89e5b71b7fd06855"}
Sep 09 13:43:32.098583 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:32.098481    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f59a0-cp45c" event={"ID":"0914355c-07e5-4492-9d15-ef1feed2decc","Type":"ContainerStarted","Data":"8d33ebc5e2ba4af50e71200d7fa1ad6baf85cd52bc50c0fae6ae85a7d668ac78"}
Sep 09 13:43:32.117094 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:32.117045    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f59a0-cp45c" podStartSLOduration=2.117031502 podStartE2EDuration="2.117031502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:43:32.114958735 +0000 UTC m=+1045.474808563" watchObservedRunningTime="2026-09-09 13:43:32.117031502 +0000 UTC m=+1045.476881328"
Sep 09 13:43:32.256758 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:32.256713    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2224db8e-bdb5-46e1-87a4-bfc2420eafcb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c5f375-wdjbr\" (UID: \"2224db8e-bdb5-46e1-87a4-bfc2420eafcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c5f375-wdjbr"
Sep 09 13:43:32.358072 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:32.358035    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2224db8e-bdb5-46e1-87a4-bfc2420eafcb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c5f375-wdjbr\" (UID: \"2224db8e-bdb5-46e1-87a4-bfc2420eafcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c5f375-wdjbr"
Sep 09 13:43:32.358709 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:32.358688    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2224db8e-bdb5-46e1-87a4-bfc2420eafcb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c5f375-wdjbr\" (UID: \"2224db8e-bdb5-46e1-87a4-bfc2420eafcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c5f375-wdjbr"
Sep 09 13:43:32.391293 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:32.391263    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c5f375-wdjbr"
Sep 09 13:43:32.668226 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:32.668145    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c5f375-wdjbr"]
Sep 09 13:43:32.674069 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:43:32.674037    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2224db8e_bdb5_46e1_87a4_bfc2420eafcb.slice/crio-ebcbbba5272f179cd91f26f54618fe480aff6786173c8ebcc6d1d211836f1f80 WatchSource:0}: Error finding container ebcbbba5272f179cd91f26f54618fe480aff6786173c8ebcc6d1d211836f1f80: Status 404 returned error can't find the container with id ebcbbba5272f179cd91f26f54618fe480aff6786173c8ebcc6d1d211836f1f80
Sep 09 13:43:33.105524 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:33.105485    2566 generic.go:356] "Generic (PLEG): container finished" podID="0914355c-07e5-4492-9d15-ef1feed2decc" containerID="e0a1b88a858abb2c2d3ddaa9c7e455567a50eb636fe77e2a89e5b71b7fd06855" exitCode=0
Sep 09 13:43:33.105958 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:33.105568    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f59a0-cp45c" event={"ID":"0914355c-07e5-4492-9d15-ef1feed2decc","Type":"ContainerDied","Data":"e0a1b88a858abb2c2d3ddaa9c7e455567a50eb636fe77e2a89e5b71b7fd06855"}
Sep 09 13:43:33.107143 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:33.107118    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c5f375-wdjbr" event={"ID":"2224db8e-bdb5-46e1-87a4-bfc2420eafcb","Type":"ContainerStarted","Data":"6a456925b31022ae28df8f595325ea9c9c3a8ff147de60e50176eb6bea0ee975"}
Sep 09 13:43:33.107257 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:33.107159    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c5f375-wdjbr" event={"ID":"2224db8e-bdb5-46e1-87a4-bfc2420eafcb","Type":"ContainerStarted","Data":"ebcbbba5272f179cd91f26f54618fe480aff6786173c8ebcc6d1d211836f1f80"}
Sep 09 13:43:33.142510 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:33.142464    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c5f375-wdjbr" podStartSLOduration=1.142451141 podStartE2EDuration="1.142451141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:43:33.140483704 +0000 UTC m=+1046.500333530" watchObservedRunningTime="2026-09-09 13:43:33.142451141 +0000 UTC m=+1046.502300967"
Sep 09 13:43:34.118141 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:34.118100    2566 generic.go:356] "Generic (PLEG): container finished" podID="2224db8e-bdb5-46e1-87a4-bfc2420eafcb" containerID="6a456925b31022ae28df8f595325ea9c9c3a8ff147de60e50176eb6bea0ee975" exitCode=0
Sep 09 13:43:34.118606 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:34.118204    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c5f375-wdjbr" event={"ID":"2224db8e-bdb5-46e1-87a4-bfc2420eafcb","Type":"ContainerDied","Data":"6a456925b31022ae28df8f595325ea9c9c3a8ff147de60e50176eb6bea0ee975"}
Sep 09 13:43:34.269736 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:34.269704    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f59a0-cp45c"
Sep 09 13:43:34.378425 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:34.378333    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0914355c-07e5-4492-9d15-ef1feed2decc-maas-api-service-ca\") pod \"0914355c-07e5-4492-9d15-ef1feed2decc\" (UID: \"0914355c-07e5-4492-9d15-ef1feed2decc\") "
Sep 09 13:43:34.378734 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:34.378712    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0914355c-07e5-4492-9d15-ef1feed2decc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0914355c-07e5-4492-9d15-ef1feed2decc" (UID: "0914355c-07e5-4492-9d15-ef1feed2decc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:43:34.480192 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:34.480154    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0914355c-07e5-4492-9d15-ef1feed2decc-maas-api-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:35.124370 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:35.124342    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f59a0-cp45c"
Sep 09 13:43:35.124878 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:35.124389    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f59a0-cp45c" event={"ID":"0914355c-07e5-4492-9d15-ef1feed2decc","Type":"ContainerDied","Data":"8d33ebc5e2ba4af50e71200d7fa1ad6baf85cd52bc50c0fae6ae85a7d668ac78"}
Sep 09 13:43:35.124878 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:35.124446    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d33ebc5e2ba4af50e71200d7fa1ad6baf85cd52bc50c0fae6ae85a7d668ac78"
Sep 09 13:43:35.288484 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:35.288459    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c5f375-wdjbr"
Sep 09 13:43:35.398893 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:35.388868    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2224db8e-bdb5-46e1-87a4-bfc2420eafcb-maas-api-service-ca\") pod \"2224db8e-bdb5-46e1-87a4-bfc2420eafcb\" (UID: \"2224db8e-bdb5-46e1-87a4-bfc2420eafcb\") "
Sep 09 13:43:35.398893 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:35.389471    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2224db8e-bdb5-46e1-87a4-bfc2420eafcb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2224db8e-bdb5-46e1-87a4-bfc2420eafcb" (UID: "2224db8e-bdb5-46e1-87a4-bfc2420eafcb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:43:35.490413 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:35.490370    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2224db8e-bdb5-46e1-87a4-bfc2420eafcb-maas-api-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:36.131194 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:36.131165    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c5f375-wdjbr"
Sep 09 13:43:36.131598 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:36.131159    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c5f375-wdjbr" event={"ID":"2224db8e-bdb5-46e1-87a4-bfc2420eafcb","Type":"ContainerDied","Data":"ebcbbba5272f179cd91f26f54618fe480aff6786173c8ebcc6d1d211836f1f80"}
Sep 09 13:43:36.131598 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:36.131296    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebcbbba5272f179cd91f26f54618fe480aff6786173c8ebcc6d1d211836f1f80"
Sep 09 13:43:42.055724 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:42.055679    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f59a0-cp45c"]
Sep 09 13:43:42.058582 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:42.058555    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4f59a0-cp45c"]
Sep 09 13:43:43.034932 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:43.034901    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6"]
Sep 09 13:43:43.035135 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:43.035109    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6" podUID="a6cc7b00-86ad-4f3f-8e9f-9a62a736554b" containerName="payload-pre-processing" containerID="cri-o://28bb6355685d93f49dbdff5ef835818bbbe17c47a7a4d64c08bad6135fda72f5" gracePeriod=30
Sep 09 13:43:43.150282 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:43.150232    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0914355c-07e5-4492-9d15-ef1feed2decc" path="/var/lib/kubelet/pods/0914355c-07e5-4492-9d15-ef1feed2decc/volumes"
Sep 09 13:43:43.168060 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:43.168031    2566 generic.go:356] "Generic (PLEG): container finished" podID="2323ec19-7ff6-470a-b880-6cde1ce7104f" containerID="5671894e44006add60f7115d25f05941b92c1b38c93e4e7cee21e03e99e5ded3" exitCode=0
Sep 09 13:43:43.168218 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:43.168102    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww" event={"ID":"2323ec19-7ff6-470a-b880-6cde1ce7104f","Type":"ContainerDied","Data":"5671894e44006add60f7115d25f05941b92c1b38c93e4e7cee21e03e99e5ded3"}
Sep 09 13:43:43.168604 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:43.168579    2566 scope.go:117] "RemoveContainer" containerID="5671894e44006add60f7115d25f05941b92c1b38c93e4e7cee21e03e99e5ded3"
Sep 09 13:43:43.169773 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:43.169750    2566 generic.go:356] "Generic (PLEG): container finished" podID="a6cc7b00-86ad-4f3f-8e9f-9a62a736554b" containerID="28bb6355685d93f49dbdff5ef835818bbbe17c47a7a4d64c08bad6135fda72f5" exitCode=0
Sep 09 13:43:43.169889 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:43.169766    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:43:43.169889 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:43.169790    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6" event={"ID":"a6cc7b00-86ad-4f3f-8e9f-9a62a736554b","Type":"ContainerDied","Data":"28bb6355685d93f49dbdff5ef835818bbbe17c47a7a4d64c08bad6135fda72f5"}
Sep 09 13:43:43.169889 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:43.169855    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:43:43.169889 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:43.169888    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:43:43.303798 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:43.303770    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6"
Sep 09 13:43:43.327285 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:43.327254    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c5f375-wdjbr"]
Sep 09 13:43:43.329959 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:43.329931    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c5f375-wdjbr"]
Sep 09 13:43:43.383482 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:43.383444    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6cc7b00-86ad-4f3f-8e9f-9a62a736554b-plugins-config-volume\") pod \"a6cc7b00-86ad-4f3f-8e9f-9a62a736554b\" (UID: \"a6cc7b00-86ad-4f3f-8e9f-9a62a736554b\") "
Sep 09 13:43:43.383641 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:43.383499    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzxml\" (UniqueName: \"kubernetes.io/projected/a6cc7b00-86ad-4f3f-8e9f-9a62a736554b-kube-api-access-vzxml\") pod \"a6cc7b00-86ad-4f3f-8e9f-9a62a736554b\" (UID: \"a6cc7b00-86ad-4f3f-8e9f-9a62a736554b\") "
Sep 09 13:43:43.383908 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:43.383884    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6cc7b00-86ad-4f3f-8e9f-9a62a736554b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a6cc7b00-86ad-4f3f-8e9f-9a62a736554b" (UID: "a6cc7b00-86ad-4f3f-8e9f-9a62a736554b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:43:43.385650 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:43.385624    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6cc7b00-86ad-4f3f-8e9f-9a62a736554b-kube-api-access-vzxml" (OuterVolumeSpecName: "kube-api-access-vzxml") pod "a6cc7b00-86ad-4f3f-8e9f-9a62a736554b" (UID: "a6cc7b00-86ad-4f3f-8e9f-9a62a736554b"). InnerVolumeSpecName "kube-api-access-vzxml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:43:43.484205 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:43.484167    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6cc7b00-86ad-4f3f-8e9f-9a62a736554b-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:43.484205 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:43.484200    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzxml\" (UniqueName: \"kubernetes.io/projected/a6cc7b00-86ad-4f3f-8e9f-9a62a736554b-kube-api-access-vzxml\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:44.180003 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:44.179969    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6"
Sep 09 13:43:44.180003 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:44.180006    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6" event={"ID":"a6cc7b00-86ad-4f3f-8e9f-9a62a736554b","Type":"ContainerDied","Data":"0e054e08103726de990ae7889c965e3729bc7da4f78b7347dd8213b6d9c64c9c"}
Sep 09 13:43:44.180535 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:44.180078    2566 scope.go:117] "RemoveContainer" containerID="28bb6355685d93f49dbdff5ef835818bbbe17c47a7a4d64c08bad6135fda72f5"
Sep 09 13:43:44.182196 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:44.182173    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww" event={"ID":"2323ec19-7ff6-470a-b880-6cde1ce7104f","Type":"ContainerStarted","Data":"044acb6b024a92e33dd8b1d6a1bf3877d3b510275f716929b0eb331e84b4a37e"}
Sep 09 13:43:44.227958 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:44.227926    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6"]
Sep 09 13:43:44.232204 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:44.232176    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c5f375-5596d6598-8hhv6"]
Sep 09 13:43:44.607248 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:44.607206    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:44.607474 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:44.607378    2566 patch_prober.go:28] interesting pod/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 09 13:43:44.607474 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:44.607423    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww" podUID="2323ec19-7ff6-470a-b880-6cde1ce7104f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 09 13:43:45.149314 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:45.149281    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2224db8e-bdb5-46e1-87a4-bfc2420eafcb" path="/var/lib/kubelet/pods/2224db8e-bdb5-46e1-87a4-bfc2420eafcb/volumes"
Sep 09 13:43:45.149622 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:45.149610    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6cc7b00-86ad-4f3f-8e9f-9a62a736554b" path="/var/lib/kubelet/pods/a6cc7b00-86ad-4f3f-8e9f-9a62a736554b/volumes"
Sep 09 13:43:45.607372 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:45.607336    2566 patch_prober.go:28] interesting pod/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 09 13:43:45.607972 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:45.607940    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww" podUID="2323ec19-7ff6-470a-b880-6cde1ce7104f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 09 13:43:45.833071 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:45.832994    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc" podUID="64b9df97-cfd1-4916-8ebb-14f29c9452f5" containerName="main" containerID="cri-o://2b479fa744515dd41fe92fcfac6af1ef379ebad7aea8617d411c21b4ac328bb2" gracePeriod=45
Sep 09 13:43:46.115397 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.115365    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"
Sep 09 13:43:46.216327 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.216225    2566 generic.go:356] "Generic (PLEG): container finished" podID="64b9df97-cfd1-4916-8ebb-14f29c9452f5" containerID="2b479fa744515dd41fe92fcfac6af1ef379ebad7aea8617d411c21b4ac328bb2" exitCode=0
Sep 09 13:43:46.216327 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.216283    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"
Sep 09 13:43:46.216572 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.216287    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc" event={"ID":"64b9df97-cfd1-4916-8ebb-14f29c9452f5","Type":"ContainerDied","Data":"2b479fa744515dd41fe92fcfac6af1ef379ebad7aea8617d411c21b4ac328bb2"}
Sep 09 13:43:46.216572 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.216403    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc" event={"ID":"64b9df97-cfd1-4916-8ebb-14f29c9452f5","Type":"ContainerDied","Data":"50877bf38962c10885cfe51f1c3eee92d50dc8f9477ac156dc06fff47c790bb3"}
Sep 09 13:43:46.216572 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.216420    2566 scope.go:117] "RemoveContainer" containerID="2b479fa744515dd41fe92fcfac6af1ef379ebad7aea8617d411c21b4ac328bb2"
Sep 09 13:43:46.220690 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.220655    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvnfn\" (UniqueName: \"kubernetes.io/projected/64b9df97-cfd1-4916-8ebb-14f29c9452f5-kube-api-access-hvnfn\") pod \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\" (UID: \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\") "
Sep 09 13:43:46.220845 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.220695    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64b9df97-cfd1-4916-8ebb-14f29c9452f5-model-cache\") pod \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\" (UID: \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\") "
Sep 09 13:43:46.220845 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.220734    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64b9df97-cfd1-4916-8ebb-14f29c9452f5-dshm\") pod \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\" (UID: \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\") "
Sep 09 13:43:46.220845 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.220791    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64b9df97-cfd1-4916-8ebb-14f29c9452f5-home\") pod \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\" (UID: \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\") "
Sep 09 13:43:46.221029 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.220877    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64b9df97-cfd1-4916-8ebb-14f29c9452f5-tls-certs\") pod \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\" (UID: \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\") "
Sep 09 13:43:46.221029 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.220902    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64b9df97-cfd1-4916-8ebb-14f29c9452f5-tmp-dir\") pod \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\" (UID: \"64b9df97-cfd1-4916-8ebb-14f29c9452f5\") "
Sep 09 13:43:46.221029 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.220992    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b9df97-cfd1-4916-8ebb-14f29c9452f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "64b9df97-cfd1-4916-8ebb-14f29c9452f5" (UID: "64b9df97-cfd1-4916-8ebb-14f29c9452f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:43:46.221196 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.221177    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b9df97-cfd1-4916-8ebb-14f29c9452f5-home" (OuterVolumeSpecName: "home") pod "64b9df97-cfd1-4916-8ebb-14f29c9452f5" (UID: "64b9df97-cfd1-4916-8ebb-14f29c9452f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:43:46.221196 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.221182    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64b9df97-cfd1-4916-8ebb-14f29c9452f5-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:46.221532 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.221494    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b9df97-cfd1-4916-8ebb-14f29c9452f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "64b9df97-cfd1-4916-8ebb-14f29c9452f5" (UID: "64b9df97-cfd1-4916-8ebb-14f29c9452f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:43:46.223188 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.223153    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64b9df97-cfd1-4916-8ebb-14f29c9452f5-kube-api-access-hvnfn" (OuterVolumeSpecName: "kube-api-access-hvnfn") pod "64b9df97-cfd1-4916-8ebb-14f29c9452f5" (UID: "64b9df97-cfd1-4916-8ebb-14f29c9452f5"). InnerVolumeSpecName "kube-api-access-hvnfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:43:46.223276 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.223232    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64b9df97-cfd1-4916-8ebb-14f29c9452f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64b9df97-cfd1-4916-8ebb-14f29c9452f5" (UID: "64b9df97-cfd1-4916-8ebb-14f29c9452f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:43:46.223340 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.223316    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b9df97-cfd1-4916-8ebb-14f29c9452f5-dshm" (OuterVolumeSpecName: "dshm") pod "64b9df97-cfd1-4916-8ebb-14f29c9452f5" (UID: "64b9df97-cfd1-4916-8ebb-14f29c9452f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:43:46.226981 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.226961    2566 scope.go:117] "RemoveContainer" containerID="2b479fa744515dd41fe92fcfac6af1ef379ebad7aea8617d411c21b4ac328bb2"
Sep 09 13:43:46.227269 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:43:46.227243    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b479fa744515dd41fe92fcfac6af1ef379ebad7aea8617d411c21b4ac328bb2\": container with ID starting with 2b479fa744515dd41fe92fcfac6af1ef379ebad7aea8617d411c21b4ac328bb2 not found: ID does not exist" containerID="2b479fa744515dd41fe92fcfac6af1ef379ebad7aea8617d411c21b4ac328bb2"
Sep 09 13:43:46.227320 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.227283    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b479fa744515dd41fe92fcfac6af1ef379ebad7aea8617d411c21b4ac328bb2"} err="failed to get container status \"2b479fa744515dd41fe92fcfac6af1ef379ebad7aea8617d411c21b4ac328bb2\": rpc error: code = NotFound desc = could not find container \"2b479fa744515dd41fe92fcfac6af1ef379ebad7aea8617d411c21b4ac328bb2\": container with ID starting with 2b479fa744515dd41fe92fcfac6af1ef379ebad7aea8617d411c21b4ac328bb2 not found: ID does not exist"
Sep 09 13:43:46.322148 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.322118    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64b9df97-cfd1-4916-8ebb-14f29c9452f5-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:46.322148 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.322145    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64b9df97-cfd1-4916-8ebb-14f29c9452f5-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:46.322148 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.322154    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvnfn\" (UniqueName: \"kubernetes.io/projected/64b9df97-cfd1-4916-8ebb-14f29c9452f5-kube-api-access-hvnfn\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:46.322377 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.322163    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64b9df97-cfd1-4916-8ebb-14f29c9452f5-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:46.322377 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.322171    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64b9df97-cfd1-4916-8ebb-14f29c9452f5-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:46.537125 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.537096    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"]
Sep 09 13:43:46.543513 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.543489    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-093e38/test-model-093e38-kserve-776878b8d5-f89zc"]
Sep 09 13:43:46.607919 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.607880    2566 patch_prober.go:28] interesting pod/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 09 13:43:46.608427 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.607948    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww" podUID="2323ec19-7ff6-470a-b880-6cde1ce7104f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 09 13:43:46.987918 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:46.987868    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr" podUID="9305809d-30ff-4c5c-b588-7aa128af0c12" containerName="main" containerID="cri-o://0d3a06aa8d879743a3f96775eed6bba7e9811ae3fdd987e07f234c9d752a6173" gracePeriod=45
Sep 09 13:43:47.150607 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:47.150484    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64b9df97-cfd1-4916-8ebb-14f29c9452f5" path="/var/lib/kubelet/pods/64b9df97-cfd1-4916-8ebb-14f29c9452f5/volumes"
Sep 09 13:43:47.223863 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:47.223826    2566 generic.go:356] "Generic (PLEG): container finished" podID="9305809d-30ff-4c5c-b588-7aa128af0c12" containerID="0d3a06aa8d879743a3f96775eed6bba7e9811ae3fdd987e07f234c9d752a6173" exitCode=0
Sep 09 13:43:47.224036 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:47.223908    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr" event={"ID":"9305809d-30ff-4c5c-b588-7aa128af0c12","Type":"ContainerDied","Data":"0d3a06aa8d879743a3f96775eed6bba7e9811ae3fdd987e07f234c9d752a6173"}
Sep 09 13:43:47.270100 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:47.270078    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"
Sep 09 13:43:47.332259 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:47.332225    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9305809d-30ff-4c5c-b588-7aa128af0c12-dshm\") pod \"9305809d-30ff-4c5c-b588-7aa128af0c12\" (UID: \"9305809d-30ff-4c5c-b588-7aa128af0c12\") "
Sep 09 13:43:47.332447 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:47.332271    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjdb6\" (UniqueName: \"kubernetes.io/projected/9305809d-30ff-4c5c-b588-7aa128af0c12-kube-api-access-jjdb6\") pod \"9305809d-30ff-4c5c-b588-7aa128af0c12\" (UID: \"9305809d-30ff-4c5c-b588-7aa128af0c12\") "
Sep 09 13:43:47.332447 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:47.332308    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9305809d-30ff-4c5c-b588-7aa128af0c12-tmp-dir\") pod \"9305809d-30ff-4c5c-b588-7aa128af0c12\" (UID: \"9305809d-30ff-4c5c-b588-7aa128af0c12\") "
Sep 09 13:43:47.332447 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:47.332342    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9305809d-30ff-4c5c-b588-7aa128af0c12-home\") pod \"9305809d-30ff-4c5c-b588-7aa128af0c12\" (UID: \"9305809d-30ff-4c5c-b588-7aa128af0c12\") "
Sep 09 13:43:47.332447 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:47.332361    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9305809d-30ff-4c5c-b588-7aa128af0c12-model-cache\") pod \"9305809d-30ff-4c5c-b588-7aa128af0c12\" (UID: \"9305809d-30ff-4c5c-b588-7aa128af0c12\") "
Sep 09 13:43:47.332677 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:47.332466    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9305809d-30ff-4c5c-b588-7aa128af0c12-tls-certs\") pod \"9305809d-30ff-4c5c-b588-7aa128af0c12\" (UID: \"9305809d-30ff-4c5c-b588-7aa128af0c12\") "
Sep 09 13:43:47.332677 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:47.332560    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9305809d-30ff-4c5c-b588-7aa128af0c12-home" (OuterVolumeSpecName: "home") pod "9305809d-30ff-4c5c-b588-7aa128af0c12" (UID: "9305809d-30ff-4c5c-b588-7aa128af0c12"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:43:47.332677 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:47.332596    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9305809d-30ff-4c5c-b588-7aa128af0c12-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9305809d-30ff-4c5c-b588-7aa128af0c12" (UID: "9305809d-30ff-4c5c-b588-7aa128af0c12"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:43:47.332837 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:47.332757    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9305809d-30ff-4c5c-b588-7aa128af0c12-model-cache" (OuterVolumeSpecName: "model-cache") pod "9305809d-30ff-4c5c-b588-7aa128af0c12" (UID: "9305809d-30ff-4c5c-b588-7aa128af0c12"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:43:47.332924 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:47.332905    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9305809d-30ff-4c5c-b588-7aa128af0c12-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:47.332964 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:47.332929    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9305809d-30ff-4c5c-b588-7aa128af0c12-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:47.332964 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:47.332946    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9305809d-30ff-4c5c-b588-7aa128af0c12-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:47.334790 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:47.334752    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9305809d-30ff-4c5c-b588-7aa128af0c12-kube-api-access-jjdb6" (OuterVolumeSpecName: "kube-api-access-jjdb6") pod "9305809d-30ff-4c5c-b588-7aa128af0c12" (UID: "9305809d-30ff-4c5c-b588-7aa128af0c12"). InnerVolumeSpecName "kube-api-access-jjdb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:43:47.334948 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:47.334868    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9305809d-30ff-4c5c-b588-7aa128af0c12-dshm" (OuterVolumeSpecName: "dshm") pod "9305809d-30ff-4c5c-b588-7aa128af0c12" (UID: "9305809d-30ff-4c5c-b588-7aa128af0c12"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:43:47.334948 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:47.334902    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9305809d-30ff-4c5c-b588-7aa128af0c12-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9305809d-30ff-4c5c-b588-7aa128af0c12" (UID: "9305809d-30ff-4c5c-b588-7aa128af0c12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:43:47.433582 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:47.433550    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9305809d-30ff-4c5c-b588-7aa128af0c12-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:47.433582 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:47.433575    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjdb6\" (UniqueName: \"kubernetes.io/projected/9305809d-30ff-4c5c-b588-7aa128af0c12-kube-api-access-jjdb6\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:47.433582 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:47.433585    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9305809d-30ff-4c5c-b588-7aa128af0c12-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:47.607062 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:47.607017    2566 patch_prober.go:28] interesting pod/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 09 13:43:47.607288 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:47.607088    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww" podUID="2323ec19-7ff6-470a-b880-6cde1ce7104f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 09 13:43:48.231181 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:48.231146    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"
Sep 09 13:43:48.231664 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:48.231187    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr" event={"ID":"9305809d-30ff-4c5c-b588-7aa128af0c12","Type":"ContainerDied","Data":"df44290a3d2d14551c75bb0b3f8eebaf852a25bbdb3e61d2594481d65ab04429"}
Sep 09 13:43:48.231729 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:48.231693    2566 scope.go:117] "RemoveContainer" containerID="0d3a06aa8d879743a3f96775eed6bba7e9811ae3fdd987e07f234c9d752a6173"
Sep 09 13:43:48.254727 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:48.254699    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"]
Sep 09 13:43:48.261806 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:48.261779    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-c5f375/explicit-model-c5f375-kserve-68d85d959-ht6rr"]
Sep 09 13:43:48.608076 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:48.607569    2566 patch_prober.go:28] interesting pod/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Sep 09 13:43:48.608076 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:48.607638    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww" podUID="2323ec19-7ff6-470a-b880-6cde1ce7104f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Sep 09 13:43:49.149991 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.149956    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9305809d-30ff-4c5c-b588-7aa128af0c12" path="/var/lib/kubelet/pods/9305809d-30ff-4c5c-b588-7aa128af0c12/volumes"
Sep 09 13:43:49.223477 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.223419    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"]
Sep 09 13:43:49.223736 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.223708    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh" podUID="c8d6623e-e731-4ace-8f3b-f44db09987c8" containerName="istio-proxy" containerID="cri-o://5ae16fd07f956588921fdf2088943b7dff60e7c04f5d564ecda0c2a81e5b2376" gracePeriod=30
Sep 09 13:43:49.611646 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.611613    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:49.612120 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.612005    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:49.612793 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.612775    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:43:49.633844 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.633807    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093e38-zzc6p"]
Sep 09 13:43:49.634147 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.634136    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9305809d-30ff-4c5c-b588-7aa128af0c12" containerName="main"
Sep 09 13:43:49.634186 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.634149    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9305809d-30ff-4c5c-b588-7aa128af0c12" containerName="main"
Sep 09 13:43:49.634186 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.634169    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0914355c-07e5-4492-9d15-ef1feed2decc" containerName="revoke-keys"
Sep 09 13:43:49.634186 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.634174    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0914355c-07e5-4492-9d15-ef1feed2decc" containerName="revoke-keys"
Sep 09 13:43:49.634186 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.634181    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2224db8e-bdb5-46e1-87a4-bfc2420eafcb" containerName="revoke-keys"
Sep 09 13:43:49.634186 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.634186    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2224db8e-bdb5-46e1-87a4-bfc2420eafcb" containerName="revoke-keys"
Sep 09 13:43:49.634352 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.634199    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6cc7b00-86ad-4f3f-8e9f-9a62a736554b" containerName="payload-pre-processing"
Sep 09 13:43:49.634352 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.634205    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6cc7b00-86ad-4f3f-8e9f-9a62a736554b" containerName="payload-pre-processing"
Sep 09 13:43:49.634352 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.634221    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64b9df97-cfd1-4916-8ebb-14f29c9452f5" containerName="main"
Sep 09 13:43:49.634352 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.634226    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="64b9df97-cfd1-4916-8ebb-14f29c9452f5" containerName="main"
Sep 09 13:43:49.634352 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.634281    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6cc7b00-86ad-4f3f-8e9f-9a62a736554b" containerName="payload-pre-processing"
Sep 09 13:43:49.634352 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.634289    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="64b9df97-cfd1-4916-8ebb-14f29c9452f5" containerName="main"
Sep 09 13:43:49.634352 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.634296    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9305809d-30ff-4c5c-b588-7aa128af0c12" containerName="main"
Sep 09 13:43:49.634352 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.634303    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0914355c-07e5-4492-9d15-ef1feed2decc" containerName="revoke-keys"
Sep 09 13:43:49.634352 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.634309    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2224db8e-bdb5-46e1-87a4-bfc2420eafcb" containerName="revoke-keys"
Sep 09 13:43:49.638405 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.638390    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093e38-zzc6p"
Sep 09 13:43:49.645073 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.644911    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lh487\""
Sep 09 13:43:49.645156 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.645135    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 13:43:49.656671 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.656649    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093e38-zzc6p"]
Sep 09 13:43:49.761286 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.761252    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/731aafcb-540e-441a-8109-afe96b210cb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-093e38-zzc6p\" (UID: \"731aafcb-540e-441a-8109-afe96b210cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093e38-zzc6p"
Sep 09 13:43:49.862379 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.862287    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/731aafcb-540e-441a-8109-afe96b210cb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-093e38-zzc6p\" (UID: \"731aafcb-540e-441a-8109-afe96b210cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093e38-zzc6p"
Sep 09 13:43:49.862938 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.862919    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/731aafcb-540e-441a-8109-afe96b210cb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-093e38-zzc6p\" (UID: \"731aafcb-540e-441a-8109-afe96b210cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093e38-zzc6p"
Sep 09 13:43:49.949020 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:49.948988    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093e38-zzc6p"
Sep 09 13:43:50.230878 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:50.230851    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093e38-zzc6p"]
Sep 09 13:43:50.232430 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:43:50.232400    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731aafcb_540e_441a_8109_afe96b210cb2.slice/crio-4b181dafd81765a5d25961ef1eb9415e25a198cf8f4a6859b3ff2ee0df9788db WatchSource:0}: Error finding container 4b181dafd81765a5d25961ef1eb9415e25a198cf8f4a6859b3ff2ee0df9788db: Status 404 returned error can't find the container with id 4b181dafd81765a5d25961ef1eb9415e25a198cf8f4a6859b3ff2ee0df9788db
Sep 09 13:43:50.242953 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:50.242922    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093e38-zzc6p" event={"ID":"731aafcb-540e-441a-8109-afe96b210cb2","Type":"ContainerStarted","Data":"4b181dafd81765a5d25961ef1eb9415e25a198cf8f4a6859b3ff2ee0df9788db"}
Sep 09 13:43:51.250722 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.250688    2566 generic.go:356] "Generic (PLEG): container finished" podID="731aafcb-540e-441a-8109-afe96b210cb2" containerID="756469308d0fc4b64bfca2896196b97b690c063ec536516e2d885c1bc7fac8c1" exitCode=0
Sep 09 13:43:51.251119 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.250777    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093e38-zzc6p" event={"ID":"731aafcb-540e-441a-8109-afe96b210cb2","Type":"ContainerDied","Data":"756469308d0fc4b64bfca2896196b97b690c063ec536516e2d885c1bc7fac8c1"}
Sep 09 13:43:51.599378 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.599346    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"]
Sep 09 13:43:51.602687 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.602668    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:51.604851 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.604830    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-c9dd9a-openshift-default-dockercfg-29k8l\""
Sep 09 13:43:51.616166 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.616146    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"]
Sep 09 13:43:51.681442 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.681407    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-credential-socket\") pod \"e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") " pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:51.681442 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.681442    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-istio-envoy\") pod \"e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") " pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:51.681637 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.681505    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-istio-data\") pod \"e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") " pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:51.681637 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.681530    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rqzk\" (UniqueName: \"kubernetes.io/projected/fc2199c0-b068-46e6-8fa8-7510766412a0-kube-api-access-4rqzk\") pod \"e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") " pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:51.681637 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.681569    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc2199c0-b068-46e6-8fa8-7510766412a0-istio-token\") pod \"e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") " pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:51.681637 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.681602    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc2199c0-b068-46e6-8fa8-7510766412a0-istiod-ca-cert\") pod \"e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") " pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:51.681637 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.681627    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-workload-certs\") pod \"e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") " pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:51.681795 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.681644    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc2199c0-b068-46e6-8fa8-7510766412a0-istio-podinfo\") pod \"e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") " pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:51.681795 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.681702    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-workload-socket\") pod \"e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") " pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:51.782291 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.782230    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-credential-socket\") pod \"e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") " pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:51.782448 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.782324    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-istio-envoy\") pod \"e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") " pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:51.782448 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.782389    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-istio-data\") pod \"e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") " pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:51.782448 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.782426    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rqzk\" (UniqueName: \"kubernetes.io/projected/fc2199c0-b068-46e6-8fa8-7510766412a0-kube-api-access-4rqzk\") pod \"e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") " pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:51.782600 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.782467    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc2199c0-b068-46e6-8fa8-7510766412a0-istio-token\") pod \"e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") " pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:51.782600 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.782524    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc2199c0-b068-46e6-8fa8-7510766412a0-istiod-ca-cert\") pod \"e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") " pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:51.782600 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.782562    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-workload-certs\") pod \"e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") " pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:51.782600 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.782588    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc2199c0-b068-46e6-8fa8-7510766412a0-istio-podinfo\") pod \"e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") " pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:51.782830 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.782646    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-workload-socket\") pod \"e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") " pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:51.782830 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.782708    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-credential-socket\") pod \"e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") " pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:51.782953 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.782888    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-istio-data\") pod \"e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") " pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:51.783095 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.783071    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-workload-certs\") pod \"e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") " pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:51.783184 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.783110    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-workload-socket\") pod \"e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") " pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:51.783341 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.783324    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc2199c0-b068-46e6-8fa8-7510766412a0-istiod-ca-cert\") pod \"e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") " pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:51.784992 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.784972    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-istio-envoy\") pod \"e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") " pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:51.785103 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.785087    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc2199c0-b068-46e6-8fa8-7510766412a0-istio-podinfo\") pod \"e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") " pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:51.789883 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.789864    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rqzk\" (UniqueName: \"kubernetes.io/projected/fc2199c0-b068-46e6-8fa8-7510766412a0-kube-api-access-4rqzk\") pod \"e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") " pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:51.790179 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.790157    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc2199c0-b068-46e6-8fa8-7510766412a0-istio-token\") pod \"e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") " pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:51.913474 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:51.913380    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:52.189425 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:52.189401    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"]
Sep 09 13:43:52.191907 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:43:52.191875    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2199c0_b068_46e6_8fa8_7510766412a0.slice/crio-66f77387b4f18dcd355ebe4970e590b3920a35d30c927bd1608d3ac57dd1d23f WatchSource:0}: Error finding container 66f77387b4f18dcd355ebe4970e590b3920a35d30c927bd1608d3ac57dd1d23f: Status 404 returned error can't find the container with id 66f77387b4f18dcd355ebe4970e590b3920a35d30c927bd1608d3ac57dd1d23f
Sep 09 13:43:52.194402 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:52.194366    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:43:52.194487 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:52.194431    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:43:52.194487 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:52.194459    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:43:52.257975 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:52.257946    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549" event={"ID":"fc2199c0-b068-46e6-8fa8-7510766412a0","Type":"ContainerStarted","Data":"e3f42a8b956534e45ecabb3649250f04795ab8180c3771536668953c08b3e689"}
Sep 09 13:43:52.258316 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:52.258007    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549" event={"ID":"fc2199c0-b068-46e6-8fa8-7510766412a0","Type":"ContainerStarted","Data":"66f77387b4f18dcd355ebe4970e590b3920a35d30c927bd1608d3ac57dd1d23f"}
Sep 09 13:43:52.287752 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:52.286751    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549" podStartSLOduration=1.286724341 podStartE2EDuration="1.286724341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:43:52.286325955 +0000 UTC m=+1065.646175781" watchObservedRunningTime="2026-09-09 13:43:52.286724341 +0000 UTC m=+1065.646574167"
Sep 09 13:43:52.376111 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:52.376080    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093e38-zzc6p"
Sep 09 13:43:52.490278 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:52.490163    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/731aafcb-540e-441a-8109-afe96b210cb2-maas-api-service-ca\") pod \"731aafcb-540e-441a-8109-afe96b210cb2\" (UID: \"731aafcb-540e-441a-8109-afe96b210cb2\") "
Sep 09 13:43:52.490547 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:52.490523    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/731aafcb-540e-441a-8109-afe96b210cb2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "731aafcb-540e-441a-8109-afe96b210cb2" (UID: "731aafcb-540e-441a-8109-afe96b210cb2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:43:52.591411 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:52.591376    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/731aafcb-540e-441a-8109-afe96b210cb2-maas-api-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:52.913763 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:52.913725    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:53.264058 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:53.264021    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093e38-zzc6p" event={"ID":"731aafcb-540e-441a-8109-afe96b210cb2","Type":"ContainerDied","Data":"4b181dafd81765a5d25961ef1eb9415e25a198cf8f4a6859b3ff2ee0df9788db"}
Sep 09 13:43:53.264422 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:53.264108    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b181dafd81765a5d25961ef1eb9415e25a198cf8f4a6859b3ff2ee0df9788db"
Sep 09 13:43:53.264422 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:53.264032    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093e38-zzc6p"
Sep 09 13:43:53.919444 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:53.919412    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:54.269727 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.269702    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:54.270717 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.270696    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:43:54.419211 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.419181    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-905b058d-mzcnp"]
Sep 09 13:43:54.419570 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.419557    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="731aafcb-540e-441a-8109-afe96b210cb2" containerName="revoke-keys"
Sep 09 13:43:54.419623 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.419572    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="731aafcb-540e-441a-8109-afe96b210cb2" containerName="revoke-keys"
Sep 09 13:43:54.419655 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.419644    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="731aafcb-540e-441a-8109-afe96b210cb2" containerName="revoke-keys"
Sep 09 13:43:54.422585 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.422569    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-905b058d-mzcnp"
Sep 09 13:43:54.424990 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.424961    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lh487\""
Sep 09 13:43:54.425113 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.425008    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 13:43:54.431551 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.431528    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-905b058d-mzcnp"]
Sep 09 13:43:54.505910 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.505888    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:43:54.511400 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.511373    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85c4918-fb24-469c-8341-af0fce2d56c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-905b058d-mzcnp\" (UID: \"a85c4918-fb24-469c-8341-af0fce2d56c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-905b058d-mzcnp"
Sep 09 13:43:54.612745 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.612707    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-credential-socket\") pod \"c8d6623e-e731-4ace-8f3b-f44db09987c8\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") "
Sep 09 13:43:54.612980 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.612796    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-istio-envoy\") pod \"c8d6623e-e731-4ace-8f3b-f44db09987c8\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") "
Sep 09 13:43:54.612980 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.612853    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8d6623e-e731-4ace-8f3b-f44db09987c8-istio-token\") pod \"c8d6623e-e731-4ace-8f3b-f44db09987c8\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") "
Sep 09 13:43:54.612980 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.612887    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stk2w\" (UniqueName: \"kubernetes.io/projected/c8d6623e-e731-4ace-8f3b-f44db09987c8-kube-api-access-stk2w\") pod \"c8d6623e-e731-4ace-8f3b-f44db09987c8\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") "
Sep 09 13:43:54.612980 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.612919    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-workload-certs\") pod \"c8d6623e-e731-4ace-8f3b-f44db09987c8\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") "
Sep 09 13:43:54.613211 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.612996    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-workload-socket\") pod \"c8d6623e-e731-4ace-8f3b-f44db09987c8\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") "
Sep 09 13:43:54.613211 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.613047    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c8d6623e-e731-4ace-8f3b-f44db09987c8" (UID: "c8d6623e-e731-4ace-8f3b-f44db09987c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:43:54.613211 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.613056    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8d6623e-e731-4ace-8f3b-f44db09987c8-istio-podinfo\") pod \"c8d6623e-e731-4ace-8f3b-f44db09987c8\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") "
Sep 09 13:43:54.613211 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.613174    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8d6623e-e731-4ace-8f3b-f44db09987c8-istiod-ca-cert\") pod \"c8d6623e-e731-4ace-8f3b-f44db09987c8\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") "
Sep 09 13:43:54.613211 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.613178    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c8d6623e-e731-4ace-8f3b-f44db09987c8" (UID: "c8d6623e-e731-4ace-8f3b-f44db09987c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:43:54.613452 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.613240    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-istio-data\") pod \"c8d6623e-e731-4ace-8f3b-f44db09987c8\" (UID: \"c8d6623e-e731-4ace-8f3b-f44db09987c8\") "
Sep 09 13:43:54.613452 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.613300    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c8d6623e-e731-4ace-8f3b-f44db09987c8" (UID: "c8d6623e-e731-4ace-8f3b-f44db09987c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:43:54.613545 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.613469    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85c4918-fb24-469c-8341-af0fce2d56c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-905b058d-mzcnp\" (UID: \"a85c4918-fb24-469c-8341-af0fce2d56c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-905b058d-mzcnp"
Sep 09 13:43:54.613545 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.613479    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8d6623e-e731-4ace-8f3b-f44db09987c8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c8d6623e-e731-4ace-8f3b-f44db09987c8" (UID: "c8d6623e-e731-4ace-8f3b-f44db09987c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:43:54.613545 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.613514    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-istio-data" (OuterVolumeSpecName: "istio-data") pod "c8d6623e-e731-4ace-8f3b-f44db09987c8" (UID: "c8d6623e-e731-4ace-8f3b-f44db09987c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:43:54.613699 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.613651    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8d6623e-e731-4ace-8f3b-f44db09987c8-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:54.613699 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.613667    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:54.613699 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.613680    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:54.613699 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.613695    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:54.613922 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.613709    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:54.614085 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.614066    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85c4918-fb24-469c-8341-af0fce2d56c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-905b058d-mzcnp\" (UID: \"a85c4918-fb24-469c-8341-af0fce2d56c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-905b058d-mzcnp"
Sep 09 13:43:54.615269 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.615239    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c8d6623e-e731-4ace-8f3b-f44db09987c8" (UID: "c8d6623e-e731-4ace-8f3b-f44db09987c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:43:54.615269 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.615261    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8d6623e-e731-4ace-8f3b-f44db09987c8-istio-token" (OuterVolumeSpecName: "istio-token") pod "c8d6623e-e731-4ace-8f3b-f44db09987c8" (UID: "c8d6623e-e731-4ace-8f3b-f44db09987c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:43:54.615514 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.615488    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8d6623e-e731-4ace-8f3b-f44db09987c8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c8d6623e-e731-4ace-8f3b-f44db09987c8" (UID: "c8d6623e-e731-4ace-8f3b-f44db09987c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 13:43:54.615575 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.615524    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8d6623e-e731-4ace-8f3b-f44db09987c8-kube-api-access-stk2w" (OuterVolumeSpecName: "kube-api-access-stk2w") pod "c8d6623e-e731-4ace-8f3b-f44db09987c8" (UID: "c8d6623e-e731-4ace-8f3b-f44db09987c8"). InnerVolumeSpecName "kube-api-access-stk2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:43:54.714932 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.714900    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8d6623e-e731-4ace-8f3b-f44db09987c8-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:54.714932 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.714929    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8d6623e-e731-4ace-8f3b-f44db09987c8-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:54.715144 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.714941    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stk2w\" (UniqueName: \"kubernetes.io/projected/c8d6623e-e731-4ace-8f3b-f44db09987c8-kube-api-access-stk2w\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:54.715144 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.714957    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8d6623e-e731-4ace-8f3b-f44db09987c8-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:54.736733 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:54.736703    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-905b058d-mzcnp"
Sep 09 13:43:55.015992 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:55.015911    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-905b058d-mzcnp"]
Sep 09 13:43:55.020501 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:43:55.020476    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85c4918_fb24_469c_8341_af0fce2d56c4.slice/crio-88de58830a7f47e9b0293c25259ec7ff19ab374c9c4b1cf442a1dcef9e322613 WatchSource:0}: Error finding container 88de58830a7f47e9b0293c25259ec7ff19ab374c9c4b1cf442a1dcef9e322613: Status 404 returned error can't find the container with id 88de58830a7f47e9b0293c25259ec7ff19ab374c9c4b1cf442a1dcef9e322613
Sep 09 13:43:55.275806 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:55.275728    2566 generic.go:356] "Generic (PLEG): container finished" podID="c8d6623e-e731-4ace-8f3b-f44db09987c8" containerID="5ae16fd07f956588921fdf2088943b7dff60e7c04f5d564ecda0c2a81e5b2376" exitCode=0
Sep 09 13:43:55.275806 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:55.275775    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"
Sep 09 13:43:55.276351 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:55.275832    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh" event={"ID":"c8d6623e-e731-4ace-8f3b-f44db09987c8","Type":"ContainerDied","Data":"5ae16fd07f956588921fdf2088943b7dff60e7c04f5d564ecda0c2a81e5b2376"}
Sep 09 13:43:55.276351 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:55.275893    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh" event={"ID":"c8d6623e-e731-4ace-8f3b-f44db09987c8","Type":"ContainerDied","Data":"139f26adcccd0a4e4ec25466a40b315ce0e40f768a3eb3d64844d34f80210ef7"}
Sep 09 13:43:55.276351 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:55.275925    2566 scope.go:117] "RemoveContainer" containerID="5ae16fd07f956588921fdf2088943b7dff60e7c04f5d564ecda0c2a81e5b2376"
Sep 09 13:43:55.277300 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:55.277272    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-905b058d-mzcnp" event={"ID":"a85c4918-fb24-469c-8341-af0fce2d56c4","Type":"ContainerStarted","Data":"88de58830a7f47e9b0293c25259ec7ff19ab374c9c4b1cf442a1dcef9e322613"}
Sep 09 13:43:55.285056 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:55.285034    2566 scope.go:117] "RemoveContainer" containerID="49d2710ad58674557cfd0a8c1cfe7569b379b0b15112bad6ea0939d197f5326c"
Sep 09 13:43:55.293579 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:55.293557    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"]
Sep 09 13:43:55.293764 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:55.293748    2566 scope.go:117] "RemoveContainer" containerID="5ae16fd07f956588921fdf2088943b7dff60e7c04f5d564ecda0c2a81e5b2376"
Sep 09 13:43:55.294090 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:43:55.294073    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ae16fd07f956588921fdf2088943b7dff60e7c04f5d564ecda0c2a81e5b2376\": container with ID starting with 5ae16fd07f956588921fdf2088943b7dff60e7c04f5d564ecda0c2a81e5b2376 not found: ID does not exist" containerID="5ae16fd07f956588921fdf2088943b7dff60e7c04f5d564ecda0c2a81e5b2376"
Sep 09 13:43:55.294152 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:55.294099    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ae16fd07f956588921fdf2088943b7dff60e7c04f5d564ecda0c2a81e5b2376"} err="failed to get container status \"5ae16fd07f956588921fdf2088943b7dff60e7c04f5d564ecda0c2a81e5b2376\": rpc error: code = NotFound desc = could not find container \"5ae16fd07f956588921fdf2088943b7dff60e7c04f5d564ecda0c2a81e5b2376\": container with ID starting with 5ae16fd07f956588921fdf2088943b7dff60e7c04f5d564ecda0c2a81e5b2376 not found: ID does not exist"
Sep 09 13:43:55.294152 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:55.294116    2566 scope.go:117] "RemoveContainer" containerID="49d2710ad58674557cfd0a8c1cfe7569b379b0b15112bad6ea0939d197f5326c"
Sep 09 13:43:55.294426 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:43:55.294411    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49d2710ad58674557cfd0a8c1cfe7569b379b0b15112bad6ea0939d197f5326c\": container with ID starting with 49d2710ad58674557cfd0a8c1cfe7569b379b0b15112bad6ea0939d197f5326c not found: ID does not exist" containerID="49d2710ad58674557cfd0a8c1cfe7569b379b0b15112bad6ea0939d197f5326c"
Sep 09 13:43:55.294472 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:55.294430    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49d2710ad58674557cfd0a8c1cfe7569b379b0b15112bad6ea0939d197f5326c"} err="failed to get container status \"49d2710ad58674557cfd0a8c1cfe7569b379b0b15112bad6ea0939d197f5326c\": rpc error: code = NotFound desc = could not find container \"49d2710ad58674557cfd0a8c1cfe7569b379b0b15112bad6ea0939d197f5326c\": container with ID starting with 49d2710ad58674557cfd0a8c1cfe7569b379b0b15112bad6ea0939d197f5326c not found: ID does not exist"
Sep 09 13:43:55.297866 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:55.297845    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4f59a0-openshift-default-5757654bd8-2rcnh"]
Sep 09 13:43:56.285711 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:56.285673    2566 generic.go:356] "Generic (PLEG): container finished" podID="a85c4918-fb24-469c-8341-af0fce2d56c4" containerID="9bd4f85a5fed27b0682992fd8459354f39929add86c323800a43743f4f0c46eb" exitCode=0
Sep 09 13:43:56.286150 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:56.285741    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-905b058d-mzcnp" event={"ID":"a85c4918-fb24-469c-8341-af0fce2d56c4","Type":"ContainerDied","Data":"9bd4f85a5fed27b0682992fd8459354f39929add86c323800a43743f4f0c46eb"}
Sep 09 13:43:57.151323 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:57.151294    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8d6623e-e731-4ace-8f3b-f44db09987c8" path="/var/lib/kubelet/pods/c8d6623e-e731-4ace-8f3b-f44db09987c8/volumes"
Sep 09 13:43:57.436116 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:57.436088    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-905b058d-mzcnp"
Sep 09 13:43:57.540344 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:57.540314    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85c4918-fb24-469c-8341-af0fce2d56c4-maas-api-service-ca\") pod \"a85c4918-fb24-469c-8341-af0fce2d56c4\" (UID: \"a85c4918-fb24-469c-8341-af0fce2d56c4\") "
Sep 09 13:43:57.540662 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:57.540640    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a85c4918-fb24-469c-8341-af0fce2d56c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a85c4918-fb24-469c-8341-af0fce2d56c4" (UID: "a85c4918-fb24-469c-8341-af0fce2d56c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:43:57.641844 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:57.641801    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85c4918-fb24-469c-8341-af0fce2d56c4-maas-api-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:43:58.297189 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:58.297159    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-905b058d-mzcnp"
Sep 09 13:43:58.297391 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:58.297195    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-905b058d-mzcnp" event={"ID":"a85c4918-fb24-469c-8341-af0fce2d56c4","Type":"ContainerDied","Data":"88de58830a7f47e9b0293c25259ec7ff19ab374c9c4b1cf442a1dcef9e322613"}
Sep 09 13:43:58.297391 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:43:58.297238    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88de58830a7f47e9b0293c25259ec7ff19ab374c9c4b1cf442a1dcef9e322613"
Sep 09 13:44:00.566239 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:00.566155    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093e38-zzc6p"]
Sep 09 13:44:00.571091 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:00.571065    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093e38-zzc6p"]
Sep 09 13:44:01.150508 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:01.150472    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="731aafcb-540e-441a-8109-afe96b210cb2" path="/var/lib/kubelet/pods/731aafcb-540e-441a-8109-afe96b210cb2/volumes"
Sep 09 13:44:05.435174 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:05.435141    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-905b058d-mzcnp"]
Sep 09 13:44:05.437595 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:05.437570    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-905b058d-mzcnp"]
Sep 09 13:44:07.150124 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:07.150092    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a85c4918-fb24-469c-8341-af0fce2d56c4" path="/var/lib/kubelet/pods/a85c4918-fb24-469c-8341-af0fce2d56c4/volumes"
Sep 09 13:44:07.519120 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:07.519023    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"]
Sep 09 13:44:07.519312 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:07.519285    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl" podUID="9ff22c80-e444-46f0-97d8-0cb8247f2231" containerName="istio-proxy" containerID="cri-o://e736a806891a50d3e4d34db269d7a178d696a6161d53fcd22ef7aecd19b94c36" gracePeriod=30
Sep 09 13:44:12.497748 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.497704    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"]
Sep 09 13:44:12.498219 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.497972    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww" podUID="2323ec19-7ff6-470a-b880-6cde1ce7104f" containerName="istio-proxy" containerID="cri-o://044acb6b024a92e33dd8b1d6a1bf3877d3b510275f716929b0eb331e84b4a37e" gracePeriod=30
Sep 09 13:44:12.774377 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.774351    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:44:12.873992 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.873944    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbsjd\" (UniqueName: \"kubernetes.io/projected/9ff22c80-e444-46f0-97d8-0cb8247f2231-kube-api-access-kbsjd\") pod \"9ff22c80-e444-46f0-97d8-0cb8247f2231\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") "
Sep 09 13:44:12.873992 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.874020    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-workload-socket\") pod \"9ff22c80-e444-46f0-97d8-0cb8247f2231\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") "
Sep 09 13:44:12.874292 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.874048    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-istio-envoy\") pod \"9ff22c80-e444-46f0-97d8-0cb8247f2231\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") "
Sep 09 13:44:12.874292 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.874092    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ff22c80-e444-46f0-97d8-0cb8247f2231-istio-token\") pod \"9ff22c80-e444-46f0-97d8-0cb8247f2231\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") "
Sep 09 13:44:12.874292 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.874123    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ff22c80-e444-46f0-97d8-0cb8247f2231-istiod-ca-cert\") pod \"9ff22c80-e444-46f0-97d8-0cb8247f2231\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") "
Sep 09 13:44:12.874292 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.874150    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-credential-socket\") pod \"9ff22c80-e444-46f0-97d8-0cb8247f2231\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") "
Sep 09 13:44:12.874292 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.874179    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ff22c80-e444-46f0-97d8-0cb8247f2231-istio-podinfo\") pod \"9ff22c80-e444-46f0-97d8-0cb8247f2231\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") "
Sep 09 13:44:12.874292 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.874204    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-istio-data\") pod \"9ff22c80-e444-46f0-97d8-0cb8247f2231\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") "
Sep 09 13:44:12.874292 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.874236    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-workload-certs\") pod \"9ff22c80-e444-46f0-97d8-0cb8247f2231\" (UID: \"9ff22c80-e444-46f0-97d8-0cb8247f2231\") "
Sep 09 13:44:12.874646 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.874320    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9ff22c80-e444-46f0-97d8-0cb8247f2231" (UID: "9ff22c80-e444-46f0-97d8-0cb8247f2231"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:44:12.874646 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.874477    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9ff22c80-e444-46f0-97d8-0cb8247f2231" (UID: "9ff22c80-e444-46f0-97d8-0cb8247f2231"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:44:12.874646 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.874525    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9ff22c80-e444-46f0-97d8-0cb8247f2231" (UID: "9ff22c80-e444-46f0-97d8-0cb8247f2231"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:44:12.874797 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.874675    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:12.874797 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.874692    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:12.874797 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.874705    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:12.874797 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.874708    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-istio-data" (OuterVolumeSpecName: "istio-data") pod "9ff22c80-e444-46f0-97d8-0cb8247f2231" (UID: "9ff22c80-e444-46f0-97d8-0cb8247f2231"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:44:12.874797 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.874725    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ff22c80-e444-46f0-97d8-0cb8247f2231-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9ff22c80-e444-46f0-97d8-0cb8247f2231" (UID: "9ff22c80-e444-46f0-97d8-0cb8247f2231"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:44:12.876359 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.876333    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9ff22c80-e444-46f0-97d8-0cb8247f2231" (UID: "9ff22c80-e444-46f0-97d8-0cb8247f2231"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:44:12.876478 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.876450    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ff22c80-e444-46f0-97d8-0cb8247f2231-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9ff22c80-e444-46f0-97d8-0cb8247f2231" (UID: "9ff22c80-e444-46f0-97d8-0cb8247f2231"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 13:44:12.876706 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.876689    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ff22c80-e444-46f0-97d8-0cb8247f2231-kube-api-access-kbsjd" (OuterVolumeSpecName: "kube-api-access-kbsjd") pod "9ff22c80-e444-46f0-97d8-0cb8247f2231" (UID: "9ff22c80-e444-46f0-97d8-0cb8247f2231"). InnerVolumeSpecName "kube-api-access-kbsjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:44:12.876768 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.876743    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ff22c80-e444-46f0-97d8-0cb8247f2231-istio-token" (OuterVolumeSpecName: "istio-token") pod "9ff22c80-e444-46f0-97d8-0cb8247f2231" (UID: "9ff22c80-e444-46f0-97d8-0cb8247f2231"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:44:12.975517 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.975481    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:12.975517 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.975511    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ff22c80-e444-46f0-97d8-0cb8247f2231-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:12.975517 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.975522    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ff22c80-e444-46f0-97d8-0cb8247f2231-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:12.975517 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.975531    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ff22c80-e444-46f0-97d8-0cb8247f2231-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:12.975793 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.975538    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ff22c80-e444-46f0-97d8-0cb8247f2231-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:12.975793 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:12.975547    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbsjd\" (UniqueName: \"kubernetes.io/projected/9ff22c80-e444-46f0-97d8-0cb8247f2231-kube-api-access-kbsjd\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:13.378270 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.378233    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf"]
Sep 09 13:44:13.378681 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.378663    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8d6623e-e731-4ace-8f3b-f44db09987c8" containerName="istio-proxy"
Sep 09 13:44:13.378775 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.378684    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8d6623e-e731-4ace-8f3b-f44db09987c8" containerName="istio-proxy"
Sep 09 13:44:13.378775 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.378698    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ff22c80-e444-46f0-97d8-0cb8247f2231" containerName="istio-proxy"
Sep 09 13:44:13.378775 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.378705    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ff22c80-e444-46f0-97d8-0cb8247f2231" containerName="istio-proxy"
Sep 09 13:44:13.378775 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.378721    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ff22c80-e444-46f0-97d8-0cb8247f2231" containerName="istio-proxy"
Sep 09 13:44:13.378775 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.378728    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ff22c80-e444-46f0-97d8-0cb8247f2231" containerName="istio-proxy"
Sep 09 13:44:13.378775 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.378745    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a85c4918-fb24-469c-8341-af0fce2d56c4" containerName="revoke-keys"
Sep 09 13:44:13.378775 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.378752    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a85c4918-fb24-469c-8341-af0fce2d56c4" containerName="revoke-keys"
Sep 09 13:44:13.378775 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.378768    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8d6623e-e731-4ace-8f3b-f44db09987c8" containerName="istio-proxy"
Sep 09 13:44:13.378775 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.378776    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8d6623e-e731-4ace-8f3b-f44db09987c8" containerName="istio-proxy"
Sep 09 13:44:13.379247 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.378879    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ff22c80-e444-46f0-97d8-0cb8247f2231" containerName="istio-proxy"
Sep 09 13:44:13.379247 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.378894    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a85c4918-fb24-469c-8341-af0fce2d56c4" containerName="revoke-keys"
Sep 09 13:44:13.379247 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.378906    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8d6623e-e731-4ace-8f3b-f44db09987c8" containerName="istio-proxy"
Sep 09 13:44:13.379247 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.378917    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8d6623e-e731-4ace-8f3b-f44db09987c8" containerName="istio-proxy"
Sep 09 13:44:13.379711 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.379687    2566 generic.go:356] "Generic (PLEG): container finished" podID="9ff22c80-e444-46f0-97d8-0cb8247f2231" containerID="e736a806891a50d3e4d34db269d7a178d696a6161d53fcd22ef7aecd19b94c36" exitCode=0
Sep 09 13:44:13.379840 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.379822    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"
Sep 09 13:44:13.382161 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.382138    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl" event={"ID":"9ff22c80-e444-46f0-97d8-0cb8247f2231","Type":"ContainerDied","Data":"e736a806891a50d3e4d34db269d7a178d696a6161d53fcd22ef7aecd19b94c36"}
Sep 09 13:44:13.382257 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.382164    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl" event={"ID":"9ff22c80-e444-46f0-97d8-0cb8247f2231","Type":"ContainerDied","Data":"67890444f6c0d343ea3d17077cb1ca106bb5f4ab05b438c9e1830e6db94d4646"}
Sep 09 13:44:13.382257 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.382180    2566 scope.go:117] "RemoveContainer" containerID="e736a806891a50d3e4d34db269d7a178d696a6161d53fcd22ef7aecd19b94c36"
Sep 09 13:44:13.382257 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.382237    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf"
Sep 09 13:44:13.384301 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.384282    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-8f2df2\""
Sep 09 13:44:13.384413 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.384354    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-8f2df2-dockercfg-9hvht\""
Sep 09 13:44:13.393141 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.393119    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf"]
Sep 09 13:44:13.393401 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.393383    2566 scope.go:117] "RemoveContainer" containerID="a05e356d07ca65e07301901373c80c84624bd06460b0faaac10ab90d9962f3fe"
Sep 09 13:44:13.402693 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.402634    2566 scope.go:117] "RemoveContainer" containerID="e736a806891a50d3e4d34db269d7a178d696a6161d53fcd22ef7aecd19b94c36"
Sep 09 13:44:13.403064 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:44:13.403039    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e736a806891a50d3e4d34db269d7a178d696a6161d53fcd22ef7aecd19b94c36\": container with ID starting with e736a806891a50d3e4d34db269d7a178d696a6161d53fcd22ef7aecd19b94c36 not found: ID does not exist" containerID="e736a806891a50d3e4d34db269d7a178d696a6161d53fcd22ef7aecd19b94c36"
Sep 09 13:44:13.403185 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.403158    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e736a806891a50d3e4d34db269d7a178d696a6161d53fcd22ef7aecd19b94c36"} err="failed to get container status \"e736a806891a50d3e4d34db269d7a178d696a6161d53fcd22ef7aecd19b94c36\": rpc error: code = NotFound desc = could not find container \"e736a806891a50d3e4d34db269d7a178d696a6161d53fcd22ef7aecd19b94c36\": container with ID starting with e736a806891a50d3e4d34db269d7a178d696a6161d53fcd22ef7aecd19b94c36 not found: ID does not exist"
Sep 09 13:44:13.403276 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.403199    2566 scope.go:117] "RemoveContainer" containerID="a05e356d07ca65e07301901373c80c84624bd06460b0faaac10ab90d9962f3fe"
Sep 09 13:44:13.403586 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:44:13.403566    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a05e356d07ca65e07301901373c80c84624bd06460b0faaac10ab90d9962f3fe\": container with ID starting with a05e356d07ca65e07301901373c80c84624bd06460b0faaac10ab90d9962f3fe not found: ID does not exist" containerID="a05e356d07ca65e07301901373c80c84624bd06460b0faaac10ab90d9962f3fe"
Sep 09 13:44:13.403658 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.403593    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a05e356d07ca65e07301901373c80c84624bd06460b0faaac10ab90d9962f3fe"} err="failed to get container status \"a05e356d07ca65e07301901373c80c84624bd06460b0faaac10ab90d9962f3fe\": rpc error: code = NotFound desc = could not find container \"a05e356d07ca65e07301901373c80c84624bd06460b0faaac10ab90d9962f3fe\": container with ID starting with a05e356d07ca65e07301901373c80c84624bd06460b0faaac10ab90d9962f3fe not found: ID does not exist"
Sep 09 13:44:13.420066 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.419986    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"]
Sep 09 13:44:13.421741 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.421721    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-093e38-openshift-default-6684f58bd-zc4nl"]
Sep 09 13:44:13.480116 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.480075    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rds56\" (UniqueName: \"kubernetes.io/projected/c7f39f3f-c016-4b65-a3a3-17aebb601d4e-kube-api-access-rds56\") pod \"payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf\" (UID: \"c7f39f3f-c016-4b65-a3a3-17aebb601d4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf"
Sep 09 13:44:13.480316 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.480224    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7f39f3f-c016-4b65-a3a3-17aebb601d4e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf\" (UID: \"c7f39f3f-c016-4b65-a3a3-17aebb601d4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf"
Sep 09 13:44:13.581031 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.580997    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7f39f3f-c016-4b65-a3a3-17aebb601d4e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf\" (UID: \"c7f39f3f-c016-4b65-a3a3-17aebb601d4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf"
Sep 09 13:44:13.581490 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.581056    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rds56\" (UniqueName: \"kubernetes.io/projected/c7f39f3f-c016-4b65-a3a3-17aebb601d4e-kube-api-access-rds56\") pod \"payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf\" (UID: \"c7f39f3f-c016-4b65-a3a3-17aebb601d4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf"
Sep 09 13:44:13.581712 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.581690    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7f39f3f-c016-4b65-a3a3-17aebb601d4e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf\" (UID: \"c7f39f3f-c016-4b65-a3a3-17aebb601d4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf"
Sep 09 13:44:13.594037 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.594006    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rds56\" (UniqueName: \"kubernetes.io/projected/c7f39f3f-c016-4b65-a3a3-17aebb601d4e-kube-api-access-rds56\") pod \"payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf\" (UID: \"c7f39f3f-c016-4b65-a3a3-17aebb601d4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf"
Sep 09 13:44:13.694051 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.693946    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf"
Sep 09 13:44:13.971095 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:13.971070    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf"]
Sep 09 13:44:13.973041 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:44:13.973012    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f39f3f_c016_4b65_a3a3_17aebb601d4e.slice/crio-c8c7132d5b77bf45be60842ee87b058103c552f5aa1164fa4af4e8e71ef1a88b WatchSource:0}: Error finding container c8c7132d5b77bf45be60842ee87b058103c552f5aa1164fa4af4e8e71ef1a88b: Status 404 returned error can't find the container with id c8c7132d5b77bf45be60842ee87b058103c552f5aa1164fa4af4e8e71ef1a88b
Sep 09 13:44:14.388988 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:14.388950    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf" event={"ID":"c7f39f3f-c016-4b65-a3a3-17aebb601d4e","Type":"ContainerStarted","Data":"3f4cb56a907a4cf8f45dd736ee67c0b85acf178c05e5c405298b444c486a0e7e"}
Sep 09 13:44:14.389182 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:14.389003    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf" event={"ID":"c7f39f3f-c016-4b65-a3a3-17aebb601d4e","Type":"ContainerStarted","Data":"c8c7132d5b77bf45be60842ee87b058103c552f5aa1164fa4af4e8e71ef1a88b"}
Sep 09 13:44:14.389182 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:14.389084    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf"
Sep 09 13:44:14.416070 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:14.416018    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf" podStartSLOduration=1.416000514 podStartE2EDuration="1.416000514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:44:14.413577494 +0000 UTC m=+1087.773427319" watchObservedRunningTime="2026-09-09 13:44:14.416000514 +0000 UTC m=+1087.775850340"
Sep 09 13:44:15.150425 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:15.150392    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ff22c80-e444-46f0-97d8-0cb8247f2231" path="/var/lib/kubelet/pods/9ff22c80-e444-46f0-97d8-0cb8247f2231/volumes"
Sep 09 13:44:17.751560 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:17.751530    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:44:17.820010 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:17.819927    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2323ec19-7ff6-470a-b880-6cde1ce7104f-istio-podinfo\") pod \"2323ec19-7ff6-470a-b880-6cde1ce7104f\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") "
Sep 09 13:44:17.820010 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:17.819965    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-istio-envoy\") pod \"2323ec19-7ff6-470a-b880-6cde1ce7104f\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") "
Sep 09 13:44:17.820010 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:17.820013    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxrmf\" (UniqueName: \"kubernetes.io/projected/2323ec19-7ff6-470a-b880-6cde1ce7104f-kube-api-access-lxrmf\") pod \"2323ec19-7ff6-470a-b880-6cde1ce7104f\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") "
Sep 09 13:44:17.820300 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:17.820041    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-workload-socket\") pod \"2323ec19-7ff6-470a-b880-6cde1ce7104f\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") "
Sep 09 13:44:17.820300 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:17.820073    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2323ec19-7ff6-470a-b880-6cde1ce7104f-istiod-ca-cert\") pod \"2323ec19-7ff6-470a-b880-6cde1ce7104f\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") "
Sep 09 13:44:17.820300 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:17.820102    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2323ec19-7ff6-470a-b880-6cde1ce7104f-istio-token\") pod \"2323ec19-7ff6-470a-b880-6cde1ce7104f\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") "
Sep 09 13:44:17.820300 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:17.820150    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-workload-certs\") pod \"2323ec19-7ff6-470a-b880-6cde1ce7104f\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") "
Sep 09 13:44:17.820300 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:17.820186    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-credential-socket\") pod \"2323ec19-7ff6-470a-b880-6cde1ce7104f\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") "
Sep 09 13:44:17.820300 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:17.820216    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-istio-data\") pod \"2323ec19-7ff6-470a-b880-6cde1ce7104f\" (UID: \"2323ec19-7ff6-470a-b880-6cde1ce7104f\") "
Sep 09 13:44:17.820606 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:17.820391    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2323ec19-7ff6-470a-b880-6cde1ce7104f" (UID: "2323ec19-7ff6-470a-b880-6cde1ce7104f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:44:17.820606 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:17.820536    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:17.820606 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:17.820565    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2323ec19-7ff6-470a-b880-6cde1ce7104f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2323ec19-7ff6-470a-b880-6cde1ce7104f" (UID: "2323ec19-7ff6-470a-b880-6cde1ce7104f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:44:17.820606 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:17.820572    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2323ec19-7ff6-470a-b880-6cde1ce7104f" (UID: "2323ec19-7ff6-470a-b880-6cde1ce7104f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:44:17.820833 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:17.820607    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2323ec19-7ff6-470a-b880-6cde1ce7104f" (UID: "2323ec19-7ff6-470a-b880-6cde1ce7104f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:44:17.820833 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:17.820690    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-istio-data" (OuterVolumeSpecName: "istio-data") pod "2323ec19-7ff6-470a-b880-6cde1ce7104f" (UID: "2323ec19-7ff6-470a-b880-6cde1ce7104f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:44:17.822302 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:17.822272    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2323ec19-7ff6-470a-b880-6cde1ce7104f" (UID: "2323ec19-7ff6-470a-b880-6cde1ce7104f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:44:17.822302 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:17.822288    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2323ec19-7ff6-470a-b880-6cde1ce7104f-istio-token" (OuterVolumeSpecName: "istio-token") pod "2323ec19-7ff6-470a-b880-6cde1ce7104f" (UID: "2323ec19-7ff6-470a-b880-6cde1ce7104f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:44:17.822455 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:17.822334    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2323ec19-7ff6-470a-b880-6cde1ce7104f-kube-api-access-lxrmf" (OuterVolumeSpecName: "kube-api-access-lxrmf") pod "2323ec19-7ff6-470a-b880-6cde1ce7104f" (UID: "2323ec19-7ff6-470a-b880-6cde1ce7104f"). InnerVolumeSpecName "kube-api-access-lxrmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:44:17.822455 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:17.822361    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2323ec19-7ff6-470a-b880-6cde1ce7104f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2323ec19-7ff6-470a-b880-6cde1ce7104f" (UID: "2323ec19-7ff6-470a-b880-6cde1ce7104f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 13:44:17.921493 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:17.921463    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:17.921493 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:17.921490    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:17.921493 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:17.921499    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2323ec19-7ff6-470a-b880-6cde1ce7104f-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:17.921729 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:17.921507    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:17.921729 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:17.921516    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxrmf\" (UniqueName: \"kubernetes.io/projected/2323ec19-7ff6-470a-b880-6cde1ce7104f-kube-api-access-lxrmf\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:17.921729 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:17.921525    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2323ec19-7ff6-470a-b880-6cde1ce7104f-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:17.921729 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:17.921533    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2323ec19-7ff6-470a-b880-6cde1ce7104f-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:17.921729 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:17.921540    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2323ec19-7ff6-470a-b880-6cde1ce7104f-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:18.409949 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:18.409918    2566 generic.go:356] "Generic (PLEG): container finished" podID="2323ec19-7ff6-470a-b880-6cde1ce7104f" containerID="044acb6b024a92e33dd8b1d6a1bf3877d3b510275f716929b0eb331e84b4a37e" exitCode=0
Sep 09 13:44:18.410130 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:18.409966    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"
Sep 09 13:44:18.410130 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:18.409983    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww" event={"ID":"2323ec19-7ff6-470a-b880-6cde1ce7104f","Type":"ContainerDied","Data":"044acb6b024a92e33dd8b1d6a1bf3877d3b510275f716929b0eb331e84b4a37e"}
Sep 09 13:44:18.410130 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:18.410027    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww" event={"ID":"2323ec19-7ff6-470a-b880-6cde1ce7104f","Type":"ContainerDied","Data":"fbaa48043d1ad911c80a9c5e1453374e3a473ce470ffa0cf179a09522d821f37"}
Sep 09 13:44:18.410130 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:18.410049    2566 scope.go:117] "RemoveContainer" containerID="044acb6b024a92e33dd8b1d6a1bf3877d3b510275f716929b0eb331e84b4a37e"
Sep 09 13:44:18.419315 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:18.419298    2566 scope.go:117] "RemoveContainer" containerID="5671894e44006add60f7115d25f05941b92c1b38c93e4e7cee21e03e99e5ded3"
Sep 09 13:44:18.427720 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:18.427701    2566 scope.go:117] "RemoveContainer" containerID="044acb6b024a92e33dd8b1d6a1bf3877d3b510275f716929b0eb331e84b4a37e"
Sep 09 13:44:18.428013 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:44:18.427993    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"044acb6b024a92e33dd8b1d6a1bf3877d3b510275f716929b0eb331e84b4a37e\": container with ID starting with 044acb6b024a92e33dd8b1d6a1bf3877d3b510275f716929b0eb331e84b4a37e not found: ID does not exist" containerID="044acb6b024a92e33dd8b1d6a1bf3877d3b510275f716929b0eb331e84b4a37e"
Sep 09 13:44:18.428072 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:18.428021    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"044acb6b024a92e33dd8b1d6a1bf3877d3b510275f716929b0eb331e84b4a37e"} err="failed to get container status \"044acb6b024a92e33dd8b1d6a1bf3877d3b510275f716929b0eb331e84b4a37e\": rpc error: code = NotFound desc = could not find container \"044acb6b024a92e33dd8b1d6a1bf3877d3b510275f716929b0eb331e84b4a37e\": container with ID starting with 044acb6b024a92e33dd8b1d6a1bf3877d3b510275f716929b0eb331e84b4a37e not found: ID does not exist"
Sep 09 13:44:18.428072 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:18.428038    2566 scope.go:117] "RemoveContainer" containerID="5671894e44006add60f7115d25f05941b92c1b38c93e4e7cee21e03e99e5ded3"
Sep 09 13:44:18.428246 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:44:18.428228    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5671894e44006add60f7115d25f05941b92c1b38c93e4e7cee21e03e99e5ded3\": container with ID starting with 5671894e44006add60f7115d25f05941b92c1b38c93e4e7cee21e03e99e5ded3 not found: ID does not exist" containerID="5671894e44006add60f7115d25f05941b92c1b38c93e4e7cee21e03e99e5ded3"
Sep 09 13:44:18.428283 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:18.428249    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5671894e44006add60f7115d25f05941b92c1b38c93e4e7cee21e03e99e5ded3"} err="failed to get container status \"5671894e44006add60f7115d25f05941b92c1b38c93e4e7cee21e03e99e5ded3\": rpc error: code = NotFound desc = could not find container \"5671894e44006add60f7115d25f05941b92c1b38c93e4e7cee21e03e99e5ded3\": container with ID starting with 5671894e44006add60f7115d25f05941b92c1b38c93e4e7cee21e03e99e5ded3 not found: ID does not exist"
Sep 09 13:44:18.431066 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:18.431042    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"]
Sep 09 13:44:18.435715 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:18.435684    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-905b058d-openshift-default-5676dddd6d-5fmww"]
Sep 09 13:44:19.149467 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:19.149430    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2323ec19-7ff6-470a-b880-6cde1ce7104f" path="/var/lib/kubelet/pods/2323ec19-7ff6-470a-b880-6cde1ce7104f/volumes"
Sep 09 13:44:25.396003 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:25.395969    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf"
Sep 09 13:44:26.481937 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.481895    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"]
Sep 09 13:44:26.482514 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.482437    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2323ec19-7ff6-470a-b880-6cde1ce7104f" containerName="istio-proxy"
Sep 09 13:44:26.482514 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.482458    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2323ec19-7ff6-470a-b880-6cde1ce7104f" containerName="istio-proxy"
Sep 09 13:44:26.482514 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.482486    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2323ec19-7ff6-470a-b880-6cde1ce7104f" containerName="istio-proxy"
Sep 09 13:44:26.482514 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.482495    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2323ec19-7ff6-470a-b880-6cde1ce7104f" containerName="istio-proxy"
Sep 09 13:44:26.482726 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.482572    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ff22c80-e444-46f0-97d8-0cb8247f2231" containerName="istio-proxy"
Sep 09 13:44:26.482726 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.482587    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2323ec19-7ff6-470a-b880-6cde1ce7104f" containerName="istio-proxy"
Sep 09 13:44:26.482726 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.482599    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2323ec19-7ff6-470a-b880-6cde1ce7104f" containerName="istio-proxy"
Sep 09 13:44:26.487124 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.487102    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:26.489806 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.489785    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-b5cba4-openshift-default-dockercfg-xst7h\""
Sep 09 13:44:26.496978 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.496949    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"]
Sep 09 13:44:26.601860 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.601790    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-workload-certs\") pod \"e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") " pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:26.602056 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.601900    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-credential-socket\") pod \"e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") " pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:26.602056 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.601961    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4535ca39-b804-4428-8a3f-3f272f672519-istiod-ca-cert\") pod \"e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") " pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:26.602056 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.602015    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-workload-socket\") pod \"e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") " pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:26.602056 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.602041    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nthsw\" (UniqueName: \"kubernetes.io/projected/4535ca39-b804-4428-8a3f-3f272f672519-kube-api-access-nthsw\") pod \"e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") " pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:26.602275 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.602090    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4535ca39-b804-4428-8a3f-3f272f672519-istio-token\") pod \"e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") " pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:26.602275 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.602157    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-istio-data\") pod \"e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") " pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:26.602275 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.602185    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-istio-envoy\") pod \"e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") " pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:26.602275 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.602249    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4535ca39-b804-4428-8a3f-3f272f672519-istio-podinfo\") pod \"e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") " pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:26.703573 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.703538    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-credential-socket\") pod \"e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") " pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:26.703757 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.703604    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4535ca39-b804-4428-8a3f-3f272f672519-istiod-ca-cert\") pod \"e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") " pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:26.703757 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.703639    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-workload-socket\") pod \"e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") " pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:26.703757 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.703669    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nthsw\" (UniqueName: \"kubernetes.io/projected/4535ca39-b804-4428-8a3f-3f272f672519-kube-api-access-nthsw\") pod \"e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") " pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:26.703757 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.703703    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4535ca39-b804-4428-8a3f-3f272f672519-istio-token\") pod \"e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") " pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:26.704038 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.703754    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-istio-data\") pod \"e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") " pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:26.704038 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.703796    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-istio-envoy\") pod \"e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") " pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:26.704169 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.704077    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-credential-socket\") pod \"e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") " pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:26.704169 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.704099    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-istio-data\") pod \"e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") " pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:26.704286 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.704222    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-workload-socket\") pod \"e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") " pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:26.704286 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.704221    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4535ca39-b804-4428-8a3f-3f272f672519-istio-podinfo\") pod \"e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") " pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:26.704401 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.704324    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-workload-certs\") pod \"e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") " pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:26.704569 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.704539    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4535ca39-b804-4428-8a3f-3f272f672519-istiod-ca-cert\") pod \"e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") " pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:26.704715 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.704650    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-workload-certs\") pod \"e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") " pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:26.706939 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.706914    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-istio-envoy\") pod \"e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") " pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:26.707174 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.707158    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4535ca39-b804-4428-8a3f-3f272f672519-istio-podinfo\") pod \"e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") " pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:26.721895 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.721860    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nthsw\" (UniqueName: \"kubernetes.io/projected/4535ca39-b804-4428-8a3f-3f272f672519-kube-api-access-nthsw\") pod \"e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") " pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:26.722067 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.721901    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4535ca39-b804-4428-8a3f-3f272f672519-istio-token\") pod \"e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") " pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:26.798930 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:26.798791    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:27.076237 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.076164    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"]
Sep 09 13:44:27.078988 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:44:27.078963    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4535ca39_b804_4428_8a3f_3f272f672519.slice/crio-ee7cdd029ef91c3226e08fd5703a43936475a7b75e00997831bca052cad105fe WatchSource:0}: Error finding container ee7cdd029ef91c3226e08fd5703a43936475a7b75e00997831bca052cad105fe: Status 404 returned error can't find the container with id ee7cdd029ef91c3226e08fd5703a43936475a7b75e00997831bca052cad105fe
Sep 09 13:44:27.081108 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.081073    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:44:27.081209 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.081137    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:44:27.081209 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.081166    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:44:27.457152 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.457120    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd" event={"ID":"4535ca39-b804-4428-8a3f-3f272f672519","Type":"ContainerStarted","Data":"0f9ee376eb50a08f235d0e3c399f29ea618782a9fdc1ca75258cc91f28022f2e"}
Sep 09 13:44:27.457341 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.457169    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd" event={"ID":"4535ca39-b804-4428-8a3f-3f272f672519","Type":"ContainerStarted","Data":"ee7cdd029ef91c3226e08fd5703a43936475a7b75e00997831bca052cad105fe"}
Sep 09 13:44:27.476385 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.476334    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd" podStartSLOduration=1.476314731 podStartE2EDuration="1.476314731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:44:27.475928996 +0000 UTC m=+1100.835778822" watchObservedRunningTime="2026-09-09 13:44:27.476314731 +0000 UTC m=+1100.836164559"
Sep 09 13:44:27.654226 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.654185    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"]
Sep 09 13:44:27.660979 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.660949    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"
Sep 09 13:44:27.663440 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.663384    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c9dd9a\"/\"openshift-service-ca.crt\""
Sep 09 13:44:27.663699 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.663630    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c9dd9a\"/\"default-dockercfg-vx777\""
Sep 09 13:44:27.663699 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.663675    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c9dd9a\"/\"noten-model-c9dd9a-kserve-self-signed-certs\""
Sep 09 13:44:27.664117 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.663952    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c9dd9a\"/\"kube-root-ca.crt\""
Sep 09 13:44:27.666110 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.666087    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"]
Sep 09 13:44:27.714203 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.714116    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23dddd8c-25e7-4ea5-9506-fe7a1126409e-model-cache\") pod \"noten-model-c9dd9a-kserve-754b456b55-vg8jl\" (UID: \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\") " pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"
Sep 09 13:44:27.714203 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.714163    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23dddd8c-25e7-4ea5-9506-fe7a1126409e-tmp-dir\") pod \"noten-model-c9dd9a-kserve-754b456b55-vg8jl\" (UID: \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\") " pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"
Sep 09 13:44:27.714203 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.714187    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23dddd8c-25e7-4ea5-9506-fe7a1126409e-home\") pod \"noten-model-c9dd9a-kserve-754b456b55-vg8jl\" (UID: \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\") " pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"
Sep 09 13:44:27.714422 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.714263    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23dddd8c-25e7-4ea5-9506-fe7a1126409e-dshm\") pod \"noten-model-c9dd9a-kserve-754b456b55-vg8jl\" (UID: \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\") " pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"
Sep 09 13:44:27.714422 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.714301    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23dddd8c-25e7-4ea5-9506-fe7a1126409e-tls-certs\") pod \"noten-model-c9dd9a-kserve-754b456b55-vg8jl\" (UID: \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\") " pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"
Sep 09 13:44:27.714422 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.714338    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcq5k\" (UniqueName: \"kubernetes.io/projected/23dddd8c-25e7-4ea5-9506-fe7a1126409e-kube-api-access-gcq5k\") pod \"noten-model-c9dd9a-kserve-754b456b55-vg8jl\" (UID: \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\") " pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"
Sep 09 13:44:27.799087 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.799050    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:27.815649 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.815622    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23dddd8c-25e7-4ea5-9506-fe7a1126409e-model-cache\") pod \"noten-model-c9dd9a-kserve-754b456b55-vg8jl\" (UID: \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\") " pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"
Sep 09 13:44:27.815839 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.815659    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23dddd8c-25e7-4ea5-9506-fe7a1126409e-tmp-dir\") pod \"noten-model-c9dd9a-kserve-754b456b55-vg8jl\" (UID: \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\") " pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"
Sep 09 13:44:27.815839 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.815679    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23dddd8c-25e7-4ea5-9506-fe7a1126409e-home\") pod \"noten-model-c9dd9a-kserve-754b456b55-vg8jl\" (UID: \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\") " pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"
Sep 09 13:44:27.815839 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.815732    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23dddd8c-25e7-4ea5-9506-fe7a1126409e-dshm\") pod \"noten-model-c9dd9a-kserve-754b456b55-vg8jl\" (UID: \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\") " pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"
Sep 09 13:44:27.815839 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.815762    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23dddd8c-25e7-4ea5-9506-fe7a1126409e-tls-certs\") pod \"noten-model-c9dd9a-kserve-754b456b55-vg8jl\" (UID: \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\") " pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"
Sep 09 13:44:27.815839 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.815800    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcq5k\" (UniqueName: \"kubernetes.io/projected/23dddd8c-25e7-4ea5-9506-fe7a1126409e-kube-api-access-gcq5k\") pod \"noten-model-c9dd9a-kserve-754b456b55-vg8jl\" (UID: \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\") " pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"
Sep 09 13:44:27.816103 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.816079    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23dddd8c-25e7-4ea5-9506-fe7a1126409e-model-cache\") pod \"noten-model-c9dd9a-kserve-754b456b55-vg8jl\" (UID: \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\") " pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"
Sep 09 13:44:27.816153 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.816123    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23dddd8c-25e7-4ea5-9506-fe7a1126409e-tmp-dir\") pod \"noten-model-c9dd9a-kserve-754b456b55-vg8jl\" (UID: \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\") " pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"
Sep 09 13:44:27.816433 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.816400    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23dddd8c-25e7-4ea5-9506-fe7a1126409e-home\") pod \"noten-model-c9dd9a-kserve-754b456b55-vg8jl\" (UID: \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\") " pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"
Sep 09 13:44:27.818342 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.818322    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23dddd8c-25e7-4ea5-9506-fe7a1126409e-dshm\") pod \"noten-model-c9dd9a-kserve-754b456b55-vg8jl\" (UID: \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\") " pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"
Sep 09 13:44:27.818459 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.818393    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23dddd8c-25e7-4ea5-9506-fe7a1126409e-tls-certs\") pod \"noten-model-c9dd9a-kserve-754b456b55-vg8jl\" (UID: \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\") " pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"
Sep 09 13:44:27.826579 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.826552    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcq5k\" (UniqueName: \"kubernetes.io/projected/23dddd8c-25e7-4ea5-9506-fe7a1126409e-kube-api-access-gcq5k\") pod \"noten-model-c9dd9a-kserve-754b456b55-vg8jl\" (UID: \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\") " pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"
Sep 09 13:44:27.971599 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:27.971511    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"
Sep 09 13:44:28.246734 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:28.246703    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"]
Sep 09 13:44:28.247580 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:44:28.247553    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23dddd8c_25e7_4ea5_9506_fe7a1126409e.slice/crio-b12c549a780399cebd288ff6755364f50570275d00d9e0e3f9f6fda029a7b52d WatchSource:0}: Error finding container b12c549a780399cebd288ff6755364f50570275d00d9e0e3f9f6fda029a7b52d: Status 404 returned error can't find the container with id b12c549a780399cebd288ff6755364f50570275d00d9e0e3f9f6fda029a7b52d
Sep 09 13:44:28.463667 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:28.463630    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl" event={"ID":"23dddd8c-25e7-4ea5-9506-fe7a1126409e","Type":"ContainerStarted","Data":"c668aa8829873ab91d30bdf16f7ef11deb558cb5191d7d6f0fe56ed8d6f6ea5d"}
Sep 09 13:44:28.463846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:28.463676    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl" event={"ID":"23dddd8c-25e7-4ea5-9506-fe7a1126409e","Type":"ContainerStarted","Data":"b12c549a780399cebd288ff6755364f50570275d00d9e0e3f9f6fda029a7b52d"}
Sep 09 13:44:28.482688 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:28.482642    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl" podStartSLOduration=1.482623113 podStartE2EDuration="1.482623113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:44:28.48091628 +0000 UTC m=+1101.840766106" watchObservedRunningTime="2026-09-09 13:44:28.482623113 +0000 UTC m=+1101.842472941"
Sep 09 13:44:28.804846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:28.804797    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:29.468490 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:29.468451    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:29.469215 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:29.469196    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:30.561373 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:30.561341    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"]
Sep 09 13:44:30.684452 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:30.684421    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"]
Sep 09 13:44:30.684671 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:30.684648    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549" podUID="fc2199c0-b068-46e6-8fa8-7510766412a0" containerName="istio-proxy" containerID="cri-o://e3f42a8b956534e45ecabb3649250f04795ab8180c3771536668953c08b3e689" gracePeriod=30
Sep 09 13:44:35.946834 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:35.946752    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:44:36.096679 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.096646    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-workload-certs\") pod \"fc2199c0-b068-46e6-8fa8-7510766412a0\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") "
Sep 09 13:44:36.096844 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.096691    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-credential-socket\") pod \"fc2199c0-b068-46e6-8fa8-7510766412a0\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") "
Sep 09 13:44:36.096844 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.096713    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc2199c0-b068-46e6-8fa8-7510766412a0-istiod-ca-cert\") pod \"fc2199c0-b068-46e6-8fa8-7510766412a0\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") "
Sep 09 13:44:36.096844 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.096744    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-istio-envoy\") pod \"fc2199c0-b068-46e6-8fa8-7510766412a0\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") "
Sep 09 13:44:36.096844 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.096787    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc2199c0-b068-46e6-8fa8-7510766412a0-istio-podinfo\") pod \"fc2199c0-b068-46e6-8fa8-7510766412a0\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") "
Sep 09 13:44:36.096844 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.096844    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc2199c0-b068-46e6-8fa8-7510766412a0-istio-token\") pod \"fc2199c0-b068-46e6-8fa8-7510766412a0\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") "
Sep 09 13:44:36.097096 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.096866    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-workload-socket\") pod \"fc2199c0-b068-46e6-8fa8-7510766412a0\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") "
Sep 09 13:44:36.097096 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.096893    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-istio-data\") pod \"fc2199c0-b068-46e6-8fa8-7510766412a0\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") "
Sep 09 13:44:36.097096 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.096925    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rqzk\" (UniqueName: \"kubernetes.io/projected/fc2199c0-b068-46e6-8fa8-7510766412a0-kube-api-access-4rqzk\") pod \"fc2199c0-b068-46e6-8fa8-7510766412a0\" (UID: \"fc2199c0-b068-46e6-8fa8-7510766412a0\") "
Sep 09 13:44:36.097096 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.096979    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fc2199c0-b068-46e6-8fa8-7510766412a0" (UID: "fc2199c0-b068-46e6-8fa8-7510766412a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:44:36.097096 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.097056    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fc2199c0-b068-46e6-8fa8-7510766412a0" (UID: "fc2199c0-b068-46e6-8fa8-7510766412a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:44:36.097351 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.097182    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fc2199c0-b068-46e6-8fa8-7510766412a0" (UID: "fc2199c0-b068-46e6-8fa8-7510766412a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:44:36.097351 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.097196    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-istio-data" (OuterVolumeSpecName: "istio-data") pod "fc2199c0-b068-46e6-8fa8-7510766412a0" (UID: "fc2199c0-b068-46e6-8fa8-7510766412a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:44:36.097467 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.097370    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:36.097467 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.097384    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc2199c0-b068-46e6-8fa8-7510766412a0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fc2199c0-b068-46e6-8fa8-7510766412a0" (UID: "fc2199c0-b068-46e6-8fa8-7510766412a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:44:36.097467 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.097393    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:36.097467 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.097411    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:36.097467 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.097427    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:36.099176 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.099150    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc2199c0-b068-46e6-8fa8-7510766412a0-istio-token" (OuterVolumeSpecName: "istio-token") pod "fc2199c0-b068-46e6-8fa8-7510766412a0" (UID: "fc2199c0-b068-46e6-8fa8-7510766412a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:44:36.099176 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.099166    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc2199c0-b068-46e6-8fa8-7510766412a0-kube-api-access-4rqzk" (OuterVolumeSpecName: "kube-api-access-4rqzk") pod "fc2199c0-b068-46e6-8fa8-7510766412a0" (UID: "fc2199c0-b068-46e6-8fa8-7510766412a0"). InnerVolumeSpecName "kube-api-access-4rqzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:44:36.099406 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.099390    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc2199c0-b068-46e6-8fa8-7510766412a0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fc2199c0-b068-46e6-8fa8-7510766412a0" (UID: "fc2199c0-b068-46e6-8fa8-7510766412a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 13:44:36.099465 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.099446    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fc2199c0-b068-46e6-8fa8-7510766412a0" (UID: "fc2199c0-b068-46e6-8fa8-7510766412a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:44:36.198402 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.198303    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rqzk\" (UniqueName: \"kubernetes.io/projected/fc2199c0-b068-46e6-8fa8-7510766412a0-kube-api-access-4rqzk\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:36.198402 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.198337    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc2199c0-b068-46e6-8fa8-7510766412a0-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:36.198402 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.198351    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc2199c0-b068-46e6-8fa8-7510766412a0-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:36.198402 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.198362    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc2199c0-b068-46e6-8fa8-7510766412a0-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:36.198402 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.198373    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc2199c0-b068-46e6-8fa8-7510766412a0-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:36.515021 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.514913    2566 generic.go:356] "Generic (PLEG): container finished" podID="fc2199c0-b068-46e6-8fa8-7510766412a0" containerID="e3f42a8b956534e45ecabb3649250f04795ab8180c3771536668953c08b3e689" exitCode=0
Sep 09 13:44:36.515021 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.514977    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"
Sep 09 13:44:36.515256 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.515015    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549" event={"ID":"fc2199c0-b068-46e6-8fa8-7510766412a0","Type":"ContainerDied","Data":"e3f42a8b956534e45ecabb3649250f04795ab8180c3771536668953c08b3e689"}
Sep 09 13:44:36.515256 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.515070    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549" event={"ID":"fc2199c0-b068-46e6-8fa8-7510766412a0","Type":"ContainerDied","Data":"66f77387b4f18dcd355ebe4970e590b3920a35d30c927bd1608d3ac57dd1d23f"}
Sep 09 13:44:36.515256 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.515097    2566 scope.go:117] "RemoveContainer" containerID="e3f42a8b956534e45ecabb3649250f04795ab8180c3771536668953c08b3e689"
Sep 09 13:44:36.524725 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.524699    2566 scope.go:117] "RemoveContainer" containerID="e3f42a8b956534e45ecabb3649250f04795ab8180c3771536668953c08b3e689"
Sep 09 13:44:36.525132 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:44:36.525114    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3f42a8b956534e45ecabb3649250f04795ab8180c3771536668953c08b3e689\": container with ID starting with e3f42a8b956534e45ecabb3649250f04795ab8180c3771536668953c08b3e689 not found: ID does not exist" containerID="e3f42a8b956534e45ecabb3649250f04795ab8180c3771536668953c08b3e689"
Sep 09 13:44:36.525219 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.525140    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3f42a8b956534e45ecabb3649250f04795ab8180c3771536668953c08b3e689"} err="failed to get container status \"e3f42a8b956534e45ecabb3649250f04795ab8180c3771536668953c08b3e689\": rpc error: code = NotFound desc = could not find container \"e3f42a8b956534e45ecabb3649250f04795ab8180c3771536668953c08b3e689\": container with ID starting with e3f42a8b956534e45ecabb3649250f04795ab8180c3771536668953c08b3e689 not found: ID does not exist"
Sep 09 13:44:36.535661 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.535636    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"]
Sep 09 13:44:36.539731 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.539702    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-c9dd9a-openshift-default-6cb99556b4-kn549"]
Sep 09 13:44:36.842010 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.841976    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"]
Sep 09 13:44:36.842354 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.842341    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc2199c0-b068-46e6-8fa8-7510766412a0" containerName="istio-proxy"
Sep 09 13:44:36.842397 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.842356    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc2199c0-b068-46e6-8fa8-7510766412a0" containerName="istio-proxy"
Sep 09 13:44:36.842432 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.842414    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc2199c0-b068-46e6-8fa8-7510766412a0" containerName="istio-proxy"
Sep 09 13:44:36.846688 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.846664    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:36.849036 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.849010    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-df48ae-openshift-default-dockercfg-jwfzk\""
Sep 09 13:44:36.864015 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:36.863990    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"]
Sep 09 13:44:37.012116 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.012068    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istio-data\") pod \"e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") " pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:37.012624 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.012128    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istiod-ca-cert\") pod \"e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") " pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:37.012624 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.012197    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p46bp\" (UniqueName: \"kubernetes.io/projected/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-kube-api-access-p46bp\") pod \"e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") " pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:37.012624 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.012226    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istio-envoy\") pod \"e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") " pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:37.012624 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.012284    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-workload-socket\") pod \"e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") " pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:37.012624 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.012317    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-credential-socket\") pod \"e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") " pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:37.012624 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.012354    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istio-token\") pod \"e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") " pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:37.012624 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.012397    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istio-podinfo\") pod \"e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") " pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:37.012624 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.012440    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-workload-certs\") pod \"e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") " pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:37.113355 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.113245    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-workload-socket\") pod \"e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") " pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:37.113595 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.113575    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-credential-socket\") pod \"e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") " pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:37.113704 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.113689    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istio-token\") pod \"e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") " pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:37.113905 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.113876    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istio-podinfo\") pod \"e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") " pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:37.114051 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.113936    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-workload-certs\") pod \"e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") " pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:37.114051 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.114018    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istio-data\") pod \"e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") " pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:37.114051 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.114045    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istiod-ca-cert\") pod \"e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") " pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:37.114224 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.114096    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p46bp\" (UniqueName: \"kubernetes.io/projected/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-kube-api-access-p46bp\") pod \"e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") " pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:37.114224 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.114118    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istio-envoy\") pod \"e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") " pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:37.114343 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.114302    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-workload-socket\") pod \"e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") " pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:37.114343 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.114312    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-credential-socket\") pod \"e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") " pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:37.114912 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.114531    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-workload-certs\") pod \"e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") " pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:37.115094 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.115027    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istio-data\") pod \"e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") " pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:37.115324 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.115300    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istiod-ca-cert\") pod \"e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") " pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:37.116653 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.116635    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istio-envoy\") pod \"e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") " pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:37.117118 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.117100    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istio-podinfo\") pod \"e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") " pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:37.123230 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.123208    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istio-token\") pod \"e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") " pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:37.123366 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.123346    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p46bp\" (UniqueName: \"kubernetes.io/projected/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-kube-api-access-p46bp\") pod \"e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") " pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:37.150327 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.150298    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc2199c0-b068-46e6-8fa8-7510766412a0" path="/var/lib/kubelet/pods/fc2199c0-b068-46e6-8fa8-7510766412a0/volumes"
Sep 09 13:44:37.157455 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.157435    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:37.460622 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.460542    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"]
Sep 09 13:44:37.463971 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:44:37.463943    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f6020ae_98a0_4ea4_b95a_32e247ab0c32.slice/crio-cbab4f2f4c8da61d8a3f1c429e9c039d7af003d26eff5e496794cb4f7d2c5edd WatchSource:0}: Error finding container cbab4f2f4c8da61d8a3f1c429e9c039d7af003d26eff5e496794cb4f7d2c5edd: Status 404 returned error can't find the container with id cbab4f2f4c8da61d8a3f1c429e9c039d7af003d26eff5e496794cb4f7d2c5edd
Sep 09 13:44:37.465901 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.465866    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:44:37.465981 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.465928    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:44:37.465981 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.465957    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:44:37.523965 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.523936    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg" event={"ID":"2f6020ae-98a0-4ea4-b95a-32e247ab0c32","Type":"ContainerStarted","Data":"cbab4f2f4c8da61d8a3f1c429e9c039d7af003d26eff5e496794cb4f7d2c5edd"}
Sep 09 13:44:37.971942 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:37.971903    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"
Sep 09 13:44:38.530904 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:38.530867    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg" event={"ID":"2f6020ae-98a0-4ea4-b95a-32e247ab0c32","Type":"ContainerStarted","Data":"4d4c0f39e8fcbb9360b8ed4c30bf7663ae9c1da3f663692b015968acc2e0dbc1"}
Sep 09 13:44:38.554950 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:38.554901    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg" podStartSLOduration=2.554880812 podStartE2EDuration="2.554880812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:44:38.552808408 +0000 UTC m=+1111.912658237" watchObservedRunningTime="2026-09-09 13:44:38.554880812 +0000 UTC m=+1111.914730637"
Sep 09 13:44:39.158397 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:39.158367    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:39.163704 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:39.163680    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:39.535962 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:39.535868    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:39.536969 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:39.536948    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:42.238234 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:42.238146    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc"]
Sep 09 13:44:42.248579 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:42.248545    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc"
Sep 09 13:44:42.251016 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:42.250993    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-df48ae-dockercfg-njwgd\""
Sep 09 13:44:42.251347 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:42.251332    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-df48ae\""
Sep 09 13:44:42.256096 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:42.256070    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc"]
Sep 09 13:44:42.362784 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:42.362751    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srzng\" (UniqueName: \"kubernetes.io/projected/892980ec-4634-41a6-8b28-f80043e08558-kube-api-access-srzng\") pod \"payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc\" (UID: \"892980ec-4634-41a6-8b28-f80043e08558\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc"
Sep 09 13:44:42.362784 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:42.362794    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/892980ec-4634-41a6-8b28-f80043e08558-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc\" (UID: \"892980ec-4634-41a6-8b28-f80043e08558\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc"
Sep 09 13:44:42.463725 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:42.463681    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srzng\" (UniqueName: \"kubernetes.io/projected/892980ec-4634-41a6-8b28-f80043e08558-kube-api-access-srzng\") pod \"payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc\" (UID: \"892980ec-4634-41a6-8b28-f80043e08558\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc"
Sep 09 13:44:42.463932 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:42.463745    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/892980ec-4634-41a6-8b28-f80043e08558-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc\" (UID: \"892980ec-4634-41a6-8b28-f80043e08558\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc"
Sep 09 13:44:42.464463 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:42.464444    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/892980ec-4634-41a6-8b28-f80043e08558-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc\" (UID: \"892980ec-4634-41a6-8b28-f80043e08558\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc"
Sep 09 13:44:42.471973 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:42.471936    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srzng\" (UniqueName: \"kubernetes.io/projected/892980ec-4634-41a6-8b28-f80043e08558-kube-api-access-srzng\") pod \"payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc\" (UID: \"892980ec-4634-41a6-8b28-f80043e08558\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc"
Sep 09 13:44:42.560018 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:42.559924    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc"
Sep 09 13:44:42.844228 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:42.844186    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc"]
Sep 09 13:44:42.846253 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:44:42.846223    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod892980ec_4634_41a6_8b28_f80043e08558.slice/crio-6496d7f79620e24b340dacf8f92be57d50ee72164c2fc95aa605fa76538d3b0a WatchSource:0}: Error finding container 6496d7f79620e24b340dacf8f92be57d50ee72164c2fc95aa605fa76538d3b0a: Status 404 returned error can't find the container with id 6496d7f79620e24b340dacf8f92be57d50ee72164c2fc95aa605fa76538d3b0a
Sep 09 13:44:43.558928 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:43.558892    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc" event={"ID":"892980ec-4634-41a6-8b28-f80043e08558","Type":"ContainerStarted","Data":"cd159c2ee842a458ef365381612b5b7db31976e097f19f96fe06dd1c53936915"}
Sep 09 13:44:43.559313 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:43.558944    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc" event={"ID":"892980ec-4634-41a6-8b28-f80043e08558","Type":"ContainerStarted","Data":"6496d7f79620e24b340dacf8f92be57d50ee72164c2fc95aa605fa76538d3b0a"}
Sep 09 13:44:43.559313 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:43.559036    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc"
Sep 09 13:44:43.578365 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:43.578317    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc" podStartSLOduration=1.578301687 podStartE2EDuration="1.578301687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:44:43.577496887 +0000 UTC m=+1116.937346727" watchObservedRunningTime="2026-09-09 13:44:43.578301687 +0000 UTC m=+1116.938151555"
Sep 09 13:44:44.956779 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:44.956748    2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": read tcp 10.132.0.2:38220->10.132.0.57:15021: read: connection reset by peer" start-of-body=
Sep 09 13:44:44.957188 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:44.956949    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd" podUID="4535ca39-b804-4428-8a3f-3f272f672519" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": read tcp 10.132.0.2:38220->10.132.0.57:15021: read: connection reset by peer"
Sep 09 13:44:45.573862 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:45.573806    2566 generic.go:356] "Generic (PLEG): container finished" podID="4535ca39-b804-4428-8a3f-3f272f672519" containerID="0f9ee376eb50a08f235d0e3c399f29ea618782a9fdc1ca75258cc91f28022f2e" exitCode=0
Sep 09 13:44:45.574075 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:45.573870    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd" event={"ID":"4535ca39-b804-4428-8a3f-3f272f672519","Type":"ContainerDied","Data":"0f9ee376eb50a08f235d0e3c399f29ea618782a9fdc1ca75258cc91f28022f2e"}
Sep 09 13:44:45.574377 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:45.574360    2566 scope.go:117] "RemoveContainer" containerID="0f9ee376eb50a08f235d0e3c399f29ea618782a9fdc1ca75258cc91f28022f2e"
Sep 09 13:44:45.575545 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:45.575521    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:44:45.575607 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:45.575590    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:44:45.575651 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:45.575624    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:44:45.581049 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:45.581023    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl" podUID="23dddd8c-25e7-4ea5-9506-fe7a1126409e" containerName="main" containerID="cri-o://c668aa8829873ab91d30bdf16f7ef11deb558cb5191d7d6f0fe56ed8d6f6ea5d" gracePeriod=45
Sep 09 13:44:45.841730 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:45.841704    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"
Sep 09 13:44:45.999365 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:45.999324    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcq5k\" (UniqueName: \"kubernetes.io/projected/23dddd8c-25e7-4ea5-9506-fe7a1126409e-kube-api-access-gcq5k\") pod \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\" (UID: \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\") "
Sep 09 13:44:45.999869 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:45.999377    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23dddd8c-25e7-4ea5-9506-fe7a1126409e-dshm\") pod \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\" (UID: \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\") "
Sep 09 13:44:45.999869 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:45.999407    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23dddd8c-25e7-4ea5-9506-fe7a1126409e-tmp-dir\") pod \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\" (UID: \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\") "
Sep 09 13:44:45.999869 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:45.999449    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23dddd8c-25e7-4ea5-9506-fe7a1126409e-home\") pod \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\" (UID: \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\") "
Sep 09 13:44:45.999869 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:45.999492    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23dddd8c-25e7-4ea5-9506-fe7a1126409e-tls-certs\") pod \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\" (UID: \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\") "
Sep 09 13:44:45.999869 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:45.999526    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23dddd8c-25e7-4ea5-9506-fe7a1126409e-model-cache\") pod \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\" (UID: \"23dddd8c-25e7-4ea5-9506-fe7a1126409e\") "
Sep 09 13:44:45.999869 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:45.999684    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23dddd8c-25e7-4ea5-9506-fe7a1126409e-home" (OuterVolumeSpecName: "home") pod "23dddd8c-25e7-4ea5-9506-fe7a1126409e" (UID: "23dddd8c-25e7-4ea5-9506-fe7a1126409e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:44:45.999869 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:45.999707    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23dddd8c-25e7-4ea5-9506-fe7a1126409e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "23dddd8c-25e7-4ea5-9506-fe7a1126409e" (UID: "23dddd8c-25e7-4ea5-9506-fe7a1126409e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:44:46.000261 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:45.999899    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23dddd8c-25e7-4ea5-9506-fe7a1126409e-model-cache" (OuterVolumeSpecName: "model-cache") pod "23dddd8c-25e7-4ea5-9506-fe7a1126409e" (UID: "23dddd8c-25e7-4ea5-9506-fe7a1126409e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:44:46.000261 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:45.999964    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23dddd8c-25e7-4ea5-9506-fe7a1126409e-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:46.000261 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:45.999983    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23dddd8c-25e7-4ea5-9506-fe7a1126409e-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:46.000261 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:45.999998    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23dddd8c-25e7-4ea5-9506-fe7a1126409e-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:46.002202 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:46.002169    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23dddd8c-25e7-4ea5-9506-fe7a1126409e-kube-api-access-gcq5k" (OuterVolumeSpecName: "kube-api-access-gcq5k") pod "23dddd8c-25e7-4ea5-9506-fe7a1126409e" (UID: "23dddd8c-25e7-4ea5-9506-fe7a1126409e"). InnerVolumeSpecName "kube-api-access-gcq5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:44:46.002202 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:46.002189    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23dddd8c-25e7-4ea5-9506-fe7a1126409e-dshm" (OuterVolumeSpecName: "dshm") pod "23dddd8c-25e7-4ea5-9506-fe7a1126409e" (UID: "23dddd8c-25e7-4ea5-9506-fe7a1126409e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:44:46.002357 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:46.002203    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23dddd8c-25e7-4ea5-9506-fe7a1126409e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "23dddd8c-25e7-4ea5-9506-fe7a1126409e" (UID: "23dddd8c-25e7-4ea5-9506-fe7a1126409e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:44:46.100620 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:46.100579    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23dddd8c-25e7-4ea5-9506-fe7a1126409e-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:46.100620 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:46.100615    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcq5k\" (UniqueName: \"kubernetes.io/projected/23dddd8c-25e7-4ea5-9506-fe7a1126409e-kube-api-access-gcq5k\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:46.100620 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:46.100627    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23dddd8c-25e7-4ea5-9506-fe7a1126409e-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:44:46.580929 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:46.580895    2566 generic.go:356] "Generic (PLEG): container finished" podID="23dddd8c-25e7-4ea5-9506-fe7a1126409e" containerID="c668aa8829873ab91d30bdf16f7ef11deb558cb5191d7d6f0fe56ed8d6f6ea5d" exitCode=0
Sep 09 13:44:46.581122 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:46.580939    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"
Sep 09 13:44:46.581122 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:46.580978    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl" event={"ID":"23dddd8c-25e7-4ea5-9506-fe7a1126409e","Type":"ContainerDied","Data":"c668aa8829873ab91d30bdf16f7ef11deb558cb5191d7d6f0fe56ed8d6f6ea5d"}
Sep 09 13:44:46.581122 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:46.581015    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl" event={"ID":"23dddd8c-25e7-4ea5-9506-fe7a1126409e","Type":"ContainerDied","Data":"b12c549a780399cebd288ff6755364f50570275d00d9e0e3f9f6fda029a7b52d"}
Sep 09 13:44:46.581122 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:46.581033    2566 scope.go:117] "RemoveContainer" containerID="c668aa8829873ab91d30bdf16f7ef11deb558cb5191d7d6f0fe56ed8d6f6ea5d"
Sep 09 13:44:46.583176 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:46.583154    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd" event={"ID":"4535ca39-b804-4428-8a3f-3f272f672519","Type":"ContainerStarted","Data":"5983eacb6eac1b48b214b4a3740e453d8975e79277ff1808ad34249e728e6075"}
Sep 09 13:44:46.591559 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:46.591535    2566 scope.go:117] "RemoveContainer" containerID="c668aa8829873ab91d30bdf16f7ef11deb558cb5191d7d6f0fe56ed8d6f6ea5d"
Sep 09 13:44:46.592030 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:44:46.591945    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c668aa8829873ab91d30bdf16f7ef11deb558cb5191d7d6f0fe56ed8d6f6ea5d\": container with ID starting with c668aa8829873ab91d30bdf16f7ef11deb558cb5191d7d6f0fe56ed8d6f6ea5d not found: ID does not exist" containerID="c668aa8829873ab91d30bdf16f7ef11deb558cb5191d7d6f0fe56ed8d6f6ea5d"
Sep 09 13:44:46.592030 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:46.591982    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c668aa8829873ab91d30bdf16f7ef11deb558cb5191d7d6f0fe56ed8d6f6ea5d"} err="failed to get container status \"c668aa8829873ab91d30bdf16f7ef11deb558cb5191d7d6f0fe56ed8d6f6ea5d\": rpc error: code = NotFound desc = could not find container \"c668aa8829873ab91d30bdf16f7ef11deb558cb5191d7d6f0fe56ed8d6f6ea5d\": container with ID starting with c668aa8829873ab91d30bdf16f7ef11deb558cb5191d7d6f0fe56ed8d6f6ea5d not found: ID does not exist"
Sep 09 13:44:46.622389 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:46.622350    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"]
Sep 09 13:44:46.628346 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:46.628317    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-c9dd9a/noten-model-c9dd9a-kserve-754b456b55-vg8jl"]
Sep 09 13:44:46.800035 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:46.799993    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:46.800334 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:46.800306    2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 09 13:44:46.800425 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:46.800370    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd" podUID="4535ca39-b804-4428-8a3f-3f272f672519" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 09 13:44:47.151939 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:47.151891    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23dddd8c-25e7-4ea5-9506-fe7a1126409e" path="/var/lib/kubelet/pods/23dddd8c-25e7-4ea5-9506-fe7a1126409e/volumes"
Sep 09 13:44:47.800278 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:47.800232    2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 09 13:44:47.800542 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:47.800311    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd" podUID="4535ca39-b804-4428-8a3f-3f272f672519" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 09 13:44:48.800038 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:48.799988    2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 09 13:44:48.800594 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:48.800066    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd" podUID="4535ca39-b804-4428-8a3f-3f272f672519" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 09 13:44:49.800300 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:49.800257    2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 09 13:44:49.800950 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:49.800907    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd" podUID="4535ca39-b804-4428-8a3f-3f272f672519" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 09 13:44:50.800310 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:50.800260    2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 09 13:44:50.800876 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:50.800336    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd" podUID="4535ca39-b804-4428-8a3f-3f272f672519" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 09 13:44:51.799606 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:51.799568    2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 09 13:44:51.799917 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:51.799889    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd" podUID="4535ca39-b804-4428-8a3f-3f272f672519" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 09 13:44:52.803267 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:52.803236    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:52.803646 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:52.803582    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:52.804192 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:52.804174    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:44:54.566646 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:54.566609    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc"
Sep 09 13:44:54.948953 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:54.948920    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"]
Sep 09 13:44:54.949306 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:54.949289    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23dddd8c-25e7-4ea5-9506-fe7a1126409e" containerName="main"
Sep 09 13:44:54.949306 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:54.949304    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="23dddd8c-25e7-4ea5-9506-fe7a1126409e" containerName="main"
Sep 09 13:44:54.949441 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:54.949360    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="23dddd8c-25e7-4ea5-9506-fe7a1126409e" containerName="main"
Sep 09 13:44:54.954238 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:54.954209    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:54.958965 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:54.958943    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-6a7dbc-openshift-default-dockercfg-sx9q8\""
Sep 09 13:44:54.969315 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:54.969291    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"]
Sep 09 13:44:54.990133 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:54.990096    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83622cac-d0f5-4aa6-8502-aa9019e9429c-istio-token\") pod \"e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") " pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:54.990294 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:54.990157    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfjts\" (UniqueName: \"kubernetes.io/projected/83622cac-d0f5-4aa6-8502-aa9019e9429c-kube-api-access-dfjts\") pod \"e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") " pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:54.990294 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:54.990274    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-istio-envoy\") pod \"e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") " pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:54.990405 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:54.990331    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83622cac-d0f5-4aa6-8502-aa9019e9429c-istio-podinfo\") pod \"e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") " pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:54.990405 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:54.990389    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-workload-socket\") pod \"e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") " pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:54.990488 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:54.990425    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-workload-certs\") pod \"e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") " pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:54.990488 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:54.990456    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-credential-socket\") pod \"e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") " pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:54.990587 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:54.990557    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83622cac-d0f5-4aa6-8502-aa9019e9429c-istiod-ca-cert\") pod \"e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") " pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:54.990646 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:54.990612    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-istio-data\") pod \"e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") " pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:55.091224 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.091187    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83622cac-d0f5-4aa6-8502-aa9019e9429c-istio-token\") pod \"e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") " pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:55.091451 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.091239    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfjts\" (UniqueName: \"kubernetes.io/projected/83622cac-d0f5-4aa6-8502-aa9019e9429c-kube-api-access-dfjts\") pod \"e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") " pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:55.091451 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.091262    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-istio-envoy\") pod \"e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") " pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:55.091451 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.091297    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83622cac-d0f5-4aa6-8502-aa9019e9429c-istio-podinfo\") pod \"e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") " pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:55.091451 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.091327    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-workload-socket\") pod \"e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") " pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:55.091451 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.091350    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-workload-certs\") pod \"e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") " pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:55.091451 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.091377    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-credential-socket\") pod \"e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") " pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:55.091758 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.091521    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83622cac-d0f5-4aa6-8502-aa9019e9429c-istiod-ca-cert\") pod \"e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") " pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:55.091758 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.091573    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-istio-data\") pod \"e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") " pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:55.091875 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.091788    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-workload-socket\") pod \"e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") " pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:55.091960 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.091896    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-workload-certs\") pod \"e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") " pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:55.092019 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.092006    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-credential-socket\") pod \"e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") " pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:55.092077 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.092062    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-istio-data\") pod \"e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") " pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:55.092335 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.092308    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83622cac-d0f5-4aa6-8502-aa9019e9429c-istiod-ca-cert\") pod \"e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") " pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:55.094072 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.094042    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-istio-envoy\") pod \"e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") " pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:55.094273 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.094258    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83622cac-d0f5-4aa6-8502-aa9019e9429c-istio-podinfo\") pod \"e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") " pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:55.098877 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.098856    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83622cac-d0f5-4aa6-8502-aa9019e9429c-istio-token\") pod \"e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") " pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:55.099063 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.099041    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfjts\" (UniqueName: \"kubernetes.io/projected/83622cac-d0f5-4aa6-8502-aa9019e9429c-kube-api-access-dfjts\") pod \"e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") " pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:55.265372 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.265290    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:55.548713 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.547856    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"]
Sep 09 13:44:55.550493 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:44:55.550464    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83622cac_d0f5_4aa6_8502_aa9019e9429c.slice/crio-df37145c651467d6c29d7921581094d58201087baf794ed348fb94f78e82f30d WatchSource:0}: Error finding container df37145c651467d6c29d7921581094d58201087baf794ed348fb94f78e82f30d: Status 404 returned error can't find the container with id df37145c651467d6c29d7921581094d58201087baf794ed348fb94f78e82f30d
Sep 09 13:44:55.553456 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.553415    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:44:55.553540 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.553508    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:44:55.553581 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.553547    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:44:55.577946 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.577918    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 13:44:55.578255 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.577977    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg" podUID="2f6020ae-98a0-4ea4-b95a-32e247ab0c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 13:44:55.645782 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.645748    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm" event={"ID":"83622cac-d0f5-4aa6-8502-aa9019e9429c","Type":"ContainerStarted","Data":"857740802d67ea248bcbc5fba9640eedafaabf0a4fff422575ed410ac985c7f3"}
Sep 09 13:44:55.645965 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.645832    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm" event={"ID":"83622cac-d0f5-4aa6-8502-aa9019e9429c","Type":"ContainerStarted","Data":"df37145c651467d6c29d7921581094d58201087baf794ed348fb94f78e82f30d"}
Sep 09 13:44:55.647703 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.647676    2566 generic.go:356] "Generic (PLEG): container finished" podID="2f6020ae-98a0-4ea4-b95a-32e247ab0c32" containerID="4d4c0f39e8fcbb9360b8ed4c30bf7663ae9c1da3f663692b015968acc2e0dbc1" exitCode=0
Sep 09 13:44:55.647832 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.647729    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg" event={"ID":"2f6020ae-98a0-4ea4-b95a-32e247ab0c32","Type":"ContainerDied","Data":"4d4c0f39e8fcbb9360b8ed4c30bf7663ae9c1da3f663692b015968acc2e0dbc1"}
Sep 09 13:44:55.648187 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.648172    2566 scope.go:117] "RemoveContainer" containerID="4d4c0f39e8fcbb9360b8ed4c30bf7663ae9c1da3f663692b015968acc2e0dbc1"
Sep 09 13:44:55.649412 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.649381    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:44:55.649535 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.649466    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:44:55.649535 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.649518    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:44:55.666143 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:55.666082    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm" podStartSLOduration=1.666063926 podStartE2EDuration="1.666063926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:44:55.66501423 +0000 UTC m=+1129.024864081" watchObservedRunningTime="2026-09-09 13:44:55.666063926 +0000 UTC m=+1129.025913752"
Sep 09 13:44:56.265722 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:56.265682    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:56.270539 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:56.270515    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:56.654614 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:56.654572    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg" event={"ID":"2f6020ae-98a0-4ea4-b95a-32e247ab0c32","Type":"ContainerStarted","Data":"a4d3b13a6352aff9309d7a225166a598ceacc4a364970f94517cc412bc1f6847"}
Sep 09 13:44:56.655084 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:56.654763    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:56.655751 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:56.655735    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:44:57.157801 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:57.157771    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:44:57.158039 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:57.158011    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 09 13:44:57.158105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:57.158079    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg" podUID="2f6020ae-98a0-4ea4-b95a-32e247ab0c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 09 13:44:58.157987 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:58.157947    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 09 13:44:58.158460 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:58.158006    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg" podUID="2f6020ae-98a0-4ea4-b95a-32e247ab0c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 09 13:44:59.158311 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:59.158275    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 09 13:44:59.158704 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:44:59.158336    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg" podUID="2f6020ae-98a0-4ea4-b95a-32e247ab0c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 09 13:45:00.143566 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:00.143531    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a7dbc-29816025-cjgwq"]
Sep 09 13:45:00.147307 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:00.147287    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816025-844hc"]
Sep 09 13:45:00.147422 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:00.147402    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a7dbc-29816025-cjgwq"
Sep 09 13:45:00.150178 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:00.150141    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lh487\""
Sep 09 13:45:00.150178 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:00.150156    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-df48ae-29816025-chk84"]
Sep 09 13:45:00.150458 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:00.150443    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816025-844hc"
Sep 09 13:45:00.153052 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:00.153029    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-df48ae-29816025-chk84"
Sep 09 13:45:00.154759 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:00.154738    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816025-844hc"]
Sep 09 13:45:00.158641 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:00.157916    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a7dbc-29816025-cjgwq"]
Sep 09 13:45:00.158641 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:00.158266    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 09 13:45:00.158641 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:00.158310    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg" podUID="2f6020ae-98a0-4ea4-b95a-32e247ab0c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 09 13:45:00.159444 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:00.159425    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-df48ae-29816025-chk84"]
Sep 09 13:45:00.161290 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:00.161136    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a7dbc-29816025-cjgwq"
Sep 09 13:45:00.170936 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:00.170911    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816025-844hc"
Sep 09 13:45:00.224158 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:00.223985    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-df48ae-29816025-chk84"
Sep 09 13:45:00.458048 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:00.457954    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a7dbc-29816025-cjgwq"]
Sep 09 13:45:00.460312 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:45:00.460278    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0848b7f_78ba_4e0a_89ec_f2d955899f11.slice/crio-426d4721f571c25e68a98760adf7669e31808d2e7925921468f98d8d64e7636f WatchSource:0}: Error finding container 426d4721f571c25e68a98760adf7669e31808d2e7925921468f98d8d64e7636f: Status 404 returned error can't find the container with id 426d4721f571c25e68a98760adf7669e31808d2e7925921468f98d8d64e7636f
Sep 09 13:45:00.485278 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:00.485253    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816025-844hc"]
Sep 09 13:45:00.548163 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:00.548141    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-df48ae-29816025-chk84"]
Sep 09 13:45:00.557022 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:45:00.556984    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f8a9d3_4a2a_4b1c_a4c4_dbd562c488bc.slice/crio-36ef3c36147a15accf2fa6e766bd1b0eb87ba2cb41e4169966f5ba783137e17b WatchSource:0}: Error finding container 36ef3c36147a15accf2fa6e766bd1b0eb87ba2cb41e4169966f5ba783137e17b: Status 404 returned error can't find the container with id 36ef3c36147a15accf2fa6e766bd1b0eb87ba2cb41e4169966f5ba783137e17b
Sep 09 13:45:00.680241 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:00.680202    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816025-844hc" event={"ID":"c2bb2921-5f53-426b-b458-d4f92008e571","Type":"ContainerStarted","Data":"c04f5d65e8666a3a2ec506e0cf74cedf7fa7b37afb96b87761078420488b7c5d"}
Sep 09 13:45:00.681512 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:00.681487    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-df48ae-29816025-chk84" event={"ID":"14f8a9d3-4a2a-4b1c-a4c4-dbd562c488bc","Type":"ContainerStarted","Data":"36ef3c36147a15accf2fa6e766bd1b0eb87ba2cb41e4169966f5ba783137e17b"}
Sep 09 13:45:00.682726 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:00.682704    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a7dbc-29816025-cjgwq" event={"ID":"e0848b7f-78ba-4e0a-89ec-f2d955899f11","Type":"ContainerStarted","Data":"426d4721f571c25e68a98760adf7669e31808d2e7925921468f98d8d64e7636f"}
Sep 09 13:45:01.158462 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:01.158432    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 09 13:45:01.158705 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:01.158682    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg" podUID="2f6020ae-98a0-4ea4-b95a-32e247ab0c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 09 13:45:01.691236 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:01.691144    2566 generic.go:356] "Generic (PLEG): container finished" podID="e0848b7f-78ba-4e0a-89ec-f2d955899f11" containerID="dddee0da9be4c8a352ae2279a7a0db3348162d14079e987e2698cbe8e683902c" exitCode=7
Sep 09 13:45:01.691236 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:01.691221    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a7dbc-29816025-cjgwq" event={"ID":"e0848b7f-78ba-4e0a-89ec-f2d955899f11","Type":"ContainerDied","Data":"dddee0da9be4c8a352ae2279a7a0db3348162d14079e987e2698cbe8e683902c"}
Sep 09 13:45:01.691478 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:01.691380    2566 scope.go:117] "RemoveContainer" containerID="dddee0da9be4c8a352ae2279a7a0db3348162d14079e987e2698cbe8e683902c"
Sep 09 13:45:01.693050 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:01.693024    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816025-844hc" event={"ID":"c2bb2921-5f53-426b-b458-d4f92008e571","Type":"ContainerStarted","Data":"f4d2528dfdf0c3de23a55cd7adb6c2dea9a2ee023a00daccea0ac8b505f82991"}
Sep 09 13:45:01.694751 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:01.694723    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-df48ae-29816025-chk84" event={"ID":"14f8a9d3-4a2a-4b1c-a4c4-dbd562c488bc","Type":"ContainerStarted","Data":"9ee976955e67a7404a963d087339f920388b5e4c7ae770eabefeda5b78b44965"}
Sep 09 13:45:01.749121 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:01.749068    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816025-844hc" podStartSLOduration=1.749051769 podStartE2EDuration="1.749051769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:45:01.735262903 +0000 UTC m=+1135.095112730" watchObservedRunningTime="2026-09-09 13:45:01.749051769 +0000 UTC m=+1135.108901595"
Sep 09 13:45:01.749906 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:01.749879    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-df48ae-29816025-chk84" podStartSLOduration=1.749868768 podStartE2EDuration="1.749868768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:45:01.748781573 +0000 UTC m=+1135.108631416" watchObservedRunningTime="2026-09-09 13:45:01.749868768 +0000 UTC m=+1135.109718594"
Sep 09 13:45:02.161619 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:02.161588    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:45:02.161998 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:02.161918    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:45:02.162609 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:02.162591    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:45:02.703466 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:02.703433    2566 generic.go:356] "Generic (PLEG): container finished" podID="e0848b7f-78ba-4e0a-89ec-f2d955899f11" containerID="d3cc6e96bac68bf33553cb9a65c69102d1de6e09dc073ec48606a1364ff4e066" exitCode=7
Sep 09 13:45:02.703660 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:02.703518    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a7dbc-29816025-cjgwq" event={"ID":"e0848b7f-78ba-4e0a-89ec-f2d955899f11","Type":"ContainerDied","Data":"d3cc6e96bac68bf33553cb9a65c69102d1de6e09dc073ec48606a1364ff4e066"}
Sep 09 13:45:02.703660 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:02.703594    2566 scope.go:117] "RemoveContainer" containerID="dddee0da9be4c8a352ae2279a7a0db3348162d14079e987e2698cbe8e683902c"
Sep 09 13:45:02.703776 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:02.703674    2566 scope.go:117] "RemoveContainer" containerID="d3cc6e96bac68bf33553cb9a65c69102d1de6e09dc073ec48606a1364ff4e066"
Sep 09 13:45:02.703970 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:45:02.703949    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6a7dbc-29816025-cjgwq_odh-ai-gateway-infra(e0848b7f-78ba-4e0a-89ec-f2d955899f11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a7dbc-29816025-cjgwq" podUID="e0848b7f-78ba-4e0a-89ec-f2d955899f11"
Sep 09 13:45:02.705138 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:02.705116    2566 generic.go:356] "Generic (PLEG): container finished" podID="c2bb2921-5f53-426b-b458-d4f92008e571" containerID="f4d2528dfdf0c3de23a55cd7adb6c2dea9a2ee023a00daccea0ac8b505f82991" exitCode=0
Sep 09 13:45:02.705245 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:02.705165    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816025-844hc" event={"ID":"c2bb2921-5f53-426b-b458-d4f92008e571","Type":"ContainerDied","Data":"f4d2528dfdf0c3de23a55cd7adb6c2dea9a2ee023a00daccea0ac8b505f82991"}
Sep 09 13:45:02.706744 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:02.706724    2566 generic.go:356] "Generic (PLEG): container finished" podID="14f8a9d3-4a2a-4b1c-a4c4-dbd562c488bc" containerID="9ee976955e67a7404a963d087339f920388b5e4c7ae770eabefeda5b78b44965" exitCode=0
Sep 09 13:45:02.706847 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:02.706799    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-df48ae-29816025-chk84" event={"ID":"14f8a9d3-4a2a-4b1c-a4c4-dbd562c488bc","Type":"ContainerDied","Data":"9ee976955e67a7404a963d087339f920388b5e4c7ae770eabefeda5b78b44965"}
Sep 09 13:45:03.715213 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:03.715186    2566 scope.go:117] "RemoveContainer" containerID="d3cc6e96bac68bf33553cb9a65c69102d1de6e09dc073ec48606a1364ff4e066"
Sep 09 13:45:03.715634 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:45:03.715436    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6a7dbc-29816025-cjgwq_odh-ai-gateway-infra(e0848b7f-78ba-4e0a-89ec-f2d955899f11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a7dbc-29816025-cjgwq" podUID="e0848b7f-78ba-4e0a-89ec-f2d955899f11"
Sep 09 13:45:03.854996 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:03.854966    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-df48ae-29816025-chk84"
Sep 09 13:45:03.891950 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:03.891918    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816025-844hc"
Sep 09 13:45:04.722261 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:04.722235    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816025-844hc"
Sep 09 13:45:04.722261 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:04.722239    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816025-844hc" event={"ID":"c2bb2921-5f53-426b-b458-d4f92008e571","Type":"ContainerDied","Data":"c04f5d65e8666a3a2ec506e0cf74cedf7fa7b37afb96b87761078420488b7c5d"}
Sep 09 13:45:04.722707 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:04.722288    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c04f5d65e8666a3a2ec506e0cf74cedf7fa7b37afb96b87761078420488b7c5d"
Sep 09 13:45:04.723641 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:04.723625    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-df48ae-29816025-chk84"
Sep 09 13:45:04.723765 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:04.723677    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-df48ae-29816025-chk84" event={"ID":"14f8a9d3-4a2a-4b1c-a4c4-dbd562c488bc","Type":"ContainerDied","Data":"36ef3c36147a15accf2fa6e766bd1b0eb87ba2cb41e4169966f5ba783137e17b"}
Sep 09 13:45:04.723765 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:04.723724    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36ef3c36147a15accf2fa6e766bd1b0eb87ba2cb41e4169966f5ba783137e17b"
Sep 09 13:45:10.763340 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.763300    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"]
Sep 09 13:45:10.763805 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.763659    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14f8a9d3-4a2a-4b1c-a4c4-dbd562c488bc" containerName="cleanup"
Sep 09 13:45:10.763805 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.763670    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="14f8a9d3-4a2a-4b1c-a4c4-dbd562c488bc" containerName="cleanup"
Sep 09 13:45:10.763805 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.763684    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2bb2921-5f53-426b-b458-d4f92008e571" containerName="cleanup"
Sep 09 13:45:10.763805 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.763689    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2bb2921-5f53-426b-b458-d4f92008e571" containerName="cleanup"
Sep 09 13:45:10.763805 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.763750    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2bb2921-5f53-426b-b458-d4f92008e571" containerName="cleanup"
Sep 09 13:45:10.763805 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.763757    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="14f8a9d3-4a2a-4b1c-a4c4-dbd562c488bc" containerName="cleanup"
Sep 09 13:45:10.768470 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.768448    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"
Sep 09 13:45:10.771267 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.771239    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8f2df2\"/\"default-dockercfg-2fx29\""
Sep 09 13:45:10.771448 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.771309    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8f2df2\"/\"auth-test-model-8f2df2-kserve-self-signed-certs\""
Sep 09 13:45:10.771531 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.771310    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-8f2df2\"/\"kube-root-ca.crt\""
Sep 09 13:45:10.771593 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.771328    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-8f2df2\"/\"openshift-service-ca.crt\""
Sep 09 13:45:10.779250 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.779224    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"]
Sep 09 13:45:10.856739 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.856700    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8hfv\" (UniqueName: \"kubernetes.io/projected/f70c6d78-ea87-4176-b94f-851ae5270e0f-kube-api-access-n8hfv\") pod \"auth-test-model-8f2df2-kserve-7d58585547-6n7lf\" (UID: \"f70c6d78-ea87-4176-b94f-851ae5270e0f\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"
Sep 09 13:45:10.856739 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.856744    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f70c6d78-ea87-4176-b94f-851ae5270e0f-model-cache\") pod \"auth-test-model-8f2df2-kserve-7d58585547-6n7lf\" (UID: \"f70c6d78-ea87-4176-b94f-851ae5270e0f\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"
Sep 09 13:45:10.856984 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.856877    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f70c6d78-ea87-4176-b94f-851ae5270e0f-home\") pod \"auth-test-model-8f2df2-kserve-7d58585547-6n7lf\" (UID: \"f70c6d78-ea87-4176-b94f-851ae5270e0f\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"
Sep 09 13:45:10.856984 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.856921    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f70c6d78-ea87-4176-b94f-851ae5270e0f-tmp-dir\") pod \"auth-test-model-8f2df2-kserve-7d58585547-6n7lf\" (UID: \"f70c6d78-ea87-4176-b94f-851ae5270e0f\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"
Sep 09 13:45:10.857056 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.856985    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f70c6d78-ea87-4176-b94f-851ae5270e0f-dshm\") pod \"auth-test-model-8f2df2-kserve-7d58585547-6n7lf\" (UID: \"f70c6d78-ea87-4176-b94f-851ae5270e0f\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"
Sep 09 13:45:10.857056 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.857023    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f70c6d78-ea87-4176-b94f-851ae5270e0f-tls-certs\") pod \"auth-test-model-8f2df2-kserve-7d58585547-6n7lf\" (UID: \"f70c6d78-ea87-4176-b94f-851ae5270e0f\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"
Sep 09 13:45:10.958041 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.958000    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8hfv\" (UniqueName: \"kubernetes.io/projected/f70c6d78-ea87-4176-b94f-851ae5270e0f-kube-api-access-n8hfv\") pod \"auth-test-model-8f2df2-kserve-7d58585547-6n7lf\" (UID: \"f70c6d78-ea87-4176-b94f-851ae5270e0f\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"
Sep 09 13:45:10.958041 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.958047    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f70c6d78-ea87-4176-b94f-851ae5270e0f-model-cache\") pod \"auth-test-model-8f2df2-kserve-7d58585547-6n7lf\" (UID: \"f70c6d78-ea87-4176-b94f-851ae5270e0f\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"
Sep 09 13:45:10.958308 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.958112    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f70c6d78-ea87-4176-b94f-851ae5270e0f-home\") pod \"auth-test-model-8f2df2-kserve-7d58585547-6n7lf\" (UID: \"f70c6d78-ea87-4176-b94f-851ae5270e0f\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"
Sep 09 13:45:10.958308 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.958137    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f70c6d78-ea87-4176-b94f-851ae5270e0f-tmp-dir\") pod \"auth-test-model-8f2df2-kserve-7d58585547-6n7lf\" (UID: \"f70c6d78-ea87-4176-b94f-851ae5270e0f\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"
Sep 09 13:45:10.958308 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.958180    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f70c6d78-ea87-4176-b94f-851ae5270e0f-dshm\") pod \"auth-test-model-8f2df2-kserve-7d58585547-6n7lf\" (UID: \"f70c6d78-ea87-4176-b94f-851ae5270e0f\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"
Sep 09 13:45:10.958308 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.958232    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f70c6d78-ea87-4176-b94f-851ae5270e0f-tls-certs\") pod \"auth-test-model-8f2df2-kserve-7d58585547-6n7lf\" (UID: \"f70c6d78-ea87-4176-b94f-851ae5270e0f\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"
Sep 09 13:45:10.958517 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.958498    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f70c6d78-ea87-4176-b94f-851ae5270e0f-home\") pod \"auth-test-model-8f2df2-kserve-7d58585547-6n7lf\" (UID: \"f70c6d78-ea87-4176-b94f-851ae5270e0f\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"
Sep 09 13:45:10.958587 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.958564    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f70c6d78-ea87-4176-b94f-851ae5270e0f-tmp-dir\") pod \"auth-test-model-8f2df2-kserve-7d58585547-6n7lf\" (UID: \"f70c6d78-ea87-4176-b94f-851ae5270e0f\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"
Sep 09 13:45:10.958801 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.958775    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f70c6d78-ea87-4176-b94f-851ae5270e0f-model-cache\") pod \"auth-test-model-8f2df2-kserve-7d58585547-6n7lf\" (UID: \"f70c6d78-ea87-4176-b94f-851ae5270e0f\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"
Sep 09 13:45:10.960530 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.960504    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f70c6d78-ea87-4176-b94f-851ae5270e0f-dshm\") pod \"auth-test-model-8f2df2-kserve-7d58585547-6n7lf\" (UID: \"f70c6d78-ea87-4176-b94f-851ae5270e0f\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"
Sep 09 13:45:10.960824 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.960789    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f70c6d78-ea87-4176-b94f-851ae5270e0f-tls-certs\") pod \"auth-test-model-8f2df2-kserve-7d58585547-6n7lf\" (UID: \"f70c6d78-ea87-4176-b94f-851ae5270e0f\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"
Sep 09 13:45:10.966584 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:10.966565    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8hfv\" (UniqueName: \"kubernetes.io/projected/f70c6d78-ea87-4176-b94f-851ae5270e0f-kube-api-access-n8hfv\") pod \"auth-test-model-8f2df2-kserve-7d58585547-6n7lf\" (UID: \"f70c6d78-ea87-4176-b94f-851ae5270e0f\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"
Sep 09 13:45:11.080446 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:11.080413    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"
Sep 09 13:45:11.376441 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:11.376403    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"]
Sep 09 13:45:11.378274 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:45:11.378231    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf70c6d78_ea87_4176_b94f_851ae5270e0f.slice/crio-731ddca9694e092aaa756fb2a4d202ac53d7676f9675905a6e4169150330bda9 WatchSource:0}: Error finding container 731ddca9694e092aaa756fb2a4d202ac53d7676f9675905a6e4169150330bda9: Status 404 returned error can't find the container with id 731ddca9694e092aaa756fb2a4d202ac53d7676f9675905a6e4169150330bda9
Sep 09 13:45:11.771299 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:11.771193    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf" event={"ID":"f70c6d78-ea87-4176-b94f-851ae5270e0f","Type":"ContainerStarted","Data":"2657826ba4f9314a62d2820c0c79d12b88d77b88e72f358eb33ffd0bd608eeeb"}
Sep 09 13:45:11.771299 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:11.771253    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf" event={"ID":"f70c6d78-ea87-4176-b94f-851ae5270e0f","Type":"ContainerStarted","Data":"731ddca9694e092aaa756fb2a4d202ac53d7676f9675905a6e4169150330bda9"}
Sep 09 13:45:11.792905 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:11.792854    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf" podStartSLOduration=1.79280617 podStartE2EDuration="1.79280617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:45:11.790978801 +0000 UTC m=+1145.150828628" watchObservedRunningTime="2026-09-09 13:45:11.79280617 +0000 UTC m=+1145.152655997"
Sep 09 13:45:12.697040 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:12.696983    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 13:45:12.697261 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:12.697051    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm" podUID="83622cac-d0f5-4aa6-8502-aa9019e9429c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 13:45:14.790385 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:14.790350    2566 generic.go:356] "Generic (PLEG): container finished" podID="83622cac-d0f5-4aa6-8502-aa9019e9429c" containerID="857740802d67ea248bcbc5fba9640eedafaabf0a4fff422575ed410ac985c7f3" exitCode=0
Sep 09 13:45:14.790824 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:14.790426    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm" event={"ID":"83622cac-d0f5-4aa6-8502-aa9019e9429c","Type":"ContainerDied","Data":"857740802d67ea248bcbc5fba9640eedafaabf0a4fff422575ed410ac985c7f3"}
Sep 09 13:45:14.790895 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:14.790881    2566 scope.go:117] "RemoveContainer" containerID="857740802d67ea248bcbc5fba9640eedafaabf0a4fff422575ed410ac985c7f3"
Sep 09 13:45:14.791991 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:14.791955    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:45:14.792091 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:14.792037    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:45:14.792091 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:14.792081    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:45:15.802171 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:15.802132    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm" event={"ID":"83622cac-d0f5-4aa6-8502-aa9019e9429c","Type":"ContainerStarted","Data":"c1f6769dca07b6bc8c67425f8f52f7e566097ef200b507b5360f68ea7ecd13e7"}
Sep 09 13:45:16.144172 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:16.144144    2566 scope.go:117] "RemoveContainer" containerID="d3cc6e96bac68bf33553cb9a65c69102d1de6e09dc073ec48606a1364ff4e066"
Sep 09 13:45:16.266030 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:16.265992    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:45:16.266359 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:16.266336    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Sep 09 13:45:16.266410 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:16.266385    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm" podUID="83622cac-d0f5-4aa6-8502-aa9019e9429c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Sep 09 13:45:16.814043 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:16.813368    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a7dbc-29816025-cjgwq" event={"ID":"e0848b7f-78ba-4e0a-89ec-f2d955899f11","Type":"ContainerStarted","Data":"59e6b58abdbbb070d08819c587b64fc687488b82eb160441f8a6ec78963341bf"}
Sep 09 13:45:16.832663 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:16.832613    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a7dbc-29816025-cjgwq" podStartSLOduration=16.83259853 podStartE2EDuration="16.83259853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:45:16.831147918 +0000 UTC m=+1150.190997744" watchObservedRunningTime="2026-09-09 13:45:16.83259853 +0000 UTC m=+1150.192448355"
Sep 09 13:45:17.266236 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:17.266186    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Sep 09 13:45:17.266447 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:17.266264    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm" podUID="83622cac-d0f5-4aa6-8502-aa9019e9429c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Sep 09 13:45:17.820192 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:17.820155    2566 generic.go:356] "Generic (PLEG): container finished" podID="e0848b7f-78ba-4e0a-89ec-f2d955899f11" containerID="59e6b58abdbbb070d08819c587b64fc687488b82eb160441f8a6ec78963341bf" exitCode=0
Sep 09 13:45:17.820192 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:17.820195    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a7dbc-29816025-cjgwq" event={"ID":"e0848b7f-78ba-4e0a-89ec-f2d955899f11","Type":"ContainerDied","Data":"59e6b58abdbbb070d08819c587b64fc687488b82eb160441f8a6ec78963341bf"}
Sep 09 13:45:17.820688 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:17.820241    2566 scope.go:117] "RemoveContainer" containerID="d3cc6e96bac68bf33553cb9a65c69102d1de6e09dc073ec48606a1364ff4e066"
Sep 09 13:45:17.848264 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:17.848235    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a7dbc-29816025-cjgwq"]
Sep 09 13:45:18.266495 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:18.266460    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Sep 09 13:45:18.266686 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:18.266519    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm" podUID="83622cac-d0f5-4aa6-8502-aa9019e9429c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Sep 09 13:45:18.970281 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:18.970257    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a7dbc-29816025-cjgwq"
Sep 09 13:45:19.265700 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:19.265660    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Sep 09 13:45:19.265906 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:19.265719    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm" podUID="83622cac-d0f5-4aa6-8502-aa9019e9429c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Sep 09 13:45:19.833683 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:19.833656    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a7dbc-29816025-cjgwq"
Sep 09 13:45:19.833892 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:19.833698    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a7dbc-29816025-cjgwq" event={"ID":"e0848b7f-78ba-4e0a-89ec-f2d955899f11","Type":"ContainerDied","Data":"426d4721f571c25e68a98760adf7669e31808d2e7925921468f98d8d64e7636f"}
Sep 09 13:45:19.833892 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:19.833745    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="426d4721f571c25e68a98760adf7669e31808d2e7925921468f98d8d64e7636f"
Sep 09 13:45:19.863104 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:19.863066    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a7dbc-29816025-cjgwq"]
Sep 09 13:45:19.872439 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:19.872411    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a7dbc-29816025-cjgwq"]
Sep 09 13:45:20.266240 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:20.266134    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Sep 09 13:45:20.266240 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:20.266196    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm" podUID="83622cac-d0f5-4aa6-8502-aa9019e9429c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Sep 09 13:45:21.080893 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:21.080843    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"
Sep 09 13:45:21.081087 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:21.080923    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"
Sep 09 13:45:21.094375 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:21.094338    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"
Sep 09 13:45:21.149772 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:21.149739    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0848b7f-78ba-4e0a-89ec-f2d955899f11" path="/var/lib/kubelet/pods/e0848b7f-78ba-4e0a-89ec-f2d955899f11/volumes"
Sep 09 13:45:21.269297 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:21.269271    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:45:21.269763 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:21.269544    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:45:21.270298 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:21.270282    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:45:21.857500 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:21.857465    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"
Sep 09 13:45:30.860702 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:30.860615    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"]
Sep 09 13:45:30.861072 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:30.861057    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0848b7f-78ba-4e0a-89ec-f2d955899f11" containerName="cleanup"
Sep 09 13:45:30.861072 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:30.861070    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0848b7f-78ba-4e0a-89ec-f2d955899f11" containerName="cleanup"
Sep 09 13:45:30.861158 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:30.861078    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0848b7f-78ba-4e0a-89ec-f2d955899f11" containerName="cleanup"
Sep 09 13:45:30.861158 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:30.861084    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0848b7f-78ba-4e0a-89ec-f2d955899f11" containerName="cleanup"
Sep 09 13:45:30.861224 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:30.861160    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0848b7f-78ba-4e0a-89ec-f2d955899f11" containerName="cleanup"
Sep 09 13:45:30.861224 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:30.861167    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0848b7f-78ba-4e0a-89ec-f2d955899f11" containerName="cleanup"
Sep 09 13:45:30.865450 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:30.865430    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"
Sep 09 13:45:30.867989 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:30.867967    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-df48ae\"/\"default-dockercfg-cf6vl\""
Sep 09 13:45:30.867989 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:30.867978    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-df48ae\"/\"kube-root-ca.crt\""
Sep 09 13:45:30.868760 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:30.868733    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-df48ae\"/\"openshift-service-ca.crt\""
Sep 09 13:45:30.868906 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:30.868887    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-df48ae\"/\"auth-test-model-df48ae-kserve-self-signed-certs\""
Sep 09 13:45:30.873003 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:30.872981    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"]
Sep 09 13:45:30.964364 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:30.964330    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7deaa929-051a-43dc-8ea5-572dc2b23533-tls-certs\") pod \"auth-test-model-df48ae-kserve-db88d98f5-w65q8\" (UID: \"7deaa929-051a-43dc-8ea5-572dc2b23533\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"
Sep 09 13:45:30.964552 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:30.964390    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7deaa929-051a-43dc-8ea5-572dc2b23533-dshm\") pod \"auth-test-model-df48ae-kserve-db88d98f5-w65q8\" (UID: \"7deaa929-051a-43dc-8ea5-572dc2b23533\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"
Sep 09 13:45:30.964552 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:30.964464    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7deaa929-051a-43dc-8ea5-572dc2b23533-tmp-dir\") pod \"auth-test-model-df48ae-kserve-db88d98f5-w65q8\" (UID: \"7deaa929-051a-43dc-8ea5-572dc2b23533\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"
Sep 09 13:45:30.964552 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:30.964513    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7deaa929-051a-43dc-8ea5-572dc2b23533-home\") pod \"auth-test-model-df48ae-kserve-db88d98f5-w65q8\" (UID: \"7deaa929-051a-43dc-8ea5-572dc2b23533\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"
Sep 09 13:45:30.964552 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:30.964537    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7deaa929-051a-43dc-8ea5-572dc2b23533-model-cache\") pod \"auth-test-model-df48ae-kserve-db88d98f5-w65q8\" (UID: \"7deaa929-051a-43dc-8ea5-572dc2b23533\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"
Sep 09 13:45:30.964697 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:30.964610    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x78xl\" (UniqueName: \"kubernetes.io/projected/7deaa929-051a-43dc-8ea5-572dc2b23533-kube-api-access-x78xl\") pod \"auth-test-model-df48ae-kserve-db88d98f5-w65q8\" (UID: \"7deaa929-051a-43dc-8ea5-572dc2b23533\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"
Sep 09 13:45:31.065414 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:31.065374    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x78xl\" (UniqueName: \"kubernetes.io/projected/7deaa929-051a-43dc-8ea5-572dc2b23533-kube-api-access-x78xl\") pod \"auth-test-model-df48ae-kserve-db88d98f5-w65q8\" (UID: \"7deaa929-051a-43dc-8ea5-572dc2b23533\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"
Sep 09 13:45:31.065414 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:31.065424    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7deaa929-051a-43dc-8ea5-572dc2b23533-tls-certs\") pod \"auth-test-model-df48ae-kserve-db88d98f5-w65q8\" (UID: \"7deaa929-051a-43dc-8ea5-572dc2b23533\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"
Sep 09 13:45:31.065661 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:31.065473    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7deaa929-051a-43dc-8ea5-572dc2b23533-dshm\") pod \"auth-test-model-df48ae-kserve-db88d98f5-w65q8\" (UID: \"7deaa929-051a-43dc-8ea5-572dc2b23533\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"
Sep 09 13:45:31.065661 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:31.065501    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7deaa929-051a-43dc-8ea5-572dc2b23533-tmp-dir\") pod \"auth-test-model-df48ae-kserve-db88d98f5-w65q8\" (UID: \"7deaa929-051a-43dc-8ea5-572dc2b23533\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"
Sep 09 13:45:31.065661 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:31.065645    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7deaa929-051a-43dc-8ea5-572dc2b23533-home\") pod \"auth-test-model-df48ae-kserve-db88d98f5-w65q8\" (UID: \"7deaa929-051a-43dc-8ea5-572dc2b23533\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"
Sep 09 13:45:31.065846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:31.065709    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7deaa929-051a-43dc-8ea5-572dc2b23533-model-cache\") pod \"auth-test-model-df48ae-kserve-db88d98f5-w65q8\" (UID: \"7deaa929-051a-43dc-8ea5-572dc2b23533\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"
Sep 09 13:45:31.065993 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:31.065976    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7deaa929-051a-43dc-8ea5-572dc2b23533-tmp-dir\") pod \"auth-test-model-df48ae-kserve-db88d98f5-w65q8\" (UID: \"7deaa929-051a-43dc-8ea5-572dc2b23533\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"
Sep 09 13:45:31.066057 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:31.066034    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7deaa929-051a-43dc-8ea5-572dc2b23533-model-cache\") pod \"auth-test-model-df48ae-kserve-db88d98f5-w65q8\" (UID: \"7deaa929-051a-43dc-8ea5-572dc2b23533\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"
Sep 09 13:45:31.066111 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:31.066035    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7deaa929-051a-43dc-8ea5-572dc2b23533-home\") pod \"auth-test-model-df48ae-kserve-db88d98f5-w65q8\" (UID: \"7deaa929-051a-43dc-8ea5-572dc2b23533\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"
Sep 09 13:45:31.067944 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:31.067920    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7deaa929-051a-43dc-8ea5-572dc2b23533-dshm\") pod \"auth-test-model-df48ae-kserve-db88d98f5-w65q8\" (UID: \"7deaa929-051a-43dc-8ea5-572dc2b23533\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"
Sep 09 13:45:31.068063 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:31.068030    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7deaa929-051a-43dc-8ea5-572dc2b23533-tls-certs\") pod \"auth-test-model-df48ae-kserve-db88d98f5-w65q8\" (UID: \"7deaa929-051a-43dc-8ea5-572dc2b23533\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"
Sep 09 13:45:31.077123 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:31.077099    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x78xl\" (UniqueName: \"kubernetes.io/projected/7deaa929-051a-43dc-8ea5-572dc2b23533-kube-api-access-x78xl\") pod \"auth-test-model-df48ae-kserve-db88d98f5-w65q8\" (UID: \"7deaa929-051a-43dc-8ea5-572dc2b23533\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"
Sep 09 13:45:31.176244 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:31.176143    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"
Sep 09 13:45:31.467503 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:31.467472    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"]
Sep 09 13:45:31.469584 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:45:31.469554    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7deaa929_051a_43dc_8ea5_572dc2b23533.slice/crio-047f8c6339eb8de8df1c9fa8ca1753e89638f10b9b4b1add8ed8946a1e3158ec WatchSource:0}: Error finding container 047f8c6339eb8de8df1c9fa8ca1753e89638f10b9b4b1add8ed8946a1e3158ec: Status 404 returned error can't find the container with id 047f8c6339eb8de8df1c9fa8ca1753e89638f10b9b4b1add8ed8946a1e3158ec
Sep 09 13:45:31.909564 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:31.909527    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8" event={"ID":"7deaa929-051a-43dc-8ea5-572dc2b23533","Type":"ContainerStarted","Data":"be9dbf7104b5f17fa3ded791ff806a33c675aa3932cc457ee7ab758d1d3d0d48"}
Sep 09 13:45:31.909564 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:31.909570    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8" event={"ID":"7deaa929-051a-43dc-8ea5-572dc2b23533","Type":"ContainerStarted","Data":"047f8c6339eb8de8df1c9fa8ca1753e89638f10b9b4b1add8ed8946a1e3158ec"}
Sep 09 13:45:31.928117 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:31.928062    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8" podStartSLOduration=1.9280449530000001 podStartE2EDuration="1.928044953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:45:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:45:31.927664855 +0000 UTC m=+1165.287514682" watchObservedRunningTime="2026-09-09 13:45:31.928044953 +0000 UTC m=+1165.287894775"
Sep 09 13:45:36.589774 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:36.589730    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f"]
Sep 09 13:45:36.590300 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:36.590181    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0848b7f-78ba-4e0a-89ec-f2d955899f11" containerName="cleanup"
Sep 09 13:45:36.590300 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:36.590194    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0848b7f-78ba-4e0a-89ec-f2d955899f11" containerName="cleanup"
Sep 09 13:45:36.590300 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:36.590262    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0848b7f-78ba-4e0a-89ec-f2d955899f11" containerName="cleanup"
Sep 09 13:45:36.593275 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:36.593260    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f"
Sep 09 13:45:36.595748 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:36.595720    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-638c40f1\""
Sep 09 13:45:36.595913 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:36.595751    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-638c40f1-dockercfg-sn246\""
Sep 09 13:45:36.605542 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:36.605514    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f"]
Sep 09 13:45:36.617038 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:36.617007    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4eb5664-773e-4eb4-b890-feb940b1aad3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f\" (UID: \"d4eb5664-773e-4eb4-b890-feb940b1aad3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f"
Sep 09 13:45:36.617179 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:36.617065    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqgjl\" (UniqueName: \"kubernetes.io/projected/d4eb5664-773e-4eb4-b890-feb940b1aad3-kube-api-access-bqgjl\") pod \"payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f\" (UID: \"d4eb5664-773e-4eb4-b890-feb940b1aad3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f"
Sep 09 13:45:36.620193 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:36.620170    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp"]
Sep 09 13:45:36.623676 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:36.623656    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp"
Sep 09 13:45:36.638534 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:36.638510    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp"]
Sep 09 13:45:36.718534 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:36.718492    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzqd4\" (UniqueName: \"kubernetes.io/projected/bb9d66fc-74f4-4a3a-8a82-a605b939a7f0-kube-api-access-xzqd4\") pod \"payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp\" (UID: \"bb9d66fc-74f4-4a3a-8a82-a605b939a7f0\") " pod="openshift-ingress/payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp"
Sep 09 13:45:36.718731 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:36.718609    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb9d66fc-74f4-4a3a-8a82-a605b939a7f0-plugins-config-volume\") pod \"payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp\" (UID: \"bb9d66fc-74f4-4a3a-8a82-a605b939a7f0\") " pod="openshift-ingress/payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp"
Sep 09 13:45:36.718731 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:36.718669    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4eb5664-773e-4eb4-b890-feb940b1aad3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f\" (UID: \"d4eb5664-773e-4eb4-b890-feb940b1aad3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f"
Sep 09 13:45:36.718867 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:36.718772    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqgjl\" (UniqueName: \"kubernetes.io/projected/d4eb5664-773e-4eb4-b890-feb940b1aad3-kube-api-access-bqgjl\") pod \"payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f\" (UID: \"d4eb5664-773e-4eb4-b890-feb940b1aad3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f"
Sep 09 13:45:36.720366 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:36.720340    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4eb5664-773e-4eb4-b890-feb940b1aad3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f\" (UID: \"d4eb5664-773e-4eb4-b890-feb940b1aad3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f"
Sep 09 13:45:36.727349 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:36.727325    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqgjl\" (UniqueName: \"kubernetes.io/projected/d4eb5664-773e-4eb4-b890-feb940b1aad3-kube-api-access-bqgjl\") pod \"payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f\" (UID: \"d4eb5664-773e-4eb4-b890-feb940b1aad3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f"
Sep 09 13:45:36.820033 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:36.819991    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzqd4\" (UniqueName: \"kubernetes.io/projected/bb9d66fc-74f4-4a3a-8a82-a605b939a7f0-kube-api-access-xzqd4\") pod \"payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp\" (UID: \"bb9d66fc-74f4-4a3a-8a82-a605b939a7f0\") " pod="openshift-ingress/payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp"
Sep 09 13:45:36.820216 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:36.820050    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb9d66fc-74f4-4a3a-8a82-a605b939a7f0-plugins-config-volume\") pod \"payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp\" (UID: \"bb9d66fc-74f4-4a3a-8a82-a605b939a7f0\") " pod="openshift-ingress/payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp"
Sep 09 13:45:36.820851 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:36.820801    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb9d66fc-74f4-4a3a-8a82-a605b939a7f0-plugins-config-volume\") pod \"payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp\" (UID: \"bb9d66fc-74f4-4a3a-8a82-a605b939a7f0\") " pod="openshift-ingress/payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp"
Sep 09 13:45:36.828677 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:36.828650    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzqd4\" (UniqueName: \"kubernetes.io/projected/bb9d66fc-74f4-4a3a-8a82-a605b939a7f0-kube-api-access-xzqd4\") pod \"payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp\" (UID: \"bb9d66fc-74f4-4a3a-8a82-a605b939a7f0\") " pod="openshift-ingress/payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp"
Sep 09 13:45:36.904258 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:36.904171    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f"
Sep 09 13:45:36.933340 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:36.933308    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp"
Sep 09 13:45:37.181478 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:37.181456    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f"]
Sep 09 13:45:37.183453 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:45:37.183431    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4eb5664_773e_4eb4_b890_feb940b1aad3.slice/crio-75e0e781eb85d33d9546df981bdb3bb688866986ec13c9ee22544975bf0e9d70 WatchSource:0}: Error finding container 75e0e781eb85d33d9546df981bdb3bb688866986ec13c9ee22544975bf0e9d70: Status 404 returned error can't find the container with id 75e0e781eb85d33d9546df981bdb3bb688866986ec13c9ee22544975bf0e9d70
Sep 09 13:45:37.216548 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:37.216524    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp"]
Sep 09 13:45:37.220962 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:45:37.220940    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb9d66fc_74f4_4a3a_8a82_a605b939a7f0.slice/crio-572886626c16cce62f8bd142eab6486637c22d5b4bf4c1f54d9640010e8c261f WatchSource:0}: Error finding container 572886626c16cce62f8bd142eab6486637c22d5b4bf4c1f54d9640010e8c261f: Status 404 returned error can't find the container with id 572886626c16cce62f8bd142eab6486637c22d5b4bf4c1f54d9640010e8c261f
Sep 09 13:45:37.944927 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:37.944888    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f" event={"ID":"d4eb5664-773e-4eb4-b890-feb940b1aad3","Type":"ContainerStarted","Data":"b7cab5bde19485fea841a478923252b2dd72d182f560ca83d83b5cdfac3a1f1e"}
Sep 09 13:45:37.945444 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:37.944957    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f" event={"ID":"d4eb5664-773e-4eb4-b890-feb940b1aad3","Type":"ContainerStarted","Data":"75e0e781eb85d33d9546df981bdb3bb688866986ec13c9ee22544975bf0e9d70"}
Sep 09 13:45:37.945777 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:37.945754    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f"
Sep 09 13:45:37.948066 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:37.947993    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp" event={"ID":"bb9d66fc-74f4-4a3a-8a82-a605b939a7f0","Type":"ContainerStarted","Data":"8ad690292e3bdc3859d13a2f053b964aaf1308501bfd0d588ea428c3a05dffc6"}
Sep 09 13:45:37.948066 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:37.948045    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp" event={"ID":"bb9d66fc-74f4-4a3a-8a82-a605b939a7f0","Type":"ContainerStarted","Data":"572886626c16cce62f8bd142eab6486637c22d5b4bf4c1f54d9640010e8c261f"}
Sep 09 13:45:37.948609 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:37.948588    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp"
Sep 09 13:45:37.961456 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:37.961376    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f" podStartSLOduration=1.961357002 podStartE2EDuration="1.961357002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:45:37.960625828 +0000 UTC m=+1171.320475655" watchObservedRunningTime="2026-09-09 13:45:37.961357002 +0000 UTC m=+1171.321206828"
Sep 09 13:45:37.976947 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:37.976894    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp" podStartSLOduration=1.976879672 podStartE2EDuration="1.976879672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:45:37.974610482 +0000 UTC m=+1171.334460321" watchObservedRunningTime="2026-09-09 13:45:37.976879672 +0000 UTC m=+1171.336729545"
Sep 09 13:45:41.177122 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:41.177084    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"
Sep 09 13:45:41.177122 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:41.177128    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"
Sep 09 13:45:41.189978 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:41.189949    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"
Sep 09 13:45:41.979554 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:41.979523    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"
Sep 09 13:45:49.959941 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:49.959908    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp"
Sep 09 13:45:49.960414 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:45:49.960326    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f"
Sep 09 13:46:07.170973 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:07.170945    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ftm96_af902e8b-d693-4341-9979-03cf69548cfb/console-operator/2.log"
Sep 09 13:46:07.178751 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:07.178729    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ftm96_af902e8b-d693-4341-9979-03cf69548cfb/console-operator/2.log"
Sep 09 13:46:07.291200 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:07.291171    2566 scope.go:117] "RemoveContainer" containerID="11c5ebf99c1e1306c635501ccdaee41e32628be8c06f6194a06aabab7e66ac4e"
Sep 09 13:46:28.696527 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:28.696494    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-638c40f1-g7898"]
Sep 09 13:46:28.701399 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:28.701380    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-638c40f1-g7898"
Sep 09 13:46:28.703723 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:28.703702    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 13:46:28.703845 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:28.703743    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lh487\""
Sep 09 13:46:28.717248 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:28.712077    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-638c40f1-g7898"]
Sep 09 13:46:28.827326 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:28.827287    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6aa0ec6-287f-49da-99c2-01fd07604e6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-638c40f1-g7898\" (UID: \"e6aa0ec6-287f-49da-99c2-01fd07604e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-638c40f1-g7898"
Sep 09 13:46:28.928701 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:28.928668    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6aa0ec6-287f-49da-99c2-01fd07604e6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-638c40f1-g7898\" (UID: \"e6aa0ec6-287f-49da-99c2-01fd07604e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-638c40f1-g7898"
Sep 09 13:46:28.929328 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:28.929304    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6aa0ec6-287f-49da-99c2-01fd07604e6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-638c40f1-g7898\" (UID: \"e6aa0ec6-287f-49da-99c2-01fd07604e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-638c40f1-g7898"
Sep 09 13:46:29.018345 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:29.018254    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-638c40f1-g7898"
Sep 09 13:46:29.288548 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:29.288449    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-638c40f1-g7898"]
Sep 09 13:46:29.291251 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:46:29.291223    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6aa0ec6_287f_49da_99c2_01fd07604e6c.slice/crio-01dcc34689367b2813cf6533346d1b4b94d89520d45aec84be009b80477010ba WatchSource:0}: Error finding container 01dcc34689367b2813cf6533346d1b4b94d89520d45aec84be009b80477010ba: Status 404 returned error can't find the container with id 01dcc34689367b2813cf6533346d1b4b94d89520d45aec84be009b80477010ba
Sep 09 13:46:30.241915 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:30.241833    2566 generic.go:356] "Generic (PLEG): container finished" podID="e6aa0ec6-287f-49da-99c2-01fd07604e6c" containerID="52bcc10af7d46d51141a9761bdebea516aaa720c4bf2d0c9826ed318939d300d" exitCode=0
Sep 09 13:46:30.241915 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:30.241898    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-638c40f1-g7898" event={"ID":"e6aa0ec6-287f-49da-99c2-01fd07604e6c","Type":"ContainerDied","Data":"52bcc10af7d46d51141a9761bdebea516aaa720c4bf2d0c9826ed318939d300d"}
Sep 09 13:46:30.242310 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:30.241936    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-638c40f1-g7898" event={"ID":"e6aa0ec6-287f-49da-99c2-01fd07604e6c","Type":"ContainerStarted","Data":"01dcc34689367b2813cf6533346d1b4b94d89520d45aec84be009b80477010ba"}
Sep 09 13:46:31.386415 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:31.386389    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-638c40f1-g7898"
Sep 09 13:46:31.449074 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:31.449035    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6aa0ec6-287f-49da-99c2-01fd07604e6c-maas-api-service-ca\") pod \"e6aa0ec6-287f-49da-99c2-01fd07604e6c\" (UID: \"e6aa0ec6-287f-49da-99c2-01fd07604e6c\") "
Sep 09 13:46:31.449460 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:31.449439    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6aa0ec6-287f-49da-99c2-01fd07604e6c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e6aa0ec6-287f-49da-99c2-01fd07604e6c" (UID: "e6aa0ec6-287f-49da-99c2-01fd07604e6c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:46:31.550858 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:31.550753    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6aa0ec6-287f-49da-99c2-01fd07604e6c-maas-api-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:46:32.253714 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:32.253688    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-638c40f1-g7898"
Sep 09 13:46:32.253925 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:32.253689    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-638c40f1-g7898" event={"ID":"e6aa0ec6-287f-49da-99c2-01fd07604e6c","Type":"ContainerDied","Data":"01dcc34689367b2813cf6533346d1b4b94d89520d45aec84be009b80477010ba"}
Sep 09 13:46:32.253925 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:32.253820    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01dcc34689367b2813cf6533346d1b4b94d89520d45aec84be009b80477010ba"
Sep 09 13:46:44.555385 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:44.555347    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp"]
Sep 09 13:46:44.555877 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:44.555622    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp" podUID="bb9d66fc-74f4-4a3a-8a82-a605b939a7f0" containerName="payload-processing" containerID="cri-o://8ad690292e3bdc3859d13a2f053b964aaf1308501bfd0d588ea428c3a05dffc6" gracePeriod=30
Sep 09 13:46:44.565309 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:44.565273    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f"]
Sep 09 13:46:44.565827 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:44.565770    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f" podUID="d4eb5664-773e-4eb4-b890-feb940b1aad3" containerName="payload-pre-processing" containerID="cri-o://b7cab5bde19485fea841a478923252b2dd72d182f560ca83d83b5cdfac3a1f1e" gracePeriod=30
Sep 09 13:46:44.751980 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:44.751949    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-638c40f1-g7898"]
Sep 09 13:46:44.753941 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:44.753920    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-638c40f1-g7898"]
Sep 09 13:46:44.848589 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:44.848564    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp"
Sep 09 13:46:44.851291 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:44.851273    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f"
Sep 09 13:46:44.983211 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:44.983175    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4eb5664-773e-4eb4-b890-feb940b1aad3-plugins-config-volume\") pod \"d4eb5664-773e-4eb4-b890-feb940b1aad3\" (UID: \"d4eb5664-773e-4eb4-b890-feb940b1aad3\") "
Sep 09 13:46:44.983393 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:44.983228    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzqd4\" (UniqueName: \"kubernetes.io/projected/bb9d66fc-74f4-4a3a-8a82-a605b939a7f0-kube-api-access-xzqd4\") pod \"bb9d66fc-74f4-4a3a-8a82-a605b939a7f0\" (UID: \"bb9d66fc-74f4-4a3a-8a82-a605b939a7f0\") "
Sep 09 13:46:44.983393 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:44.983271    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqgjl\" (UniqueName: \"kubernetes.io/projected/d4eb5664-773e-4eb4-b890-feb940b1aad3-kube-api-access-bqgjl\") pod \"d4eb5664-773e-4eb4-b890-feb940b1aad3\" (UID: \"d4eb5664-773e-4eb4-b890-feb940b1aad3\") "
Sep 09 13:46:44.983393 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:44.983304    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb9d66fc-74f4-4a3a-8a82-a605b939a7f0-plugins-config-volume\") pod \"bb9d66fc-74f4-4a3a-8a82-a605b939a7f0\" (UID: \"bb9d66fc-74f4-4a3a-8a82-a605b939a7f0\") "
Sep 09 13:46:44.983658 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:44.983628    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4eb5664-773e-4eb4-b890-feb940b1aad3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d4eb5664-773e-4eb4-b890-feb940b1aad3" (UID: "d4eb5664-773e-4eb4-b890-feb940b1aad3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:46:44.983724 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:44.983701    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb9d66fc-74f4-4a3a-8a82-a605b939a7f0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bb9d66fc-74f4-4a3a-8a82-a605b939a7f0" (UID: "bb9d66fc-74f4-4a3a-8a82-a605b939a7f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:46:44.985420 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:44.985400    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4eb5664-773e-4eb4-b890-feb940b1aad3-kube-api-access-bqgjl" (OuterVolumeSpecName: "kube-api-access-bqgjl") pod "d4eb5664-773e-4eb4-b890-feb940b1aad3" (UID: "d4eb5664-773e-4eb4-b890-feb940b1aad3"). InnerVolumeSpecName "kube-api-access-bqgjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:46:44.985496 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:44.985472    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb9d66fc-74f4-4a3a-8a82-a605b939a7f0-kube-api-access-xzqd4" (OuterVolumeSpecName: "kube-api-access-xzqd4") pod "bb9d66fc-74f4-4a3a-8a82-a605b939a7f0" (UID: "bb9d66fc-74f4-4a3a-8a82-a605b939a7f0"). InnerVolumeSpecName "kube-api-access-xzqd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:46:45.085037 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:45.085003    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqgjl\" (UniqueName: \"kubernetes.io/projected/d4eb5664-773e-4eb4-b890-feb940b1aad3-kube-api-access-bqgjl\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:46:45.085037 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:45.085035    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb9d66fc-74f4-4a3a-8a82-a605b939a7f0-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:46:45.085214 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:45.085046    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4eb5664-773e-4eb4-b890-feb940b1aad3-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:46:45.085214 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:45.085055    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzqd4\" (UniqueName: \"kubernetes.io/projected/bb9d66fc-74f4-4a3a-8a82-a605b939a7f0-kube-api-access-xzqd4\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:46:45.149993 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:45.149963    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6aa0ec6-287f-49da-99c2-01fd07604e6c" path="/var/lib/kubelet/pods/e6aa0ec6-287f-49da-99c2-01fd07604e6c/volumes"
Sep 09 13:46:45.326842 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:45.326791    2566 generic.go:356] "Generic (PLEG): container finished" podID="bb9d66fc-74f4-4a3a-8a82-a605b939a7f0" containerID="8ad690292e3bdc3859d13a2f053b964aaf1308501bfd0d588ea428c3a05dffc6" exitCode=0
Sep 09 13:46:45.326998 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:45.326860    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp"
Sep 09 13:46:45.326998 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:45.326869    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp" event={"ID":"bb9d66fc-74f4-4a3a-8a82-a605b939a7f0","Type":"ContainerDied","Data":"8ad690292e3bdc3859d13a2f053b964aaf1308501bfd0d588ea428c3a05dffc6"}
Sep 09 13:46:45.326998 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:45.326923    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp" event={"ID":"bb9d66fc-74f4-4a3a-8a82-a605b939a7f0","Type":"ContainerDied","Data":"572886626c16cce62f8bd142eab6486637c22d5b4bf4c1f54d9640010e8c261f"}
Sep 09 13:46:45.326998 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:45.326945    2566 scope.go:117] "RemoveContainer" containerID="8ad690292e3bdc3859d13a2f053b964aaf1308501bfd0d588ea428c3a05dffc6"
Sep 09 13:46:45.328402 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:45.328376    2566 generic.go:356] "Generic (PLEG): container finished" podID="d4eb5664-773e-4eb4-b890-feb940b1aad3" containerID="b7cab5bde19485fea841a478923252b2dd72d182f560ca83d83b5cdfac3a1f1e" exitCode=0
Sep 09 13:46:45.328498 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:45.328431    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f" event={"ID":"d4eb5664-773e-4eb4-b890-feb940b1aad3","Type":"ContainerDied","Data":"b7cab5bde19485fea841a478923252b2dd72d182f560ca83d83b5cdfac3a1f1e"}
Sep 09 13:46:45.328498 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:45.328475    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f" event={"ID":"d4eb5664-773e-4eb4-b890-feb940b1aad3","Type":"ContainerDied","Data":"75e0e781eb85d33d9546df981bdb3bb688866986ec13c9ee22544975bf0e9d70"}
Sep 09 13:46:45.328498 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:45.328436    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f"
Sep 09 13:46:45.336205 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:45.336184    2566 scope.go:117] "RemoveContainer" containerID="8ad690292e3bdc3859d13a2f053b964aaf1308501bfd0d588ea428c3a05dffc6"
Sep 09 13:46:45.336488 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:46:45.336471    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ad690292e3bdc3859d13a2f053b964aaf1308501bfd0d588ea428c3a05dffc6\": container with ID starting with 8ad690292e3bdc3859d13a2f053b964aaf1308501bfd0d588ea428c3a05dffc6 not found: ID does not exist" containerID="8ad690292e3bdc3859d13a2f053b964aaf1308501bfd0d588ea428c3a05dffc6"
Sep 09 13:46:45.336551 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:45.336498    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ad690292e3bdc3859d13a2f053b964aaf1308501bfd0d588ea428c3a05dffc6"} err="failed to get container status \"8ad690292e3bdc3859d13a2f053b964aaf1308501bfd0d588ea428c3a05dffc6\": rpc error: code = NotFound desc = could not find container \"8ad690292e3bdc3859d13a2f053b964aaf1308501bfd0d588ea428c3a05dffc6\": container with ID starting with 8ad690292e3bdc3859d13a2f053b964aaf1308501bfd0d588ea428c3a05dffc6 not found: ID does not exist"
Sep 09 13:46:45.336551 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:45.336516    2566 scope.go:117] "RemoveContainer" containerID="b7cab5bde19485fea841a478923252b2dd72d182f560ca83d83b5cdfac3a1f1e"
Sep 09 13:46:45.346614 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:45.346593    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp"]
Sep 09 13:46:45.348620 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:45.348601    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-638c40f1-55b9bc88bf-lzxlp"]
Sep 09 13:46:45.351738 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:45.351718    2566 scope.go:117] "RemoveContainer" containerID="b7cab5bde19485fea841a478923252b2dd72d182f560ca83d83b5cdfac3a1f1e"
Sep 09 13:46:45.352035 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:46:45.352016    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7cab5bde19485fea841a478923252b2dd72d182f560ca83d83b5cdfac3a1f1e\": container with ID starting with b7cab5bde19485fea841a478923252b2dd72d182f560ca83d83b5cdfac3a1f1e not found: ID does not exist" containerID="b7cab5bde19485fea841a478923252b2dd72d182f560ca83d83b5cdfac3a1f1e"
Sep 09 13:46:45.352090 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:45.352044    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7cab5bde19485fea841a478923252b2dd72d182f560ca83d83b5cdfac3a1f1e"} err="failed to get container status \"b7cab5bde19485fea841a478923252b2dd72d182f560ca83d83b5cdfac3a1f1e\": rpc error: code = NotFound desc = could not find container \"b7cab5bde19485fea841a478923252b2dd72d182f560ca83d83b5cdfac3a1f1e\": container with ID starting with b7cab5bde19485fea841a478923252b2dd72d182f560ca83d83b5cdfac3a1f1e not found: ID does not exist"
Sep 09 13:46:45.359605 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:45.359583    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f"]
Sep 09 13:46:45.363671 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:45.363650    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-638c40f1-5dcf8b878b-lh27f"]
Sep 09 13:46:47.150703 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:47.150672    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb9d66fc-74f4-4a3a-8a82-a605b939a7f0" path="/var/lib/kubelet/pods/bb9d66fc-74f4-4a3a-8a82-a605b939a7f0/volumes"
Sep 09 13:46:47.151089 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:46:47.151078    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4eb5664-773e-4eb4-b890-feb940b1aad3" path="/var/lib/kubelet/pods/d4eb5664-773e-4eb4-b890-feb940b1aad3/volumes"
Sep 09 13:47:07.311681 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:47:07.311601    2566 scope.go:117] "RemoveContainer" containerID="740a79d962f7ba6f3aebeb4fa8abed59ec9b7ac5889cbd19568d860973026206"
Sep 09 13:47:07.320140 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:47:07.320120    2566 scope.go:117] "RemoveContainer" containerID="31ae7ec6bc1a409beae67a65d0ebfd18f994595a47977e0a01e3ee7b89ed49cc"
Sep 09 13:47:48.073803 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:47:48.073759    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"]
Sep 09 13:47:49.263853 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:47:49.263801    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"]
Sep 09 13:48:03.095603 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.095557    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf" podUID="f70c6d78-ea87-4176-b94f-851ae5270e0f" containerName="main" containerID="cri-o://2657826ba4f9314a62d2820c0c79d12b88d77b88e72f358eb33ffd0bd608eeeb" gracePeriod=45
Sep 09 13:48:03.354135 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.354060    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"
Sep 09 13:48:03.450874 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.450840    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f70c6d78-ea87-4176-b94f-851ae5270e0f-tmp-dir\") pod \"f70c6d78-ea87-4176-b94f-851ae5270e0f\" (UID: \"f70c6d78-ea87-4176-b94f-851ae5270e0f\") "
Sep 09 13:48:03.451069 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.450914    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f70c6d78-ea87-4176-b94f-851ae5270e0f-dshm\") pod \"f70c6d78-ea87-4176-b94f-851ae5270e0f\" (UID: \"f70c6d78-ea87-4176-b94f-851ae5270e0f\") "
Sep 09 13:48:03.451069 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.450941    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f70c6d78-ea87-4176-b94f-851ae5270e0f-home\") pod \"f70c6d78-ea87-4176-b94f-851ae5270e0f\" (UID: \"f70c6d78-ea87-4176-b94f-851ae5270e0f\") "
Sep 09 13:48:03.451069 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.450965    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8hfv\" (UniqueName: \"kubernetes.io/projected/f70c6d78-ea87-4176-b94f-851ae5270e0f-kube-api-access-n8hfv\") pod \"f70c6d78-ea87-4176-b94f-851ae5270e0f\" (UID: \"f70c6d78-ea87-4176-b94f-851ae5270e0f\") "
Sep 09 13:48:03.451069 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.450990    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f70c6d78-ea87-4176-b94f-851ae5270e0f-tls-certs\") pod \"f70c6d78-ea87-4176-b94f-851ae5270e0f\" (UID: \"f70c6d78-ea87-4176-b94f-851ae5270e0f\") "
Sep 09 13:48:03.451069 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.451039    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f70c6d78-ea87-4176-b94f-851ae5270e0f-model-cache\") pod \"f70c6d78-ea87-4176-b94f-851ae5270e0f\" (UID: \"f70c6d78-ea87-4176-b94f-851ae5270e0f\") "
Sep 09 13:48:03.451352 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.451190    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70c6d78-ea87-4176-b94f-851ae5270e0f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f70c6d78-ea87-4176-b94f-851ae5270e0f" (UID: "f70c6d78-ea87-4176-b94f-851ae5270e0f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:48:03.451352 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.451244    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70c6d78-ea87-4176-b94f-851ae5270e0f-home" (OuterVolumeSpecName: "home") pod "f70c6d78-ea87-4176-b94f-851ae5270e0f" (UID: "f70c6d78-ea87-4176-b94f-851ae5270e0f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:48:03.451457 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.451411    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70c6d78-ea87-4176-b94f-851ae5270e0f-model-cache" (OuterVolumeSpecName: "model-cache") pod "f70c6d78-ea87-4176-b94f-851ae5270e0f" (UID: "f70c6d78-ea87-4176-b94f-851ae5270e0f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:48:03.451457 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.451436    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f70c6d78-ea87-4176-b94f-851ae5270e0f-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:48:03.451549 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.451456    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f70c6d78-ea87-4176-b94f-851ae5270e0f-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:48:03.453341 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.453316    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70c6d78-ea87-4176-b94f-851ae5270e0f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f70c6d78-ea87-4176-b94f-851ae5270e0f" (UID: "f70c6d78-ea87-4176-b94f-851ae5270e0f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:48:03.453424 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.453355    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70c6d78-ea87-4176-b94f-851ae5270e0f-dshm" (OuterVolumeSpecName: "dshm") pod "f70c6d78-ea87-4176-b94f-851ae5270e0f" (UID: "f70c6d78-ea87-4176-b94f-851ae5270e0f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:48:03.453460 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.453418    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70c6d78-ea87-4176-b94f-851ae5270e0f-kube-api-access-n8hfv" (OuterVolumeSpecName: "kube-api-access-n8hfv") pod "f70c6d78-ea87-4176-b94f-851ae5270e0f" (UID: "f70c6d78-ea87-4176-b94f-851ae5270e0f"). InnerVolumeSpecName "kube-api-access-n8hfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:48:03.552543 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.552497    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8hfv\" (UniqueName: \"kubernetes.io/projected/f70c6d78-ea87-4176-b94f-851ae5270e0f-kube-api-access-n8hfv\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:48:03.552543 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.552537    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f70c6d78-ea87-4176-b94f-851ae5270e0f-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:48:03.552543 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.552553    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f70c6d78-ea87-4176-b94f-851ae5270e0f-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:48:03.552791 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.552564    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f70c6d78-ea87-4176-b94f-851ae5270e0f-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:48:03.744287 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.744195    2566 generic.go:356] "Generic (PLEG): container finished" podID="f70c6d78-ea87-4176-b94f-851ae5270e0f" containerID="2657826ba4f9314a62d2820c0c79d12b88d77b88e72f358eb33ffd0bd608eeeb" exitCode=0
Sep 09 13:48:03.744287 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.744235    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"
Sep 09 13:48:03.744507 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.744281    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf" event={"ID":"f70c6d78-ea87-4176-b94f-851ae5270e0f","Type":"ContainerDied","Data":"2657826ba4f9314a62d2820c0c79d12b88d77b88e72f358eb33ffd0bd608eeeb"}
Sep 09 13:48:03.744507 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.744326    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf" event={"ID":"f70c6d78-ea87-4176-b94f-851ae5270e0f","Type":"ContainerDied","Data":"731ddca9694e092aaa756fb2a4d202ac53d7676f9675905a6e4169150330bda9"}
Sep 09 13:48:03.744507 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.744347    2566 scope.go:117] "RemoveContainer" containerID="2657826ba4f9314a62d2820c0c79d12b88d77b88e72f358eb33ffd0bd608eeeb"
Sep 09 13:48:03.754397 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.754376    2566 scope.go:117] "RemoveContainer" containerID="2657826ba4f9314a62d2820c0c79d12b88d77b88e72f358eb33ffd0bd608eeeb"
Sep 09 13:48:03.754666 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:48:03.754649    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2657826ba4f9314a62d2820c0c79d12b88d77b88e72f358eb33ffd0bd608eeeb\": container with ID starting with 2657826ba4f9314a62d2820c0c79d12b88d77b88e72f358eb33ffd0bd608eeeb not found: ID does not exist" containerID="2657826ba4f9314a62d2820c0c79d12b88d77b88e72f358eb33ffd0bd608eeeb"
Sep 09 13:48:03.754718 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.754675    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2657826ba4f9314a62d2820c0c79d12b88d77b88e72f358eb33ffd0bd608eeeb"} err="failed to get container status \"2657826ba4f9314a62d2820c0c79d12b88d77b88e72f358eb33ffd0bd608eeeb\": rpc error: code = NotFound desc = could not find container \"2657826ba4f9314a62d2820c0c79d12b88d77b88e72f358eb33ffd0bd608eeeb\": container with ID starting with 2657826ba4f9314a62d2820c0c79d12b88d77b88e72f358eb33ffd0bd608eeeb not found: ID does not exist"
Sep 09 13:48:03.765868 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.765837    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"]
Sep 09 13:48:03.769567 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:03.769540    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f2df2/auth-test-model-8f2df2-kserve-7d58585547-6n7lf"]
Sep 09 13:48:04.284185 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.284149    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8" podUID="7deaa929-051a-43dc-8ea5-572dc2b23533" containerName="main" containerID="cri-o://be9dbf7104b5f17fa3ded791ff806a33c675aa3932cc457ee7ab758d1d3d0d48" gracePeriod=45
Sep 09 13:48:04.535150 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.535077    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"
Sep 09 13:48:04.663369 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.663331    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x78xl\" (UniqueName: \"kubernetes.io/projected/7deaa929-051a-43dc-8ea5-572dc2b23533-kube-api-access-x78xl\") pod \"7deaa929-051a-43dc-8ea5-572dc2b23533\" (UID: \"7deaa929-051a-43dc-8ea5-572dc2b23533\") "
Sep 09 13:48:04.663574 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.663383    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7deaa929-051a-43dc-8ea5-572dc2b23533-model-cache\") pod \"7deaa929-051a-43dc-8ea5-572dc2b23533\" (UID: \"7deaa929-051a-43dc-8ea5-572dc2b23533\") "
Sep 09 13:48:04.663574 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.663417    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7deaa929-051a-43dc-8ea5-572dc2b23533-tls-certs\") pod \"7deaa929-051a-43dc-8ea5-572dc2b23533\" (UID: \"7deaa929-051a-43dc-8ea5-572dc2b23533\") "
Sep 09 13:48:04.663574 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.663449    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7deaa929-051a-43dc-8ea5-572dc2b23533-dshm\") pod \"7deaa929-051a-43dc-8ea5-572dc2b23533\" (UID: \"7deaa929-051a-43dc-8ea5-572dc2b23533\") "
Sep 09 13:48:04.663574 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.663476    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7deaa929-051a-43dc-8ea5-572dc2b23533-tmp-dir\") pod \"7deaa929-051a-43dc-8ea5-572dc2b23533\" (UID: \"7deaa929-051a-43dc-8ea5-572dc2b23533\") "
Sep 09 13:48:04.663789 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.663627    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7deaa929-051a-43dc-8ea5-572dc2b23533-home\") pod \"7deaa929-051a-43dc-8ea5-572dc2b23533\" (UID: \"7deaa929-051a-43dc-8ea5-572dc2b23533\") "
Sep 09 13:48:04.663789 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.663659    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7deaa929-051a-43dc-8ea5-572dc2b23533-model-cache" (OuterVolumeSpecName: "model-cache") pod "7deaa929-051a-43dc-8ea5-572dc2b23533" (UID: "7deaa929-051a-43dc-8ea5-572dc2b23533"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:48:04.663789 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.663764    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7deaa929-051a-43dc-8ea5-572dc2b23533-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7deaa929-051a-43dc-8ea5-572dc2b23533" (UID: "7deaa929-051a-43dc-8ea5-572dc2b23533"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:48:04.663977 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.663872    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7deaa929-051a-43dc-8ea5-572dc2b23533-home" (OuterVolumeSpecName: "home") pod "7deaa929-051a-43dc-8ea5-572dc2b23533" (UID: "7deaa929-051a-43dc-8ea5-572dc2b23533"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:48:04.664031 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.664007    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7deaa929-051a-43dc-8ea5-572dc2b23533-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:48:04.664031 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.664020    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7deaa929-051a-43dc-8ea5-572dc2b23533-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:48:04.664031 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.664030    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7deaa929-051a-43dc-8ea5-572dc2b23533-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:48:04.665870 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.665840    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7deaa929-051a-43dc-8ea5-572dc2b23533-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7deaa929-051a-43dc-8ea5-572dc2b23533" (UID: "7deaa929-051a-43dc-8ea5-572dc2b23533"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:48:04.665973 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.665868    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7deaa929-051a-43dc-8ea5-572dc2b23533-dshm" (OuterVolumeSpecName: "dshm") pod "7deaa929-051a-43dc-8ea5-572dc2b23533" (UID: "7deaa929-051a-43dc-8ea5-572dc2b23533"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:48:04.665973 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.665945    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7deaa929-051a-43dc-8ea5-572dc2b23533-kube-api-access-x78xl" (OuterVolumeSpecName: "kube-api-access-x78xl") pod "7deaa929-051a-43dc-8ea5-572dc2b23533" (UID: "7deaa929-051a-43dc-8ea5-572dc2b23533"). InnerVolumeSpecName "kube-api-access-x78xl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:48:04.752004 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.751964    2566 generic.go:356] "Generic (PLEG): container finished" podID="7deaa929-051a-43dc-8ea5-572dc2b23533" containerID="be9dbf7104b5f17fa3ded791ff806a33c675aa3932cc457ee7ab758d1d3d0d48" exitCode=0
Sep 09 13:48:04.752209 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.752009    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"
Sep 09 13:48:04.752209 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.752049    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8" event={"ID":"7deaa929-051a-43dc-8ea5-572dc2b23533","Type":"ContainerDied","Data":"be9dbf7104b5f17fa3ded791ff806a33c675aa3932cc457ee7ab758d1d3d0d48"}
Sep 09 13:48:04.752209 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.752094    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8" event={"ID":"7deaa929-051a-43dc-8ea5-572dc2b23533","Type":"ContainerDied","Data":"047f8c6339eb8de8df1c9fa8ca1753e89638f10b9b4b1add8ed8946a1e3158ec"}
Sep 09 13:48:04.752209 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.752112    2566 scope.go:117] "RemoveContainer" containerID="be9dbf7104b5f17fa3ded791ff806a33c675aa3932cc457ee7ab758d1d3d0d48"
Sep 09 13:48:04.761647 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.761622    2566 scope.go:117] "RemoveContainer" containerID="be9dbf7104b5f17fa3ded791ff806a33c675aa3932cc457ee7ab758d1d3d0d48"
Sep 09 13:48:04.761972 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:48:04.761956    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be9dbf7104b5f17fa3ded791ff806a33c675aa3932cc457ee7ab758d1d3d0d48\": container with ID starting with be9dbf7104b5f17fa3ded791ff806a33c675aa3932cc457ee7ab758d1d3d0d48 not found: ID does not exist" containerID="be9dbf7104b5f17fa3ded791ff806a33c675aa3932cc457ee7ab758d1d3d0d48"
Sep 09 13:48:04.762039 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.761982    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be9dbf7104b5f17fa3ded791ff806a33c675aa3932cc457ee7ab758d1d3d0d48"} err="failed to get container status \"be9dbf7104b5f17fa3ded791ff806a33c675aa3932cc457ee7ab758d1d3d0d48\": rpc error: code = NotFound desc = could not find container \"be9dbf7104b5f17fa3ded791ff806a33c675aa3932cc457ee7ab758d1d3d0d48\": container with ID starting with be9dbf7104b5f17fa3ded791ff806a33c675aa3932cc457ee7ab758d1d3d0d48 not found: ID does not exist"
Sep 09 13:48:04.765026 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.765004    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x78xl\" (UniqueName: \"kubernetes.io/projected/7deaa929-051a-43dc-8ea5-572dc2b23533-kube-api-access-x78xl\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:48:04.765129 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.765028    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7deaa929-051a-43dc-8ea5-572dc2b23533-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:48:04.765129 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.765040    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7deaa929-051a-43dc-8ea5-572dc2b23533-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:48:04.773997 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.773971    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"]
Sep 09 13:48:04.777166 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:04.777141    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-df48ae/auth-test-model-df48ae-kserve-db88d98f5-w65q8"]
Sep 09 13:48:05.151416 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:05.151374    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7deaa929-051a-43dc-8ea5-572dc2b23533" path="/var/lib/kubelet/pods/7deaa929-051a-43dc-8ea5-572dc2b23533/volumes"
Sep 09 13:48:05.151926 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:05.151905    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f70c6d78-ea87-4176-b94f-851ae5270e0f" path="/var/lib/kubelet/pods/f70c6d78-ea87-4176-b94f-851ae5270e0f/volumes"
Sep 09 13:48:07.349700 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:07.349667    2566 scope.go:117] "RemoveContainer" containerID="8f979c80f95fbd8f31bf0da6e1251e361d82f7f7153825376ec40ae2ded217e0"
Sep 09 13:48:07.357955 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:07.357936    2566 scope.go:117] "RemoveContainer" containerID="4e7b009861d40b868c57e291fb5c766bd268b99a621592e2628380d93e2073aa"
Sep 09 13:48:27.919561 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:27.919526    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"]
Sep 09 13:48:27.920078 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:27.919885    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7deaa929-051a-43dc-8ea5-572dc2b23533" containerName="main"
Sep 09 13:48:27.920078 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:27.919897    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7deaa929-051a-43dc-8ea5-572dc2b23533" containerName="main"
Sep 09 13:48:27.920078 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:27.919904    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70c6d78-ea87-4176-b94f-851ae5270e0f" containerName="main"
Sep 09 13:48:27.920078 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:27.919909    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70c6d78-ea87-4176-b94f-851ae5270e0f" containerName="main"
Sep 09 13:48:27.920078 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:27.919925    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4eb5664-773e-4eb4-b890-feb940b1aad3" containerName="payload-pre-processing"
Sep 09 13:48:27.920078 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:27.919931    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4eb5664-773e-4eb4-b890-feb940b1aad3" containerName="payload-pre-processing"
Sep 09 13:48:27.920078 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:27.919937    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb9d66fc-74f4-4a3a-8a82-a605b939a7f0" containerName="payload-processing"
Sep 09 13:48:27.920078 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:27.919942    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb9d66fc-74f4-4a3a-8a82-a605b939a7f0" containerName="payload-processing"
Sep 09 13:48:27.920078 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:27.919957    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6aa0ec6-287f-49da-99c2-01fd07604e6c" containerName="revoke-keys"
Sep 09 13:48:27.920078 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:27.919963    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6aa0ec6-287f-49da-99c2-01fd07604e6c" containerName="revoke-keys"
Sep 09 13:48:27.920078 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:27.920015    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4eb5664-773e-4eb4-b890-feb940b1aad3" containerName="payload-pre-processing"
Sep 09 13:48:27.920078 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:27.920022    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb9d66fc-74f4-4a3a-8a82-a605b939a7f0" containerName="payload-processing"
Sep 09 13:48:27.920078 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:27.920030    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7deaa929-051a-43dc-8ea5-572dc2b23533" containerName="main"
Sep 09 13:48:27.920078 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:27.920036    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f70c6d78-ea87-4176-b94f-851ae5270e0f" containerName="main"
Sep 09 13:48:27.920078 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:27.920043    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6aa0ec6-287f-49da-99c2-01fd07604e6c" containerName="revoke-keys"
Sep 09 13:48:27.924426 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:27.924408    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"
Sep 09 13:48:27.927303 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:27.927277    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-df48ae\"/\"openshift-service-ca.crt\""
Sep 09 13:48:27.927303 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:27.927297    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-df48ae\"/\"kube-root-ca.crt\""
Sep 09 13:48:27.927500 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:27.927343    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-df48ae\"/\"default-dockercfg-cf6vl\""
Sep 09 13:48:27.927500 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:27.927343    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-df48ae\"/\"sub-test-model-df48ae-kserve-self-signed-certs\""
Sep 09 13:48:27.934964 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:27.934937    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"]
Sep 09 13:48:27.972484 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:27.972440    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df2b86eb-f905-4134-916b-1d821947a339-home\") pod \"sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8\" (UID: \"df2b86eb-f905-4134-916b-1d821947a339\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"
Sep 09 13:48:27.972677 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:27.972527    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2b86eb-f905-4134-916b-1d821947a339-tmp-dir\") pod \"sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8\" (UID: \"df2b86eb-f905-4134-916b-1d821947a339\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"
Sep 09 13:48:27.972677 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:27.972591    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x62nk\" (UniqueName: \"kubernetes.io/projected/df2b86eb-f905-4134-916b-1d821947a339-kube-api-access-x62nk\") pod \"sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8\" (UID: \"df2b86eb-f905-4134-916b-1d821947a339\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"
Sep 09 13:48:27.972677 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:27.972629    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df2b86eb-f905-4134-916b-1d821947a339-tls-certs\") pod \"sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8\" (UID: \"df2b86eb-f905-4134-916b-1d821947a339\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"
Sep 09 13:48:27.972873 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:27.972713    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df2b86eb-f905-4134-916b-1d821947a339-model-cache\") pod \"sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8\" (UID: \"df2b86eb-f905-4134-916b-1d821947a339\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"
Sep 09 13:48:27.972873 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:27.972747    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df2b86eb-f905-4134-916b-1d821947a339-dshm\") pod \"sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8\" (UID: \"df2b86eb-f905-4134-916b-1d821947a339\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"
Sep 09 13:48:28.074139 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:28.074104    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2b86eb-f905-4134-916b-1d821947a339-tmp-dir\") pod \"sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8\" (UID: \"df2b86eb-f905-4134-916b-1d821947a339\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"
Sep 09 13:48:28.074341 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:28.074148    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x62nk\" (UniqueName: \"kubernetes.io/projected/df2b86eb-f905-4134-916b-1d821947a339-kube-api-access-x62nk\") pod \"sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8\" (UID: \"df2b86eb-f905-4134-916b-1d821947a339\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"
Sep 09 13:48:28.074341 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:28.074167    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df2b86eb-f905-4134-916b-1d821947a339-tls-certs\") pod \"sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8\" (UID: \"df2b86eb-f905-4134-916b-1d821947a339\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"
Sep 09 13:48:28.074341 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:28.074220    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df2b86eb-f905-4134-916b-1d821947a339-model-cache\") pod \"sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8\" (UID: \"df2b86eb-f905-4134-916b-1d821947a339\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"
Sep 09 13:48:28.074341 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:28.074241    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df2b86eb-f905-4134-916b-1d821947a339-dshm\") pod \"sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8\" (UID: \"df2b86eb-f905-4134-916b-1d821947a339\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"
Sep 09 13:48:28.074341 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:28.074268    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df2b86eb-f905-4134-916b-1d821947a339-home\") pod \"sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8\" (UID: \"df2b86eb-f905-4134-916b-1d821947a339\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"
Sep 09 13:48:28.074620 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:28.074571    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df2b86eb-f905-4134-916b-1d821947a339-home\") pod \"sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8\" (UID: \"df2b86eb-f905-4134-916b-1d821947a339\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"
Sep 09 13:48:28.074620 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:28.074573    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2b86eb-f905-4134-916b-1d821947a339-tmp-dir\") pod \"sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8\" (UID: \"df2b86eb-f905-4134-916b-1d821947a339\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"
Sep 09 13:48:28.074725 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:28.074629    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df2b86eb-f905-4134-916b-1d821947a339-model-cache\") pod \"sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8\" (UID: \"df2b86eb-f905-4134-916b-1d821947a339\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"
Sep 09 13:48:28.076567 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:28.076547    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df2b86eb-f905-4134-916b-1d821947a339-dshm\") pod \"sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8\" (UID: \"df2b86eb-f905-4134-916b-1d821947a339\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"
Sep 09 13:48:28.076839 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:28.076803    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df2b86eb-f905-4134-916b-1d821947a339-tls-certs\") pod \"sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8\" (UID: \"df2b86eb-f905-4134-916b-1d821947a339\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"
Sep 09 13:48:28.082689 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:28.082664    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x62nk\" (UniqueName: \"kubernetes.io/projected/df2b86eb-f905-4134-916b-1d821947a339-kube-api-access-x62nk\") pod \"sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8\" (UID: \"df2b86eb-f905-4134-916b-1d821947a339\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"
Sep 09 13:48:28.235365 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:28.235279    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"
Sep 09 13:48:28.512171 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:28.512142    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"]
Sep 09 13:48:28.513796 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:48:28.513758    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf2b86eb_f905_4134_916b_1d821947a339.slice/crio-bf0a3161de112b3536c8cfad13129d55d3138500b9290eadb3db47bdcb3382d6 WatchSource:0}: Error finding container bf0a3161de112b3536c8cfad13129d55d3138500b9290eadb3db47bdcb3382d6: Status 404 returned error can't find the container with id bf0a3161de112b3536c8cfad13129d55d3138500b9290eadb3db47bdcb3382d6
Sep 09 13:48:28.873611 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:28.873578    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8" event={"ID":"df2b86eb-f905-4134-916b-1d821947a339","Type":"ContainerStarted","Data":"f808b217965772c7d37ddd5d42f181e9e2f04488dd7738bc3146b7fbb87e222c"}
Sep 09 13:48:28.873611 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:28.873623    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8" event={"ID":"df2b86eb-f905-4134-916b-1d821947a339","Type":"ContainerStarted","Data":"bf0a3161de112b3536c8cfad13129d55d3138500b9290eadb3db47bdcb3382d6"}
Sep 09 13:48:28.892747 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:28.892698    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8" podStartSLOduration=1.8926839979999999 podStartE2EDuration="1.892683998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:48:28.890745263 +0000 UTC m=+1342.250595089" watchObservedRunningTime="2026-09-09 13:48:28.892683998 +0000 UTC m=+1342.252533824"
Sep 09 13:48:38.236297 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:38.236257    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"
Sep 09 13:48:38.236865 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:38.236314    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"
Sep 09 13:48:38.248605 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:38.248582    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"
Sep 09 13:48:38.941373 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:38.941340    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"
Sep 09 13:48:50.629609 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:50.629579    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl"]
Sep 09 13:48:50.634686 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:50.634668    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl"
Sep 09 13:48:50.636905 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:50.636885    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-464c54\""
Sep 09 13:48:50.637018 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:50.636947    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-464c54-dockercfg-ztnwz\""
Sep 09 13:48:50.644395 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:50.644372    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl"]
Sep 09 13:48:50.665211 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:50.665185    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2"]
Sep 09 13:48:50.668426 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:50.668411    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2"
Sep 09 13:48:50.678633 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:50.678606    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2"]
Sep 09 13:48:50.703609 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:50.703572    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2\" (UID: \"dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2"
Sep 09 13:48:50.703896 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:50.703871    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b40db715-280a-448c-bf26-4ac0eb383937-plugins-config-volume\") pod \"payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl\" (UID: \"b40db715-280a-448c-bf26-4ac0eb383937\") " pod="openshift-ingress/payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl"
Sep 09 13:48:50.704182 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:50.704132    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g74j\" (UniqueName: \"kubernetes.io/projected/b40db715-280a-448c-bf26-4ac0eb383937-kube-api-access-6g74j\") pod \"payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl\" (UID: \"b40db715-280a-448c-bf26-4ac0eb383937\") " pod="openshift-ingress/payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl"
Sep 09 13:48:50.708032 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:50.708007    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk2n7\" (UniqueName: \"kubernetes.io/projected/dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15-kube-api-access-vk2n7\") pod \"payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2\" (UID: \"dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2"
Sep 09 13:48:50.809085 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:50.809053    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2\" (UID: \"dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2"
Sep 09 13:48:50.809280 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:50.809109    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b40db715-280a-448c-bf26-4ac0eb383937-plugins-config-volume\") pod \"payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl\" (UID: \"b40db715-280a-448c-bf26-4ac0eb383937\") " pod="openshift-ingress/payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl"
Sep 09 13:48:50.809280 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:50.809139    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g74j\" (UniqueName: \"kubernetes.io/projected/b40db715-280a-448c-bf26-4ac0eb383937-kube-api-access-6g74j\") pod \"payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl\" (UID: \"b40db715-280a-448c-bf26-4ac0eb383937\") " pod="openshift-ingress/payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl"
Sep 09 13:48:50.809280 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:50.809169    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk2n7\" (UniqueName: \"kubernetes.io/projected/dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15-kube-api-access-vk2n7\") pod \"payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2\" (UID: \"dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2"
Sep 09 13:48:50.809836 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:50.809790    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b40db715-280a-448c-bf26-4ac0eb383937-plugins-config-volume\") pod \"payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl\" (UID: \"b40db715-280a-448c-bf26-4ac0eb383937\") " pod="openshift-ingress/payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl"
Sep 09 13:48:50.809893 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:50.809847    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2\" (UID: \"dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2"
Sep 09 13:48:50.817895 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:50.817872    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g74j\" (UniqueName: \"kubernetes.io/projected/b40db715-280a-448c-bf26-4ac0eb383937-kube-api-access-6g74j\") pod \"payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl\" (UID: \"b40db715-280a-448c-bf26-4ac0eb383937\") " pod="openshift-ingress/payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl"
Sep 09 13:48:50.818068 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:50.818048    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk2n7\" (UniqueName: \"kubernetes.io/projected/dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15-kube-api-access-vk2n7\") pod \"payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2\" (UID: \"dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2"
Sep 09 13:48:50.946234 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:50.946150    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl"
Sep 09 13:48:50.979629 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:50.979597    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2"
Sep 09 13:48:51.220652 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:51.220629    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl"]
Sep 09 13:48:51.222277 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:48:51.222249    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb40db715_280a_448c_bf26_4ac0eb383937.slice/crio-c3e201f4e323c7044223cca921cbc5e3e80d540b760fc086d6647fe49ffc3636 WatchSource:0}: Error finding container c3e201f4e323c7044223cca921cbc5e3e80d540b760fc086d6647fe49ffc3636: Status 404 returned error can't find the container with id c3e201f4e323c7044223cca921cbc5e3e80d540b760fc086d6647fe49ffc3636
Sep 09 13:48:51.265357 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:51.265189    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2"]
Sep 09 13:48:51.267099 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:48:51.267072    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfde75b2_2dd1_41d1_a8cd_8bb22ee20b15.slice/crio-e814cab942ce544dad13bac7a9480590af7b5462055a39ad827ccfb3bc908456 WatchSource:0}: Error finding container e814cab942ce544dad13bac7a9480590af7b5462055a39ad827ccfb3bc908456: Status 404 returned error can't find the container with id e814cab942ce544dad13bac7a9480590af7b5462055a39ad827ccfb3bc908456
Sep 09 13:48:52.027880 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:52.027840    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl" event={"ID":"b40db715-280a-448c-bf26-4ac0eb383937","Type":"ContainerStarted","Data":"aad022f1c0f493f889b0007b5dc3cf472a928f9494ce5952c121a9672731b7b9"}
Sep 09 13:48:52.028375 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:52.027913    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl" event={"ID":"b40db715-280a-448c-bf26-4ac0eb383937","Type":"ContainerStarted","Data":"c3e201f4e323c7044223cca921cbc5e3e80d540b760fc086d6647fe49ffc3636"}
Sep 09 13:48:52.028375 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:52.027945    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl"
Sep 09 13:48:52.029667 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:52.029640    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2" event={"ID":"dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15","Type":"ContainerStarted","Data":"d616c436e3a5b30f38f33d15c8ec4cb47ce99de460adb1166070b78ae14afa5a"}
Sep 09 13:48:52.029801 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:52.029684    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2" event={"ID":"dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15","Type":"ContainerStarted","Data":"e814cab942ce544dad13bac7a9480590af7b5462055a39ad827ccfb3bc908456"}
Sep 09 13:48:52.029801 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:52.029715    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2"
Sep 09 13:48:52.045625 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:52.045579    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl" podStartSLOduration=2.045565573 podStartE2EDuration="2.045565573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:48:52.044622221 +0000 UTC m=+1365.404472049" watchObservedRunningTime="2026-09-09 13:48:52.045565573 +0000 UTC m=+1365.405415399"
Sep 09 13:48:52.061570 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:48:52.061533    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2" podStartSLOduration=2.061518945 podStartE2EDuration="2.061518945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:48:52.059263095 +0000 UTC m=+1365.419112920" watchObservedRunningTime="2026-09-09 13:48:52.061518945 +0000 UTC m=+1365.421368770"
Sep 09 13:49:03.038248 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:03.038209    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl"
Sep 09 13:49:03.038777 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:03.038405    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2"
Sep 09 13:49:12.252296 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:12.252262    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"]
Sep 09 13:49:14.434538 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.434506    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"]
Sep 09 13:49:14.438215 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.438181    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:14.440462 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.440435    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4bfe1d-openshift-default-dockercfg-z5bx5\""
Sep 09 13:49:14.452746 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.452720    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"]
Sep 09 13:49:14.542497 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.542463    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istiod-ca-cert\") pod \"e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") " pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:14.542497 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.542501    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-credential-socket\") pod \"e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") " pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:14.542791 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.542597    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istio-envoy\") pod \"e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") " pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:14.542791 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.542625    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istio-podinfo\") pod \"e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") " pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:14.542791 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.542690    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-workload-socket\") pod \"e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") " pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:14.542791 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.542707    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bplnw\" (UniqueName: \"kubernetes.io/projected/0eda8b5c-04c2-42a4-83a6-3432587bbb33-kube-api-access-bplnw\") pod \"e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") " pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:14.542791 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.542733    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istio-token\") pod \"e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") " pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:14.543093 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.542797    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-workload-certs\") pod \"e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") " pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:14.543093 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.542845    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istio-data\") pod \"e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") " pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:14.644097 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.644061    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istio-envoy\") pod \"e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") " pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:14.644308 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.644112    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istio-podinfo\") pod \"e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") " pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:14.644308 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.644169    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-workload-socket\") pod \"e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") " pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:14.644308 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.644196    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bplnw\" (UniqueName: \"kubernetes.io/projected/0eda8b5c-04c2-42a4-83a6-3432587bbb33-kube-api-access-bplnw\") pod \"e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") " pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:14.644308 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.644221    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istio-token\") pod \"e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") " pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:14.644308 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.644279    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-workload-certs\") pod \"e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") " pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:14.644308 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.644302    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istio-data\") pod \"e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") " pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:14.644640 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.644470    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istiod-ca-cert\") pod \"e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") " pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:14.644640 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.644514    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-credential-socket\") pod \"e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") " pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:14.644640 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.644587    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-workload-socket\") pod \"e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") " pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:14.644782 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.644639    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istio-data\") pod \"e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") " pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:14.644782 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.644705    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-workload-certs\") pod \"e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") " pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:14.644954 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.644937    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-credential-socket\") pod \"e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") " pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:14.645286 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.645264    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istiod-ca-cert\") pod \"e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") " pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:14.646648 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.646627    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istio-envoy\") pod \"e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") " pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:14.646849 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.646831    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istio-podinfo\") pod \"e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") " pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:14.652519 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.652500    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istio-token\") pod \"e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") " pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:14.652727 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.652707    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bplnw\" (UniqueName: \"kubernetes.io/projected/0eda8b5c-04c2-42a4-83a6-3432587bbb33-kube-api-access-bplnw\") pod \"e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") " pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:14.749954 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:14.749866    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:15.027372 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:15.027347    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"]
Sep 09 13:49:15.029583 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:49:15.029553    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eda8b5c_04c2_42a4_83a6_3432587bbb33.slice/crio-799a83f542232ea331642e0559e9c9fe333958e33496856caf5e290cba7aa096 WatchSource:0}: Error finding container 799a83f542232ea331642e0559e9c9fe333958e33496856caf5e290cba7aa096: Status 404 returned error can't find the container with id 799a83f542232ea331642e0559e9c9fe333958e33496856caf5e290cba7aa096
Sep 09 13:49:15.031968 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:15.031935    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:49:15.032067 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:15.032002    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:49:15.032067 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:15.032031    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:49:15.155553 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:15.155518    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4" event={"ID":"0eda8b5c-04c2-42a4-83a6-3432587bbb33","Type":"ContainerStarted","Data":"74ce67586c4ef7b14ab36c9fa7a4aa1c65216f1d2d0c63e28743ae30822c0cac"}
Sep 09 13:49:15.155764 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:15.155577    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4" event={"ID":"0eda8b5c-04c2-42a4-83a6-3432587bbb33","Type":"ContainerStarted","Data":"799a83f542232ea331642e0559e9c9fe333958e33496856caf5e290cba7aa096"}
Sep 09 13:49:15.175530 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:15.175477    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4" podStartSLOduration=1.175459622 podStartE2EDuration="1.175459622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:49:15.17358298 +0000 UTC m=+1388.533432806" watchObservedRunningTime="2026-09-09 13:49:15.175459622 +0000 UTC m=+1388.535309449"
Sep 09 13:49:15.750888 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:15.750852    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:16.755794 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:16.755764    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:17.166796 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:17.166762    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:17.167469 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:17.167449    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:27.272016 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:27.271958    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8" podUID="df2b86eb-f905-4134-916b-1d821947a339" containerName="main" containerID="cri-o://f808b217965772c7d37ddd5d42f181e9e2f04488dd7738bc3146b7fbb87e222c" gracePeriod=45
Sep 09 13:49:27.543237 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:27.543213    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"
Sep 09 13:49:27.566096 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:27.566066    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2b86eb-f905-4134-916b-1d821947a339-tmp-dir\") pod \"df2b86eb-f905-4134-916b-1d821947a339\" (UID: \"df2b86eb-f905-4134-916b-1d821947a339\") "
Sep 09 13:49:27.566376 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:27.566359    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df2b86eb-f905-4134-916b-1d821947a339-model-cache\") pod \"df2b86eb-f905-4134-916b-1d821947a339\" (UID: \"df2b86eb-f905-4134-916b-1d821947a339\") "
Sep 09 13:49:27.566695 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:27.566677    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x62nk\" (UniqueName: \"kubernetes.io/projected/df2b86eb-f905-4134-916b-1d821947a339-kube-api-access-x62nk\") pod \"df2b86eb-f905-4134-916b-1d821947a339\" (UID: \"df2b86eb-f905-4134-916b-1d821947a339\") "
Sep 09 13:49:27.567302 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:27.567282    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df2b86eb-f905-4134-916b-1d821947a339-tls-certs\") pod \"df2b86eb-f905-4134-916b-1d821947a339\" (UID: \"df2b86eb-f905-4134-916b-1d821947a339\") "
Sep 09 13:49:27.567470 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:27.567452    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df2b86eb-f905-4134-916b-1d821947a339-dshm\") pod \"df2b86eb-f905-4134-916b-1d821947a339\" (UID: \"df2b86eb-f905-4134-916b-1d821947a339\") "
Sep 09 13:49:27.567591 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:27.567579    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df2b86eb-f905-4134-916b-1d821947a339-home\") pod \"df2b86eb-f905-4134-916b-1d821947a339\" (UID: \"df2b86eb-f905-4134-916b-1d821947a339\") "
Sep 09 13:49:27.568274 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:27.566520    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2b86eb-f905-4134-916b-1d821947a339-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "df2b86eb-f905-4134-916b-1d821947a339" (UID: "df2b86eb-f905-4134-916b-1d821947a339"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:49:27.568397 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:27.566625    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2b86eb-f905-4134-916b-1d821947a339-model-cache" (OuterVolumeSpecName: "model-cache") pod "df2b86eb-f905-4134-916b-1d821947a339" (UID: "df2b86eb-f905-4134-916b-1d821947a339"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:49:27.568493 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:27.568250    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2b86eb-f905-4134-916b-1d821947a339-home" (OuterVolumeSpecName: "home") pod "df2b86eb-f905-4134-916b-1d821947a339" (UID: "df2b86eb-f905-4134-916b-1d821947a339"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:49:27.576070 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:27.575990    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2b86eb-f905-4134-916b-1d821947a339-kube-api-access-x62nk" (OuterVolumeSpecName: "kube-api-access-x62nk") pod "df2b86eb-f905-4134-916b-1d821947a339" (UID: "df2b86eb-f905-4134-916b-1d821947a339"). InnerVolumeSpecName "kube-api-access-x62nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:49:27.576351 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:27.576313    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2b86eb-f905-4134-916b-1d821947a339-dshm" (OuterVolumeSpecName: "dshm") pod "df2b86eb-f905-4134-916b-1d821947a339" (UID: "df2b86eb-f905-4134-916b-1d821947a339"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:49:27.576661 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:27.576608    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df2b86eb-f905-4134-916b-1d821947a339-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "df2b86eb-f905-4134-916b-1d821947a339" (UID: "df2b86eb-f905-4134-916b-1d821947a339"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:49:27.668488 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:27.668445    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df2b86eb-f905-4134-916b-1d821947a339-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:49:27.668488 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:27.668485    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df2b86eb-f905-4134-916b-1d821947a339-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:49:27.668488 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:27.668496    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2b86eb-f905-4134-916b-1d821947a339-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:49:27.668741 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:27.668506    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df2b86eb-f905-4134-916b-1d821947a339-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:49:27.668741 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:27.668520    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x62nk\" (UniqueName: \"kubernetes.io/projected/df2b86eb-f905-4134-916b-1d821947a339-kube-api-access-x62nk\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:49:27.668741 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:27.668530    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df2b86eb-f905-4134-916b-1d821947a339-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:49:28.223905 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:28.223872    2566 generic.go:356] "Generic (PLEG): container finished" podID="df2b86eb-f905-4134-916b-1d821947a339" containerID="f808b217965772c7d37ddd5d42f181e9e2f04488dd7738bc3146b7fbb87e222c" exitCode=0
Sep 09 13:49:28.224111 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:28.223925    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"
Sep 09 13:49:28.224111 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:28.223946    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8" event={"ID":"df2b86eb-f905-4134-916b-1d821947a339","Type":"ContainerDied","Data":"f808b217965772c7d37ddd5d42f181e9e2f04488dd7738bc3146b7fbb87e222c"}
Sep 09 13:49:28.224111 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:28.223998    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8" event={"ID":"df2b86eb-f905-4134-916b-1d821947a339","Type":"ContainerDied","Data":"bf0a3161de112b3536c8cfad13129d55d3138500b9290eadb3db47bdcb3382d6"}
Sep 09 13:49:28.224111 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:28.224020    2566 scope.go:117] "RemoveContainer" containerID="f808b217965772c7d37ddd5d42f181e9e2f04488dd7738bc3146b7fbb87e222c"
Sep 09 13:49:28.233233 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:28.233213    2566 scope.go:117] "RemoveContainer" containerID="f808b217965772c7d37ddd5d42f181e9e2f04488dd7738bc3146b7fbb87e222c"
Sep 09 13:49:28.233490 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:49:28.233471    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f808b217965772c7d37ddd5d42f181e9e2f04488dd7738bc3146b7fbb87e222c\": container with ID starting with f808b217965772c7d37ddd5d42f181e9e2f04488dd7738bc3146b7fbb87e222c not found: ID does not exist" containerID="f808b217965772c7d37ddd5d42f181e9e2f04488dd7738bc3146b7fbb87e222c"
Sep 09 13:49:28.233553 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:28.233501    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f808b217965772c7d37ddd5d42f181e9e2f04488dd7738bc3146b7fbb87e222c"} err="failed to get container status \"f808b217965772c7d37ddd5d42f181e9e2f04488dd7738bc3146b7fbb87e222c\": rpc error: code = NotFound desc = could not find container \"f808b217965772c7d37ddd5d42f181e9e2f04488dd7738bc3146b7fbb87e222c\": container with ID starting with f808b217965772c7d37ddd5d42f181e9e2f04488dd7738bc3146b7fbb87e222c not found: ID does not exist"
Sep 09 13:49:28.245082 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:28.245060    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"]
Sep 09 13:49:28.249541 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:28.249522    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-df48ae/sub-test-model-df48ae-kserve-7c4d8bff55-4pzm8"]
Sep 09 13:49:29.149760 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:29.149726    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df2b86eb-f905-4134-916b-1d821947a339" path="/var/lib/kubelet/pods/df2b86eb-f905-4134-916b-1d821947a339/volumes"
Sep 09 13:49:37.272792 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:37.272757    2566 generic.go:356] "Generic (PLEG): container finished" podID="0eda8b5c-04c2-42a4-83a6-3432587bbb33" containerID="74ce67586c4ef7b14ab36c9fa7a4aa1c65216f1d2d0c63e28743ae30822c0cac" exitCode=0
Sep 09 13:49:37.273246 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:37.272844    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4" event={"ID":"0eda8b5c-04c2-42a4-83a6-3432587bbb33","Type":"ContainerDied","Data":"74ce67586c4ef7b14ab36c9fa7a4aa1c65216f1d2d0c63e28743ae30822c0cac"}
Sep 09 13:49:37.273303 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:37.273286    2566 scope.go:117] "RemoveContainer" containerID="74ce67586c4ef7b14ab36c9fa7a4aa1c65216f1d2d0c63e28743ae30822c0cac"
Sep 09 13:49:37.274456 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:37.274427    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:49:37.274521 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:37.274507    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:49:37.274561 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:37.274543    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:49:38.280167 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.280126    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4" event={"ID":"0eda8b5c-04c2-42a4-83a6-3432587bbb33","Type":"ContainerStarted","Data":"71f8c813f74ec6f0e4f5b8c36a810a4c921cdadbec794880cc1bb39d6eece532"}
Sep 09 13:49:38.563146 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.563057    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"]
Sep 09 13:49:38.563599 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.563577    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df2b86eb-f905-4134-916b-1d821947a339" containerName="main"
Sep 09 13:49:38.563599 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.563602    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="df2b86eb-f905-4134-916b-1d821947a339" containerName="main"
Sep 09 13:49:38.563778 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.563717    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="df2b86eb-f905-4134-916b-1d821947a339" containerName="main"
Sep 09 13:49:38.566916 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.566897    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"
Sep 09 13:49:38.569210 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.569192    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8f2df2\"/\"default-dockercfg-2fx29\""
Sep 09 13:49:38.569338 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.569304    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8f2df2\"/\"rate-test-model-8f2df2-kserve-self-signed-certs\""
Sep 09 13:49:38.569891 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.569874    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-8f2df2\"/\"kube-root-ca.crt\""
Sep 09 13:49:38.569991 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.569882    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-8f2df2\"/\"openshift-service-ca.crt\""
Sep 09 13:49:38.575327 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.575304    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"]
Sep 09 13:49:38.683121 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.683088    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/884129c9-b984-4ed8-bcda-1464a001c9a8-tls-certs\") pod \"rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx\" (UID: \"884129c9-b984-4ed8-bcda-1464a001c9a8\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"
Sep 09 13:49:38.683121 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.683126    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/884129c9-b984-4ed8-bcda-1464a001c9a8-dshm\") pod \"rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx\" (UID: \"884129c9-b984-4ed8-bcda-1464a001c9a8\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"
Sep 09 13:49:38.683338 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.683144    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc6p2\" (UniqueName: \"kubernetes.io/projected/884129c9-b984-4ed8-bcda-1464a001c9a8-kube-api-access-bc6p2\") pod \"rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx\" (UID: \"884129c9-b984-4ed8-bcda-1464a001c9a8\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"
Sep 09 13:49:38.683338 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.683176    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/884129c9-b984-4ed8-bcda-1464a001c9a8-tmp-dir\") pod \"rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx\" (UID: \"884129c9-b984-4ed8-bcda-1464a001c9a8\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"
Sep 09 13:49:38.683338 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.683201    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/884129c9-b984-4ed8-bcda-1464a001c9a8-model-cache\") pod \"rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx\" (UID: \"884129c9-b984-4ed8-bcda-1464a001c9a8\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"
Sep 09 13:49:38.683338 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.683251    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/884129c9-b984-4ed8-bcda-1464a001c9a8-home\") pod \"rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx\" (UID: \"884129c9-b984-4ed8-bcda-1464a001c9a8\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"
Sep 09 13:49:38.750981 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.750939    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:38.751255 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.751231    2566 patch_prober.go:28] interesting pod/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 09 13:49:38.751317 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.751295    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4" podUID="0eda8b5c-04c2-42a4-83a6-3432587bbb33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 09 13:49:38.784382 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.784349    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/884129c9-b984-4ed8-bcda-1464a001c9a8-tmp-dir\") pod \"rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx\" (UID: \"884129c9-b984-4ed8-bcda-1464a001c9a8\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"
Sep 09 13:49:38.784558 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.784391    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/884129c9-b984-4ed8-bcda-1464a001c9a8-model-cache\") pod \"rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx\" (UID: \"884129c9-b984-4ed8-bcda-1464a001c9a8\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"
Sep 09 13:49:38.784558 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.784445    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/884129c9-b984-4ed8-bcda-1464a001c9a8-home\") pod \"rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx\" (UID: \"884129c9-b984-4ed8-bcda-1464a001c9a8\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"
Sep 09 13:49:38.784558 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.784525    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/884129c9-b984-4ed8-bcda-1464a001c9a8-tls-certs\") pod \"rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx\" (UID: \"884129c9-b984-4ed8-bcda-1464a001c9a8\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"
Sep 09 13:49:38.784558 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.784550    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/884129c9-b984-4ed8-bcda-1464a001c9a8-dshm\") pod \"rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx\" (UID: \"884129c9-b984-4ed8-bcda-1464a001c9a8\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"
Sep 09 13:49:38.784715 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.784568    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc6p2\" (UniqueName: \"kubernetes.io/projected/884129c9-b984-4ed8-bcda-1464a001c9a8-kube-api-access-bc6p2\") pod \"rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx\" (UID: \"884129c9-b984-4ed8-bcda-1464a001c9a8\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"
Sep 09 13:49:38.784892 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.784867    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/884129c9-b984-4ed8-bcda-1464a001c9a8-tmp-dir\") pod \"rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx\" (UID: \"884129c9-b984-4ed8-bcda-1464a001c9a8\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"
Sep 09 13:49:38.784949 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.784905    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/884129c9-b984-4ed8-bcda-1464a001c9a8-model-cache\") pod \"rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx\" (UID: \"884129c9-b984-4ed8-bcda-1464a001c9a8\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"
Sep 09 13:49:38.784949 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.784929    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/884129c9-b984-4ed8-bcda-1464a001c9a8-home\") pod \"rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx\" (UID: \"884129c9-b984-4ed8-bcda-1464a001c9a8\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"
Sep 09 13:49:38.786907 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.786890    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/884129c9-b984-4ed8-bcda-1464a001c9a8-dshm\") pod \"rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx\" (UID: \"884129c9-b984-4ed8-bcda-1464a001c9a8\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"
Sep 09 13:49:38.787194 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.787165    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/884129c9-b984-4ed8-bcda-1464a001c9a8-tls-certs\") pod \"rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx\" (UID: \"884129c9-b984-4ed8-bcda-1464a001c9a8\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"
Sep 09 13:49:38.797552 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.797514    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc6p2\" (UniqueName: \"kubernetes.io/projected/884129c9-b984-4ed8-bcda-1464a001c9a8-kube-api-access-bc6p2\") pod \"rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx\" (UID: \"884129c9-b984-4ed8-bcda-1464a001c9a8\") " pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"
Sep 09 13:49:38.877463 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:38.877422    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"
Sep 09 13:49:39.177538 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:39.177465    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"]
Sep 09 13:49:39.180027 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:49:39.180003    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod884129c9_b984_4ed8_bcda_1464a001c9a8.slice/crio-3c9f6371f298dacbfd4963ec4c885e28ea0c1f396d2bacdea0a0e872b933a795 WatchSource:0}: Error finding container 3c9f6371f298dacbfd4963ec4c885e28ea0c1f396d2bacdea0a0e872b933a795: Status 404 returned error can't find the container with id 3c9f6371f298dacbfd4963ec4c885e28ea0c1f396d2bacdea0a0e872b933a795
Sep 09 13:49:39.288419 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:39.288385    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx" event={"ID":"884129c9-b984-4ed8-bcda-1464a001c9a8","Type":"ContainerStarted","Data":"67d1469ffda9c80b2087814de6bc722cf822cdfaa2f23341ae1df98089742ed1"}
Sep 09 13:49:39.288419 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:39.288429    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx" event={"ID":"884129c9-b984-4ed8-bcda-1464a001c9a8","Type":"ContainerStarted","Data":"3c9f6371f298dacbfd4963ec4c885e28ea0c1f396d2bacdea0a0e872b933a795"}
Sep 09 13:49:39.311349 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:39.311284    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx" podStartSLOduration=1.31126139 podStartE2EDuration="1.31126139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:49:39.309092313 +0000 UTC m=+1412.668942140" watchObservedRunningTime="2026-09-09 13:49:39.31126139 +0000 UTC m=+1412.671111217"
Sep 09 13:49:39.751051 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:39.751012    2566 patch_prober.go:28] interesting pod/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 09 13:49:39.751242 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:39.751068    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4" podUID="0eda8b5c-04c2-42a4-83a6-3432587bbb33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 09 13:49:40.751219 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:40.751179    2566 patch_prober.go:28] interesting pod/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 09 13:49:40.751622 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:40.751239    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4" podUID="0eda8b5c-04c2-42a4-83a6-3432587bbb33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 09 13:49:41.751217 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:41.751179    2566 patch_prober.go:28] interesting pod/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 09 13:49:41.751405 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:41.751240    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4" podUID="0eda8b5c-04c2-42a4-83a6-3432587bbb33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 09 13:49:42.750832 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:42.750770    2566 patch_prober.go:28] interesting pod/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 09 13:49:42.751007 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:42.750865    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4" podUID="0eda8b5c-04c2-42a4-83a6-3432587bbb33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 09 13:49:43.754732 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:43.754699    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:43.755143 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:43.754999    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:43.755740 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:43.755720    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:49:44.816601 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:44.816567    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-464c54-256q2"]
Sep 09 13:49:44.820248 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:44.820230    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-464c54-256q2"
Sep 09 13:49:44.822512 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:44.822488    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 13:49:44.822645 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:44.822611    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lh487\""
Sep 09 13:49:44.827808 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:44.827784    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-464c54-256q2"]
Sep 09 13:49:44.965745 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:44.965704    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea9f2cc-898c-4db9-83ff-4a1c14c446fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-464c54-256q2\" (UID: \"5ea9f2cc-898c-4db9-83ff-4a1c14c446fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-464c54-256q2"
Sep 09 13:49:45.066669 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:45.066582    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea9f2cc-898c-4db9-83ff-4a1c14c446fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-464c54-256q2\" (UID: \"5ea9f2cc-898c-4db9-83ff-4a1c14c446fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-464c54-256q2"
Sep 09 13:49:45.067270 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:45.067249    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea9f2cc-898c-4db9-83ff-4a1c14c446fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-464c54-256q2\" (UID: \"5ea9f2cc-898c-4db9-83ff-4a1c14c446fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-464c54-256q2"
Sep 09 13:49:45.133430 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:45.133400    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-464c54-256q2"
Sep 09 13:49:45.405644 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:45.405623    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-464c54-256q2"]
Sep 09 13:49:45.407853 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:49:45.407825    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea9f2cc_898c_4db9_83ff_4a1c14c446fa.slice/crio-dd6959fac994c83575c8d0841c7d56e1f6065361548de429d0349ed2c11557c7 WatchSource:0}: Error finding container dd6959fac994c83575c8d0841c7d56e1f6065361548de429d0349ed2c11557c7: Status 404 returned error can't find the container with id dd6959fac994c83575c8d0841c7d56e1f6065361548de429d0349ed2c11557c7
Sep 09 13:49:46.335022 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:46.334992    2566 generic.go:356] "Generic (PLEG): container finished" podID="5ea9f2cc-898c-4db9-83ff-4a1c14c446fa" containerID="6f05e0e0a283907592716917c65fd31f7ec69b3b3f2957e42a95fe20932ebaa5" exitCode=0
Sep 09 13:49:46.335409 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:46.335076    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-464c54-256q2" event={"ID":"5ea9f2cc-898c-4db9-83ff-4a1c14c446fa","Type":"ContainerDied","Data":"6f05e0e0a283907592716917c65fd31f7ec69b3b3f2957e42a95fe20932ebaa5"}
Sep 09 13:49:46.335409 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:46.335132    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-464c54-256q2" event={"ID":"5ea9f2cc-898c-4db9-83ff-4a1c14c446fa","Type":"ContainerStarted","Data":"dd6959fac994c83575c8d0841c7d56e1f6065361548de429d0349ed2c11557c7"}
Sep 09 13:49:47.485429 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:47.485401    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-464c54-256q2"
Sep 09 13:49:47.595339 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:47.595247    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea9f2cc-898c-4db9-83ff-4a1c14c446fa-maas-api-service-ca\") pod \"5ea9f2cc-898c-4db9-83ff-4a1c14c446fa\" (UID: \"5ea9f2cc-898c-4db9-83ff-4a1c14c446fa\") "
Sep 09 13:49:47.595673 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:47.595648    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea9f2cc-898c-4db9-83ff-4a1c14c446fa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5ea9f2cc-898c-4db9-83ff-4a1c14c446fa" (UID: "5ea9f2cc-898c-4db9-83ff-4a1c14c446fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:49:47.696278 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:47.696241    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea9f2cc-898c-4db9-83ff-4a1c14c446fa-maas-api-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:49:48.346765 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:48.346734    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-464c54-256q2"
Sep 09 13:49:48.346765 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:48.346745    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-464c54-256q2" event={"ID":"5ea9f2cc-898c-4db9-83ff-4a1c14c446fa","Type":"ContainerDied","Data":"dd6959fac994c83575c8d0841c7d56e1f6065361548de429d0349ed2c11557c7"}
Sep 09 13:49:48.347048 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:48.346792    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd6959fac994c83575c8d0841c7d56e1f6065361548de429d0349ed2c11557c7"
Sep 09 13:49:48.878603 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:48.878568    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"
Sep 09 13:49:48.879107 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:48.878623    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"
Sep 09 13:49:48.891458 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:48.891427    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"
Sep 09 13:49:49.363037 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:49:49.363009    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"
Sep 09 13:50:00.618640 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:00.618545    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl"]
Sep 09 13:50:00.619121 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:00.618864    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl" podUID="b40db715-280a-448c-bf26-4ac0eb383937" containerName="payload-processing" containerID="cri-o://aad022f1c0f493f889b0007b5dc3cf472a928f9494ce5952c121a9672731b7b9" gracePeriod=30
Sep 09 13:50:00.630214 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:00.630181    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2"]
Sep 09 13:50:00.630433 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:00.630398    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2" podUID="dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15" containerName="payload-pre-processing" containerID="cri-o://d616c436e3a5b30f38f33d15c8ec4cb47ce99de460adb1166070b78ae14afa5a" gracePeriod=30
Sep 09 13:50:00.802690 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:00.802614    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-464c54-256q2"]
Sep 09 13:50:00.808106 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:00.808055    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-464c54-256q2"]
Sep 09 13:50:00.903057 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:00.903026    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl"
Sep 09 13:50:00.905837 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:00.905801    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2"
Sep 09 13:50:01.010286 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.010228    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6g74j\" (UniqueName: \"kubernetes.io/projected/b40db715-280a-448c-bf26-4ac0eb383937-kube-api-access-6g74j\") pod \"b40db715-280a-448c-bf26-4ac0eb383937\" (UID: \"b40db715-280a-448c-bf26-4ac0eb383937\") "
Sep 09 13:50:01.010286 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.010276    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b40db715-280a-448c-bf26-4ac0eb383937-plugins-config-volume\") pod \"b40db715-280a-448c-bf26-4ac0eb383937\" (UID: \"b40db715-280a-448c-bf26-4ac0eb383937\") "
Sep 09 13:50:01.010286 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.010298    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15-plugins-config-volume\") pod \"dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15\" (UID: \"dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15\") "
Sep 09 13:50:01.010594 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.010325    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vk2n7\" (UniqueName: \"kubernetes.io/projected/dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15-kube-api-access-vk2n7\") pod \"dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15\" (UID: \"dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15\") "
Sep 09 13:50:01.010682 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.010642    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b40db715-280a-448c-bf26-4ac0eb383937-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b40db715-280a-448c-bf26-4ac0eb383937" (UID: "b40db715-280a-448c-bf26-4ac0eb383937"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:50:01.010783 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.010744    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15" (UID: "dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:50:01.012409 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.012379    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15-kube-api-access-vk2n7" (OuterVolumeSpecName: "kube-api-access-vk2n7") pod "dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15" (UID: "dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15"). InnerVolumeSpecName "kube-api-access-vk2n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:50:01.012409 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.012407    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b40db715-280a-448c-bf26-4ac0eb383937-kube-api-access-6g74j" (OuterVolumeSpecName: "kube-api-access-6g74j") pod "b40db715-280a-448c-bf26-4ac0eb383937" (UID: "b40db715-280a-448c-bf26-4ac0eb383937"). InnerVolumeSpecName "kube-api-access-6g74j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:50:01.111382 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.111341    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g74j\" (UniqueName: \"kubernetes.io/projected/b40db715-280a-448c-bf26-4ac0eb383937-kube-api-access-6g74j\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:50:01.111382 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.111375    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b40db715-280a-448c-bf26-4ac0eb383937-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:50:01.111568 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.111389    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:50:01.111568 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.111405    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk2n7\" (UniqueName: \"kubernetes.io/projected/dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15-kube-api-access-vk2n7\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:50:01.149942 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.149847    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ea9f2cc-898c-4db9-83ff-4a1c14c446fa" path="/var/lib/kubelet/pods/5ea9f2cc-898c-4db9-83ff-4a1c14c446fa/volumes"
Sep 09 13:50:01.425258 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.425172    2566 generic.go:356] "Generic (PLEG): container finished" podID="dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15" containerID="d616c436e3a5b30f38f33d15c8ec4cb47ce99de460adb1166070b78ae14afa5a" exitCode=0
Sep 09 13:50:01.425258 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.425213    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2"
Sep 09 13:50:01.425500 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.425256    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2" event={"ID":"dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15","Type":"ContainerDied","Data":"d616c436e3a5b30f38f33d15c8ec4cb47ce99de460adb1166070b78ae14afa5a"}
Sep 09 13:50:01.425500 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.425312    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2" event={"ID":"dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15","Type":"ContainerDied","Data":"e814cab942ce544dad13bac7a9480590af7b5462055a39ad827ccfb3bc908456"}
Sep 09 13:50:01.425500 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.425339    2566 scope.go:117] "RemoveContainer" containerID="d616c436e3a5b30f38f33d15c8ec4cb47ce99de460adb1166070b78ae14afa5a"
Sep 09 13:50:01.426702 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.426680    2566 generic.go:356] "Generic (PLEG): container finished" podID="b40db715-280a-448c-bf26-4ac0eb383937" containerID="aad022f1c0f493f889b0007b5dc3cf472a928f9494ce5952c121a9672731b7b9" exitCode=0
Sep 09 13:50:01.426824 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.426723    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl"
Sep 09 13:50:01.426824 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.426753    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl" event={"ID":"b40db715-280a-448c-bf26-4ac0eb383937","Type":"ContainerDied","Data":"aad022f1c0f493f889b0007b5dc3cf472a928f9494ce5952c121a9672731b7b9"}
Sep 09 13:50:01.426824 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.426796    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl" event={"ID":"b40db715-280a-448c-bf26-4ac0eb383937","Type":"ContainerDied","Data":"c3e201f4e323c7044223cca921cbc5e3e80d540b760fc086d6647fe49ffc3636"}
Sep 09 13:50:01.434215 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.434191    2566 scope.go:117] "RemoveContainer" containerID="d616c436e3a5b30f38f33d15c8ec4cb47ce99de460adb1166070b78ae14afa5a"
Sep 09 13:50:01.434564 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:50:01.434545    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d616c436e3a5b30f38f33d15c8ec4cb47ce99de460adb1166070b78ae14afa5a\": container with ID starting with d616c436e3a5b30f38f33d15c8ec4cb47ce99de460adb1166070b78ae14afa5a not found: ID does not exist" containerID="d616c436e3a5b30f38f33d15c8ec4cb47ce99de460adb1166070b78ae14afa5a"
Sep 09 13:50:01.434628 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.434572    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d616c436e3a5b30f38f33d15c8ec4cb47ce99de460adb1166070b78ae14afa5a"} err="failed to get container status \"d616c436e3a5b30f38f33d15c8ec4cb47ce99de460adb1166070b78ae14afa5a\": rpc error: code = NotFound desc = could not find container \"d616c436e3a5b30f38f33d15c8ec4cb47ce99de460adb1166070b78ae14afa5a\": container with ID starting with d616c436e3a5b30f38f33d15c8ec4cb47ce99de460adb1166070b78ae14afa5a not found: ID does not exist"
Sep 09 13:50:01.434628 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.434588    2566 scope.go:117] "RemoveContainer" containerID="aad022f1c0f493f889b0007b5dc3cf472a928f9494ce5952c121a9672731b7b9"
Sep 09 13:50:01.442254 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.442220    2566 scope.go:117] "RemoveContainer" containerID="aad022f1c0f493f889b0007b5dc3cf472a928f9494ce5952c121a9672731b7b9"
Sep 09 13:50:01.442702 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:50:01.442672    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aad022f1c0f493f889b0007b5dc3cf472a928f9494ce5952c121a9672731b7b9\": container with ID starting with aad022f1c0f493f889b0007b5dc3cf472a928f9494ce5952c121a9672731b7b9 not found: ID does not exist" containerID="aad022f1c0f493f889b0007b5dc3cf472a928f9494ce5952c121a9672731b7b9"
Sep 09 13:50:01.442826 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.442709    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aad022f1c0f493f889b0007b5dc3cf472a928f9494ce5952c121a9672731b7b9"} err="failed to get container status \"aad022f1c0f493f889b0007b5dc3cf472a928f9494ce5952c121a9672731b7b9\": rpc error: code = NotFound desc = could not find container \"aad022f1c0f493f889b0007b5dc3cf472a928f9494ce5952c121a9672731b7b9\": container with ID starting with aad022f1c0f493f889b0007b5dc3cf472a928f9494ce5952c121a9672731b7b9 not found: ID does not exist"
Sep 09 13:50:01.444799 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.444780    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl"]
Sep 09 13:50:01.448361 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.448343    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-464c54-674979dfc9-bxqrl"]
Sep 09 13:50:01.458991 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.458972    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2"]
Sep 09 13:50:01.462827 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:01.462792    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-464c54-796b584bcd-qjlx2"]
Sep 09 13:50:03.150011 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:03.149978    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b40db715-280a-448c-bf26-4ac0eb383937" path="/var/lib/kubelet/pods/b40db715-280a-448c-bf26-4ac0eb383937/volumes"
Sep 09 13:50:03.150421 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:03.150406    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15" path="/var/lib/kubelet/pods/dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15/volumes"
Sep 09 13:50:07.395863 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:07.395800    2566 scope.go:117] "RemoveContainer" containerID="6a456925b31022ae28df8f595325ea9c9c3a8ff147de60e50176eb6bea0ee975"
Sep 09 13:50:07.404082 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:07.404060    2566 scope.go:117] "RemoveContainer" containerID="e0a1b88a858abb2c2d3ddaa9c7e455567a50eb636fe77e2a89e5b71b7fd06855"
Sep 09 13:50:07.411449 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:07.411420    2566 scope.go:117] "RemoveContainer" containerID="9bd4f85a5fed27b0682992fd8459354f39929add86c323800a43743f4f0c46eb"
Sep 09 13:50:07.418992 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:07.418974    2566 scope.go:117] "RemoveContainer" containerID="756469308d0fc4b64bfca2896196b97b690c063ec536516e2d885c1bc7fac8c1"
Sep 09 13:50:07.799037 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:07.798957    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bfe1d-vg7sr"]
Sep 09 13:50:07.799334 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:07.799321    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15" containerName="payload-pre-processing"
Sep 09 13:50:07.799378 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:07.799336    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15" containerName="payload-pre-processing"
Sep 09 13:50:07.799378 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:07.799348    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ea9f2cc-898c-4db9-83ff-4a1c14c446fa" containerName="revoke-keys"
Sep 09 13:50:07.799378 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:07.799353    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ea9f2cc-898c-4db9-83ff-4a1c14c446fa" containerName="revoke-keys"
Sep 09 13:50:07.799378 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:07.799363    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b40db715-280a-448c-bf26-4ac0eb383937" containerName="payload-processing"
Sep 09 13:50:07.799378 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:07.799370    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b40db715-280a-448c-bf26-4ac0eb383937" containerName="payload-processing"
Sep 09 13:50:07.799570 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:07.799441    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfde75b2-2dd1-41d1-a8cd-8bb22ee20b15" containerName="payload-pre-processing"
Sep 09 13:50:07.799570 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:07.799449    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b40db715-280a-448c-bf26-4ac0eb383937" containerName="payload-processing"
Sep 09 13:50:07.799570 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:07.799456    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ea9f2cc-898c-4db9-83ff-4a1c14c446fa" containerName="revoke-keys"
Sep 09 13:50:07.804340 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:07.804319    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bfe1d-vg7sr"
Sep 09 13:50:07.806578 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:07.806558    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 13:50:07.806672 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:07.806610    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lh487\""
Sep 09 13:50:07.810567 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:07.810546    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bfe1d-vg7sr"]
Sep 09 13:50:07.877150 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:07.877116    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaad5e3b-299a-493a-be30-5dc7e6b55b27-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4bfe1d-vg7sr\" (UID: \"eaad5e3b-299a-493a-be30-5dc7e6b55b27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bfe1d-vg7sr"
Sep 09 13:50:07.977928 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:07.977894    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaad5e3b-299a-493a-be30-5dc7e6b55b27-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4bfe1d-vg7sr\" (UID: \"eaad5e3b-299a-493a-be30-5dc7e6b55b27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bfe1d-vg7sr"
Sep 09 13:50:07.978554 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:07.978529    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaad5e3b-299a-493a-be30-5dc7e6b55b27-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4bfe1d-vg7sr\" (UID: \"eaad5e3b-299a-493a-be30-5dc7e6b55b27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bfe1d-vg7sr"
Sep 09 13:50:08.115927 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:08.115895    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bfe1d-vg7sr"
Sep 09 13:50:08.386131 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:08.386109    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bfe1d-vg7sr"]
Sep 09 13:50:08.387904 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:50:08.387879    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaad5e3b_299a_493a_be30_5dc7e6b55b27.slice/crio-c4a9f93b1747021a52b8669b0fc0560ea8e6b786224676ca3230b9dc3fa4674a WatchSource:0}: Error finding container c4a9f93b1747021a52b8669b0fc0560ea8e6b786224676ca3230b9dc3fa4674a: Status 404 returned error can't find the container with id c4a9f93b1747021a52b8669b0fc0560ea8e6b786224676ca3230b9dc3fa4674a
Sep 09 13:50:08.470865 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:08.470807    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bfe1d-vg7sr" event={"ID":"eaad5e3b-299a-493a-be30-5dc7e6b55b27","Type":"ContainerStarted","Data":"c4a9f93b1747021a52b8669b0fc0560ea8e6b786224676ca3230b9dc3fa4674a"}
Sep 09 13:50:09.477160 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:09.477071    2566 generic.go:356] "Generic (PLEG): container finished" podID="eaad5e3b-299a-493a-be30-5dc7e6b55b27" containerID="b84f114f9f79b226d38b1601decccc65333c79999f08b9ee21b64009e98b23d4" exitCode=0
Sep 09 13:50:09.477504 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:09.477161    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bfe1d-vg7sr" event={"ID":"eaad5e3b-299a-493a-be30-5dc7e6b55b27","Type":"ContainerDied","Data":"b84f114f9f79b226d38b1601decccc65333c79999f08b9ee21b64009e98b23d4"}
Sep 09 13:50:10.458769 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.458734    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"]
Sep 09 13:50:10.462038 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.462020    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"
Sep 09 13:50:10.465305 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.465282    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-df48ae\"/\"default-dockercfg-cf6vl\""
Sep 09 13:50:10.465423 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.465303    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-df48ae\"/\"openshift-service-ca.crt\""
Sep 09 13:50:10.465423 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.465307    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-df48ae\"/\"rate-test-model-df48ae-kserve-self-signed-certs\""
Sep 09 13:50:10.465423 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.465364    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-df48ae\"/\"kube-root-ca.crt\""
Sep 09 13:50:10.473384 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.473363    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"]
Sep 09 13:50:10.503925 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.503893    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55d41a13-de5d-463f-bded-619051df6fdc-model-cache\") pod \"rate-test-model-df48ae-kserve-84f65749ff-fcjxx\" (UID: \"55d41a13-de5d-463f-bded-619051df6fdc\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"
Sep 09 13:50:10.504307 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.503995    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z54sd\" (UniqueName: \"kubernetes.io/projected/55d41a13-de5d-463f-bded-619051df6fdc-kube-api-access-z54sd\") pod \"rate-test-model-df48ae-kserve-84f65749ff-fcjxx\" (UID: \"55d41a13-de5d-463f-bded-619051df6fdc\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"
Sep 09 13:50:10.504307 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.504166    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55d41a13-de5d-463f-bded-619051df6fdc-home\") pod \"rate-test-model-df48ae-kserve-84f65749ff-fcjxx\" (UID: \"55d41a13-de5d-463f-bded-619051df6fdc\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"
Sep 09 13:50:10.504307 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.504204    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55d41a13-de5d-463f-bded-619051df6fdc-dshm\") pod \"rate-test-model-df48ae-kserve-84f65749ff-fcjxx\" (UID: \"55d41a13-de5d-463f-bded-619051df6fdc\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"
Sep 09 13:50:10.504453 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.504317    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55d41a13-de5d-463f-bded-619051df6fdc-tls-certs\") pod \"rate-test-model-df48ae-kserve-84f65749ff-fcjxx\" (UID: \"55d41a13-de5d-463f-bded-619051df6fdc\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"
Sep 09 13:50:10.504453 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.504365    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55d41a13-de5d-463f-bded-619051df6fdc-tmp-dir\") pod \"rate-test-model-df48ae-kserve-84f65749ff-fcjxx\" (UID: \"55d41a13-de5d-463f-bded-619051df6fdc\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"
Sep 09 13:50:10.605584 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.605552    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z54sd\" (UniqueName: \"kubernetes.io/projected/55d41a13-de5d-463f-bded-619051df6fdc-kube-api-access-z54sd\") pod \"rate-test-model-df48ae-kserve-84f65749ff-fcjxx\" (UID: \"55d41a13-de5d-463f-bded-619051df6fdc\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"
Sep 09 13:50:10.605732 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.605635    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55d41a13-de5d-463f-bded-619051df6fdc-home\") pod \"rate-test-model-df48ae-kserve-84f65749ff-fcjxx\" (UID: \"55d41a13-de5d-463f-bded-619051df6fdc\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"
Sep 09 13:50:10.605732 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.605666    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55d41a13-de5d-463f-bded-619051df6fdc-dshm\") pod \"rate-test-model-df48ae-kserve-84f65749ff-fcjxx\" (UID: \"55d41a13-de5d-463f-bded-619051df6fdc\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"
Sep 09 13:50:10.605732 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.605707    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55d41a13-de5d-463f-bded-619051df6fdc-tls-certs\") pod \"rate-test-model-df48ae-kserve-84f65749ff-fcjxx\" (UID: \"55d41a13-de5d-463f-bded-619051df6fdc\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"
Sep 09 13:50:10.605943 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.605734    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55d41a13-de5d-463f-bded-619051df6fdc-tmp-dir\") pod \"rate-test-model-df48ae-kserve-84f65749ff-fcjxx\" (UID: \"55d41a13-de5d-463f-bded-619051df6fdc\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"
Sep 09 13:50:10.605943 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.605786    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55d41a13-de5d-463f-bded-619051df6fdc-model-cache\") pod \"rate-test-model-df48ae-kserve-84f65749ff-fcjxx\" (UID: \"55d41a13-de5d-463f-bded-619051df6fdc\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"
Sep 09 13:50:10.606053 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.606028    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55d41a13-de5d-463f-bded-619051df6fdc-home\") pod \"rate-test-model-df48ae-kserve-84f65749ff-fcjxx\" (UID: \"55d41a13-de5d-463f-bded-619051df6fdc\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"
Sep 09 13:50:10.606108 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.606089    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55d41a13-de5d-463f-bded-619051df6fdc-model-cache\") pod \"rate-test-model-df48ae-kserve-84f65749ff-fcjxx\" (UID: \"55d41a13-de5d-463f-bded-619051df6fdc\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"
Sep 09 13:50:10.606208 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.606174    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55d41a13-de5d-463f-bded-619051df6fdc-tmp-dir\") pod \"rate-test-model-df48ae-kserve-84f65749ff-fcjxx\" (UID: \"55d41a13-de5d-463f-bded-619051df6fdc\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"
Sep 09 13:50:10.607972 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.607951    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55d41a13-de5d-463f-bded-619051df6fdc-dshm\") pod \"rate-test-model-df48ae-kserve-84f65749ff-fcjxx\" (UID: \"55d41a13-de5d-463f-bded-619051df6fdc\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"
Sep 09 13:50:10.608198 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.608180    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55d41a13-de5d-463f-bded-619051df6fdc-tls-certs\") pod \"rate-test-model-df48ae-kserve-84f65749ff-fcjxx\" (UID: \"55d41a13-de5d-463f-bded-619051df6fdc\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"
Sep 09 13:50:10.614115 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.614097    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z54sd\" (UniqueName: \"kubernetes.io/projected/55d41a13-de5d-463f-bded-619051df6fdc-kube-api-access-z54sd\") pod \"rate-test-model-df48ae-kserve-84f65749ff-fcjxx\" (UID: \"55d41a13-de5d-463f-bded-619051df6fdc\") " pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"
Sep 09 13:50:10.644169 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.644143    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bfe1d-vg7sr"
Sep 09 13:50:10.706782 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.706750    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaad5e3b-299a-493a-be30-5dc7e6b55b27-maas-api-service-ca\") pod \"eaad5e3b-299a-493a-be30-5dc7e6b55b27\" (UID: \"eaad5e3b-299a-493a-be30-5dc7e6b55b27\") "
Sep 09 13:50:10.707169 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.707150    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaad5e3b-299a-493a-be30-5dc7e6b55b27-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eaad5e3b-299a-493a-be30-5dc7e6b55b27" (UID: "eaad5e3b-299a-493a-be30-5dc7e6b55b27"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:50:10.772512 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.772431    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"
Sep 09 13:50:10.808551 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:10.808522    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaad5e3b-299a-493a-be30-5dc7e6b55b27-maas-api-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:50:11.043262 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:11.043238    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"]
Sep 09 13:50:11.045352 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:50:11.045321    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55d41a13_de5d_463f_bded_619051df6fdc.slice/crio-bc4b65aaee7edf69130fb28e3545bd7bdfbe2df3d07cf8d10bdf4a82f6f40940 WatchSource:0}: Error finding container bc4b65aaee7edf69130fb28e3545bd7bdfbe2df3d07cf8d10bdf4a82f6f40940: Status 404 returned error can't find the container with id bc4b65aaee7edf69130fb28e3545bd7bdfbe2df3d07cf8d10bdf4a82f6f40940
Sep 09 13:50:11.497612 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:11.497582    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bfe1d-vg7sr"
Sep 09 13:50:11.497806 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:11.497620    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bfe1d-vg7sr" event={"ID":"eaad5e3b-299a-493a-be30-5dc7e6b55b27","Type":"ContainerDied","Data":"c4a9f93b1747021a52b8669b0fc0560ea8e6b786224676ca3230b9dc3fa4674a"}
Sep 09 13:50:11.497806 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:11.497678    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4a9f93b1747021a52b8669b0fc0560ea8e6b786224676ca3230b9dc3fa4674a"
Sep 09 13:50:11.499410 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:11.499388    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx" event={"ID":"55d41a13-de5d-463f-bded-619051df6fdc","Type":"ContainerStarted","Data":"ec3aa1dc4e3622284bbdc6ff3734dedad20ef1aa61de4c55383f819b64d80b08"}
Sep 09 13:50:11.499547 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:11.499423    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx" event={"ID":"55d41a13-de5d-463f-bded-619051df6fdc","Type":"ContainerStarted","Data":"bc4b65aaee7edf69130fb28e3545bd7bdfbe2df3d07cf8d10bdf4a82f6f40940"}
Sep 09 13:50:11.517660 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:11.517620    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx" podStartSLOduration=1.5176080870000002 podStartE2EDuration="1.517608087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:50:11.516281115 +0000 UTC m=+1444.876130942" watchObservedRunningTime="2026-09-09 13:50:11.517608087 +0000 UTC m=+1444.877457914"
Sep 09 13:50:20.773267 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:20.773226    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"
Sep 09 13:50:20.773669 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:20.773278    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"
Sep 09 13:50:20.785938 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:20.785908    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"
Sep 09 13:50:21.561226 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:21.561198    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"
Sep 09 13:50:23.565965 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:23.565930    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bfe1d-vg7sr"]
Sep 09 13:50:23.568321 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:23.568297    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bfe1d-vg7sr"]
Sep 09 13:50:25.150296 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:25.150257    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaad5e3b-299a-493a-be30-5dc7e6b55b27" path="/var/lib/kubelet/pods/eaad5e3b-299a-493a-be30-5dc7e6b55b27/volumes"
Sep 09 13:50:30.329940 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:30.329907    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"]
Sep 09 13:50:30.330394 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:30.330145    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4" podUID="0eda8b5c-04c2-42a4-83a6-3432587bbb33" containerName="istio-proxy" containerID="cri-o://71f8c813f74ec6f0e4f5b8c36a810a4c921cdadbec794880cc1bb39d6eece532" gracePeriod=30
Sep 09 13:50:30.690718 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:30.690633    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b5cba4-n7f5r"]
Sep 09 13:50:30.691024 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:30.691011    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaad5e3b-299a-493a-be30-5dc7e6b55b27" containerName="revoke-keys"
Sep 09 13:50:30.691082 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:30.691026    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaad5e3b-299a-493a-be30-5dc7e6b55b27" containerName="revoke-keys"
Sep 09 13:50:30.691120 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:30.691098    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaad5e3b-299a-493a-be30-5dc7e6b55b27" containerName="revoke-keys"
Sep 09 13:50:30.695148 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:30.695132    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b5cba4-n7f5r"
Sep 09 13:50:30.697918 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:30.697890    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 13:50:30.697918 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:30.697891    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lh487\""
Sep 09 13:50:30.700649 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:30.700618    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b5cba4-n7f5r"]
Sep 09 13:50:30.806711 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:30.806679    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ec8b0b-35b0-44bc-8d46-7c11ab76aad4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b5cba4-n7f5r\" (UID: \"f2ec8b0b-35b0-44bc-8d46-7c11ab76aad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b5cba4-n7f5r"
Sep 09 13:50:30.907957 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:30.907920    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ec8b0b-35b0-44bc-8d46-7c11ab76aad4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b5cba4-n7f5r\" (UID: \"f2ec8b0b-35b0-44bc-8d46-7c11ab76aad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b5cba4-n7f5r"
Sep 09 13:50:30.908597 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:30.908563    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ec8b0b-35b0-44bc-8d46-7c11ab76aad4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b5cba4-n7f5r\" (UID: \"f2ec8b0b-35b0-44bc-8d46-7c11ab76aad4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b5cba4-n7f5r"
Sep 09 13:50:31.008559 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:31.008481    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b5cba4-n7f5r"
Sep 09 13:50:31.278887 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:31.278852    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b5cba4-n7f5r"]
Sep 09 13:50:31.280315 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:50:31.280288    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ec8b0b_35b0_44bc_8d46_7c11ab76aad4.slice/crio-0e63bd9c429205b24bccbb4bcc0fd1f8709a5600e1c2dcff6b3c8d1db0ab2255 WatchSource:0}: Error finding container 0e63bd9c429205b24bccbb4bcc0fd1f8709a5600e1c2dcff6b3c8d1db0ab2255: Status 404 returned error can't find the container with id 0e63bd9c429205b24bccbb4bcc0fd1f8709a5600e1c2dcff6b3c8d1db0ab2255
Sep 09 13:50:31.603542 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:31.603507    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b5cba4-n7f5r" event={"ID":"f2ec8b0b-35b0-44bc-8d46-7c11ab76aad4","Type":"ContainerStarted","Data":"544eef58c858872b65a835154c94153008af759c72a366236bf7cf461de6fe55"}
Sep 09 13:50:31.604023 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:31.603568    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b5cba4-n7f5r" event={"ID":"f2ec8b0b-35b0-44bc-8d46-7c11ab76aad4","Type":"ContainerStarted","Data":"0e63bd9c429205b24bccbb4bcc0fd1f8709a5600e1c2dcff6b3c8d1db0ab2255"}
Sep 09 13:50:31.619128 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:31.619083    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b5cba4-n7f5r" podStartSLOduration=1.619067698 podStartE2EDuration="1.619067698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:50:31.61745728 +0000 UTC m=+1464.977307107" watchObservedRunningTime="2026-09-09 13:50:31.619067698 +0000 UTC m=+1464.978917523"
Sep 09 13:50:32.609869 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:32.609829    2566 generic.go:356] "Generic (PLEG): container finished" podID="f2ec8b0b-35b0-44bc-8d46-7c11ab76aad4" containerID="544eef58c858872b65a835154c94153008af759c72a366236bf7cf461de6fe55" exitCode=0
Sep 09 13:50:32.609869 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:32.609863    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b5cba4-n7f5r" event={"ID":"f2ec8b0b-35b0-44bc-8d46-7c11ab76aad4","Type":"ContainerDied","Data":"544eef58c858872b65a835154c94153008af759c72a366236bf7cf461de6fe55"}
Sep 09 13:50:33.748251 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:33.748225    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b5cba4-n7f5r"
Sep 09 13:50:33.832858 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:33.832831    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ec8b0b-35b0-44bc-8d46-7c11ab76aad4-maas-api-service-ca\") pod \"f2ec8b0b-35b0-44bc-8d46-7c11ab76aad4\" (UID: \"f2ec8b0b-35b0-44bc-8d46-7c11ab76aad4\") "
Sep 09 13:50:33.833172 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:33.833152    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ec8b0b-35b0-44bc-8d46-7c11ab76aad4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f2ec8b0b-35b0-44bc-8d46-7c11ab76aad4" (UID: "f2ec8b0b-35b0-44bc-8d46-7c11ab76aad4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:50:33.933832 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:33.933741    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ec8b0b-35b0-44bc-8d46-7c11ab76aad4-maas-api-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:50:34.621215 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:34.621187    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b5cba4-n7f5r"
Sep 09 13:50:34.621387 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:34.621233    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b5cba4-n7f5r" event={"ID":"f2ec8b0b-35b0-44bc-8d46-7c11ab76aad4","Type":"ContainerDied","Data":"0e63bd9c429205b24bccbb4bcc0fd1f8709a5600e1c2dcff6b3c8d1db0ab2255"}
Sep 09 13:50:34.621387 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:34.621278    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e63bd9c429205b24bccbb4bcc0fd1f8709a5600e1c2dcff6b3c8d1db0ab2255"
Sep 09 13:50:35.602209 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.602142    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:50:35.629392 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.629356    2566 generic.go:356] "Generic (PLEG): container finished" podID="0eda8b5c-04c2-42a4-83a6-3432587bbb33" containerID="71f8c813f74ec6f0e4f5b8c36a810a4c921cdadbec794880cc1bb39d6eece532" exitCode=0
Sep 09 13:50:35.629533 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.629396    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4" event={"ID":"0eda8b5c-04c2-42a4-83a6-3432587bbb33","Type":"ContainerDied","Data":"71f8c813f74ec6f0e4f5b8c36a810a4c921cdadbec794880cc1bb39d6eece532"}
Sep 09 13:50:35.629533 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.629466    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4" event={"ID":"0eda8b5c-04c2-42a4-83a6-3432587bbb33","Type":"ContainerDied","Data":"799a83f542232ea331642e0559e9c9fe333958e33496856caf5e290cba7aa096"}
Sep 09 13:50:35.629533 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.629408    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"
Sep 09 13:50:35.629533 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.629496    2566 scope.go:117] "RemoveContainer" containerID="71f8c813f74ec6f0e4f5b8c36a810a4c921cdadbec794880cc1bb39d6eece532"
Sep 09 13:50:35.638693 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.638670    2566 scope.go:117] "RemoveContainer" containerID="74ce67586c4ef7b14ab36c9fa7a4aa1c65216f1d2d0c63e28743ae30822c0cac"
Sep 09 13:50:35.647673 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.647638    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istio-podinfo\") pod \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") "
Sep 09 13:50:35.647807 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.647684    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istiod-ca-cert\") pod \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") "
Sep 09 13:50:35.647807 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.647710    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bplnw\" (UniqueName: \"kubernetes.io/projected/0eda8b5c-04c2-42a4-83a6-3432587bbb33-kube-api-access-bplnw\") pod \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") "
Sep 09 13:50:35.647807 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.647748    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-workload-certs\") pod \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") "
Sep 09 13:50:35.647807 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.647775    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istio-envoy\") pod \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") "
Sep 09 13:50:35.647807 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.647795    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istio-token\") pod \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") "
Sep 09 13:50:35.648096 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.647874    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-credential-socket\") pod \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") "
Sep 09 13:50:35.648096 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.647919    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istio-data\") pod \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") "
Sep 09 13:50:35.648096 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.647947    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-workload-socket\") pod \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\" (UID: \"0eda8b5c-04c2-42a4-83a6-3432587bbb33\") "
Sep 09 13:50:35.648513 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.648443    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0eda8b5c-04c2-42a4-83a6-3432587bbb33" (UID: "0eda8b5c-04c2-42a4-83a6-3432587bbb33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:50:35.648605 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.648543    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0eda8b5c-04c2-42a4-83a6-3432587bbb33" (UID: "0eda8b5c-04c2-42a4-83a6-3432587bbb33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:50:35.649450 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.649410    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0eda8b5c-04c2-42a4-83a6-3432587bbb33" (UID: "0eda8b5c-04c2-42a4-83a6-3432587bbb33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:50:35.649649 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.649553    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0eda8b5c-04c2-42a4-83a6-3432587bbb33" (UID: "0eda8b5c-04c2-42a4-83a6-3432587bbb33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:50:35.649801 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.649742    2566 scope.go:117] "RemoveContainer" containerID="71f8c813f74ec6f0e4f5b8c36a810a4c921cdadbec794880cc1bb39d6eece532"
Sep 09 13:50:35.650041 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.650013    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istio-data" (OuterVolumeSpecName: "istio-data") pod "0eda8b5c-04c2-42a4-83a6-3432587bbb33" (UID: "0eda8b5c-04c2-42a4-83a6-3432587bbb33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:50:35.650394 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:50:35.650351    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71f8c813f74ec6f0e4f5b8c36a810a4c921cdadbec794880cc1bb39d6eece532\": container with ID starting with 71f8c813f74ec6f0e4f5b8c36a810a4c921cdadbec794880cc1bb39d6eece532 not found: ID does not exist" containerID="71f8c813f74ec6f0e4f5b8c36a810a4c921cdadbec794880cc1bb39d6eece532"
Sep 09 13:50:35.650504 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.650390    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71f8c813f74ec6f0e4f5b8c36a810a4c921cdadbec794880cc1bb39d6eece532"} err="failed to get container status \"71f8c813f74ec6f0e4f5b8c36a810a4c921cdadbec794880cc1bb39d6eece532\": rpc error: code = NotFound desc = could not find container \"71f8c813f74ec6f0e4f5b8c36a810a4c921cdadbec794880cc1bb39d6eece532\": container with ID starting with 71f8c813f74ec6f0e4f5b8c36a810a4c921cdadbec794880cc1bb39d6eece532 not found: ID does not exist"
Sep 09 13:50:35.650504 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.650418    2566 scope.go:117] "RemoveContainer" containerID="74ce67586c4ef7b14ab36c9fa7a4aa1c65216f1d2d0c63e28743ae30822c0cac"
Sep 09 13:50:35.650957 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:50:35.650922    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74ce67586c4ef7b14ab36c9fa7a4aa1c65216f1d2d0c63e28743ae30822c0cac\": container with ID starting with 74ce67586c4ef7b14ab36c9fa7a4aa1c65216f1d2d0c63e28743ae30822c0cac not found: ID does not exist" containerID="74ce67586c4ef7b14ab36c9fa7a4aa1c65216f1d2d0c63e28743ae30822c0cac"
Sep 09 13:50:35.651041 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.650959    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74ce67586c4ef7b14ab36c9fa7a4aa1c65216f1d2d0c63e28743ae30822c0cac"} err="failed to get container status \"74ce67586c4ef7b14ab36c9fa7a4aa1c65216f1d2d0c63e28743ae30822c0cac\": rpc error: code = NotFound desc = could not find container \"74ce67586c4ef7b14ab36c9fa7a4aa1c65216f1d2d0c63e28743ae30822c0cac\": container with ID starting with 74ce67586c4ef7b14ab36c9fa7a4aa1c65216f1d2d0c63e28743ae30822c0cac not found: ID does not exist"
Sep 09 13:50:35.651113 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.651084    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0eda8b5c-04c2-42a4-83a6-3432587bbb33" (UID: "0eda8b5c-04c2-42a4-83a6-3432587bbb33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 13:50:35.651169 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.651084    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0eda8b5c-04c2-42a4-83a6-3432587bbb33" (UID: "0eda8b5c-04c2-42a4-83a6-3432587bbb33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:50:35.651891 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.651873    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eda8b5c-04c2-42a4-83a6-3432587bbb33-kube-api-access-bplnw" (OuterVolumeSpecName: "kube-api-access-bplnw") pod "0eda8b5c-04c2-42a4-83a6-3432587bbb33" (UID: "0eda8b5c-04c2-42a4-83a6-3432587bbb33"). InnerVolumeSpecName "kube-api-access-bplnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:50:35.652250 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.652230    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istio-token" (OuterVolumeSpecName: "istio-token") pod "0eda8b5c-04c2-42a4-83a6-3432587bbb33" (UID: "0eda8b5c-04c2-42a4-83a6-3432587bbb33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:50:35.748805 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.748770    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:50:35.748805 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.748802    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:50:35.749037 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.748843    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bplnw\" (UniqueName: \"kubernetes.io/projected/0eda8b5c-04c2-42a4-83a6-3432587bbb33-kube-api-access-bplnw\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:50:35.749037 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.748857    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:50:35.749037 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.748869    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:50:35.749037 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.748880    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:50:35.749037 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.748891    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:50:35.749037 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.748902    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:50:35.749037 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.748913    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0eda8b5c-04c2-42a4-83a6-3432587bbb33-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:50:35.952753 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.952723    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"]
Sep 09 13:50:35.957989 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:35.957961    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4bfe1d-openshift-default-59f779fc9d-w6rj4"]
Sep 09 13:50:37.150231 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:37.150190    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0eda8b5c-04c2-42a4-83a6-3432587bbb33" path="/var/lib/kubelet/pods/0eda8b5c-04c2-42a4-83a6-3432587bbb33/volumes"
Sep 09 13:50:41.704524 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:41.704489    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b5cba4-n7f5r"]
Sep 09 13:50:41.709992 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:41.709968    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b5cba4-n7f5r"]
Sep 09 13:50:43.149787 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:43.149753    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2ec8b0b-35b0-44bc-8d46-7c11ab76aad4" path="/var/lib/kubelet/pods/f2ec8b0b-35b0-44bc-8d46-7c11ab76aad4/volumes"
Sep 09 13:50:48.045301 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:48.045264    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"]
Sep 09 13:50:48.045736 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:48.045536    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd" podUID="4535ca39-b804-4428-8a3f-3f272f672519" containerName="istio-proxy" containerID="cri-o://5983eacb6eac1b48b214b4a3740e453d8975e79277ff1808ad34249e728e6075" gracePeriod=30
Sep 09 13:50:48.373982 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:48.373940    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6a7dbc-b8bwr"]
Sep 09 13:50:48.374403 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:48.374384    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eda8b5c-04c2-42a4-83a6-3432587bbb33" containerName="istio-proxy"
Sep 09 13:50:48.374492 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:48.374406    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eda8b5c-04c2-42a4-83a6-3432587bbb33" containerName="istio-proxy"
Sep 09 13:50:48.374492 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:48.374422    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2ec8b0b-35b0-44bc-8d46-7c11ab76aad4" containerName="revoke-keys"
Sep 09 13:50:48.374492 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:48.374431    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2ec8b0b-35b0-44bc-8d46-7c11ab76aad4" containerName="revoke-keys"
Sep 09 13:50:48.374492 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:48.374457    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eda8b5c-04c2-42a4-83a6-3432587bbb33" containerName="istio-proxy"
Sep 09 13:50:48.374492 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:48.374466    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eda8b5c-04c2-42a4-83a6-3432587bbb33" containerName="istio-proxy"
Sep 09 13:50:48.374746 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:48.374578    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eda8b5c-04c2-42a4-83a6-3432587bbb33" containerName="istio-proxy"
Sep 09 13:50:48.374746 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:48.374595    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2ec8b0b-35b0-44bc-8d46-7c11ab76aad4" containerName="revoke-keys"
Sep 09 13:50:48.379124 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:48.379102    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6a7dbc-b8bwr"
Sep 09 13:50:48.381117 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:48.381091    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 13:50:48.381250 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:48.381164    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lh487\""
Sep 09 13:50:48.384660 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:48.384629    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6a7dbc-b8bwr"]
Sep 09 13:50:48.475032 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:48.474994    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4526231e-b71e-4dbb-acdd-cf4b5b99116f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6a7dbc-b8bwr\" (UID: \"4526231e-b71e-4dbb-acdd-cf4b5b99116f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6a7dbc-b8bwr"
Sep 09 13:50:48.576113 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:48.576076    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4526231e-b71e-4dbb-acdd-cf4b5b99116f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6a7dbc-b8bwr\" (UID: \"4526231e-b71e-4dbb-acdd-cf4b5b99116f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6a7dbc-b8bwr"
Sep 09 13:50:48.576724 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:48.576702    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4526231e-b71e-4dbb-acdd-cf4b5b99116f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6a7dbc-b8bwr\" (UID: \"4526231e-b71e-4dbb-acdd-cf4b5b99116f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6a7dbc-b8bwr"
Sep 09 13:50:48.691330 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:48.691225    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6a7dbc-b8bwr"
Sep 09 13:50:48.968594 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:48.968549    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6a7dbc-b8bwr"]
Sep 09 13:50:48.971149 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:50:48.971117    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4526231e_b71e_4dbb_acdd_cf4b5b99116f.slice/crio-06ccbcbb755e20a72da71e69481687e2b6ed2f1defa2b14e410945f8f43be712 WatchSource:0}: Error finding container 06ccbcbb755e20a72da71e69481687e2b6ed2f1defa2b14e410945f8f43be712: Status 404 returned error can't find the container with id 06ccbcbb755e20a72da71e69481687e2b6ed2f1defa2b14e410945f8f43be712
Sep 09 13:50:49.707379 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:49.707344    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6a7dbc-b8bwr" event={"ID":"4526231e-b71e-4dbb-acdd-cf4b5b99116f","Type":"ContainerStarted","Data":"e468b87e0377a451caf339b85d141bc88b37f5f835933a2f90c3ff4e4c4c1744"}
Sep 09 13:50:49.707807 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:49.707411    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6a7dbc-b8bwr" event={"ID":"4526231e-b71e-4dbb-acdd-cf4b5b99116f","Type":"ContainerStarted","Data":"06ccbcbb755e20a72da71e69481687e2b6ed2f1defa2b14e410945f8f43be712"}
Sep 09 13:50:49.727337 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:49.727279    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6a7dbc-b8bwr" podStartSLOduration=1.72725842 podStartE2EDuration="1.72725842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:50:49.726141532 +0000 UTC m=+1483.085991357" watchObservedRunningTime="2026-09-09 13:50:49.72725842 +0000 UTC m=+1483.087108248"
Sep 09 13:50:50.713900 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:50.713855    2566 generic.go:356] "Generic (PLEG): container finished" podID="4526231e-b71e-4dbb-acdd-cf4b5b99116f" containerID="e468b87e0377a451caf339b85d141bc88b37f5f835933a2f90c3ff4e4c4c1744" exitCode=0
Sep 09 13:50:50.714322 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:50.713905    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6a7dbc-b8bwr" event={"ID":"4526231e-b71e-4dbb-acdd-cf4b5b99116f","Type":"ContainerDied","Data":"e468b87e0377a451caf339b85d141bc88b37f5f835933a2f90c3ff4e4c4c1744"}
Sep 09 13:50:51.852479 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:51.852449    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6a7dbc-b8bwr"
Sep 09 13:50:51.915998 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:51.915962    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4526231e-b71e-4dbb-acdd-cf4b5b99116f-maas-api-service-ca\") pod \"4526231e-b71e-4dbb-acdd-cf4b5b99116f\" (UID: \"4526231e-b71e-4dbb-acdd-cf4b5b99116f\") "
Sep 09 13:50:51.916356 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:51.916333    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4526231e-b71e-4dbb-acdd-cf4b5b99116f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4526231e-b71e-4dbb-acdd-cf4b5b99116f" (UID: "4526231e-b71e-4dbb-acdd-cf4b5b99116f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:50:52.016791 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:52.016700    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4526231e-b71e-4dbb-acdd-cf4b5b99116f-maas-api-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:50:52.725387 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:52.725358    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6a7dbc-b8bwr"
Sep 09 13:50:52.725387 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:52.725374    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6a7dbc-b8bwr" event={"ID":"4526231e-b71e-4dbb-acdd-cf4b5b99116f","Type":"ContainerDied","Data":"06ccbcbb755e20a72da71e69481687e2b6ed2f1defa2b14e410945f8f43be712"}
Sep 09 13:50:52.725643 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:52.725422    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06ccbcbb755e20a72da71e69481687e2b6ed2f1defa2b14e410945f8f43be712"
Sep 09 13:50:52.806228 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:52.806199    2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 13:50:52.806373 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:52.806258    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd" podUID="4535ca39-b804-4428-8a3f-3f272f672519" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 13:50:53.302648 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.302621    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:50:53.429633 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.429598    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nthsw\" (UniqueName: \"kubernetes.io/projected/4535ca39-b804-4428-8a3f-3f272f672519-kube-api-access-nthsw\") pod \"4535ca39-b804-4428-8a3f-3f272f672519\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") "
Sep 09 13:50:53.429804 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.429656    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-credential-socket\") pod \"4535ca39-b804-4428-8a3f-3f272f672519\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") "
Sep 09 13:50:53.429804 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.429767    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4535ca39-b804-4428-8a3f-3f272f672519-istio-podinfo\") pod \"4535ca39-b804-4428-8a3f-3f272f672519\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") "
Sep 09 13:50:53.429921 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.429805    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4535ca39-b804-4428-8a3f-3f272f672519-istio-token\") pod \"4535ca39-b804-4428-8a3f-3f272f672519\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") "
Sep 09 13:50:53.429921 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.429877    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-istio-envoy\") pod \"4535ca39-b804-4428-8a3f-3f272f672519\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") "
Sep 09 13:50:53.429921 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.429909    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-istio-data\") pod \"4535ca39-b804-4428-8a3f-3f272f672519\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") "
Sep 09 13:50:53.430122 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.429937    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4535ca39-b804-4428-8a3f-3f272f672519-istiod-ca-cert\") pod \"4535ca39-b804-4428-8a3f-3f272f672519\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") "
Sep 09 13:50:53.430122 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.429965    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-workload-certs\") pod \"4535ca39-b804-4428-8a3f-3f272f672519\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") "
Sep 09 13:50:53.430122 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.429970    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4535ca39-b804-4428-8a3f-3f272f672519" (UID: "4535ca39-b804-4428-8a3f-3f272f672519"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:50:53.430122 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.429991    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-workload-socket\") pod \"4535ca39-b804-4428-8a3f-3f272f672519\" (UID: \"4535ca39-b804-4428-8a3f-3f272f672519\") "
Sep 09 13:50:53.430346 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.430305    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4535ca39-b804-4428-8a3f-3f272f672519" (UID: "4535ca39-b804-4428-8a3f-3f272f672519"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:50:53.430423 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.430402    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:50:53.430485 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.430416    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4535ca39-b804-4428-8a3f-3f272f672519-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4535ca39-b804-4428-8a3f-3f272f672519" (UID: "4535ca39-b804-4428-8a3f-3f272f672519"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:50:53.430485 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.430426    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:50:53.430587 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.430527    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4535ca39-b804-4428-8a3f-3f272f672519" (UID: "4535ca39-b804-4428-8a3f-3f272f672519"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:50:53.430747 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.430723    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-istio-data" (OuterVolumeSpecName: "istio-data") pod "4535ca39-b804-4428-8a3f-3f272f672519" (UID: "4535ca39-b804-4428-8a3f-3f272f672519"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:50:53.431972 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.431953    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4535ca39-b804-4428-8a3f-3f272f672519" (UID: "4535ca39-b804-4428-8a3f-3f272f672519"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:50:53.432084 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.432062    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4535ca39-b804-4428-8a3f-3f272f672519-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4535ca39-b804-4428-8a3f-3f272f672519" (UID: "4535ca39-b804-4428-8a3f-3f272f672519"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 13:50:53.432284 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.432264    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4535ca39-b804-4428-8a3f-3f272f672519-istio-token" (OuterVolumeSpecName: "istio-token") pod "4535ca39-b804-4428-8a3f-3f272f672519" (UID: "4535ca39-b804-4428-8a3f-3f272f672519"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:50:53.432357 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.432309    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4535ca39-b804-4428-8a3f-3f272f672519-kube-api-access-nthsw" (OuterVolumeSpecName: "kube-api-access-nthsw") pod "4535ca39-b804-4428-8a3f-3f272f672519" (UID: "4535ca39-b804-4428-8a3f-3f272f672519"). InnerVolumeSpecName "kube-api-access-nthsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:50:53.531466 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.531429    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:50:53.531466 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.531460    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:50:53.531466 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.531472    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4535ca39-b804-4428-8a3f-3f272f672519-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:50:53.531684 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.531485    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4535ca39-b804-4428-8a3f-3f272f672519-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:50:53.531684 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.531497    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nthsw\" (UniqueName: \"kubernetes.io/projected/4535ca39-b804-4428-8a3f-3f272f672519-kube-api-access-nthsw\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:50:53.531684 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.531511    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4535ca39-b804-4428-8a3f-3f272f672519-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:50:53.531684 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.531523    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4535ca39-b804-4428-8a3f-3f272f672519-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:50:53.733501 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.733418    2566 generic.go:356] "Generic (PLEG): container finished" podID="4535ca39-b804-4428-8a3f-3f272f672519" containerID="5983eacb6eac1b48b214b4a3740e453d8975e79277ff1808ad34249e728e6075" exitCode=0
Sep 09 13:50:53.733501 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.733464    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"
Sep 09 13:50:53.733683 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.733499    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd" event={"ID":"4535ca39-b804-4428-8a3f-3f272f672519","Type":"ContainerDied","Data":"5983eacb6eac1b48b214b4a3740e453d8975e79277ff1808ad34249e728e6075"}
Sep 09 13:50:53.733683 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.733552    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd" event={"ID":"4535ca39-b804-4428-8a3f-3f272f672519","Type":"ContainerDied","Data":"ee7cdd029ef91c3226e08fd5703a43936475a7b75e00997831bca052cad105fe"}
Sep 09 13:50:53.733683 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.733575    2566 scope.go:117] "RemoveContainer" containerID="5983eacb6eac1b48b214b4a3740e453d8975e79277ff1808ad34249e728e6075"
Sep 09 13:50:53.745873 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.745855    2566 scope.go:117] "RemoveContainer" containerID="0f9ee376eb50a08f235d0e3c399f29ea618782a9fdc1ca75258cc91f28022f2e"
Sep 09 13:50:53.754592 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.754565    2566 scope.go:117] "RemoveContainer" containerID="5983eacb6eac1b48b214b4a3740e453d8975e79277ff1808ad34249e728e6075"
Sep 09 13:50:53.754885 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:50:53.754863    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5983eacb6eac1b48b214b4a3740e453d8975e79277ff1808ad34249e728e6075\": container with ID starting with 5983eacb6eac1b48b214b4a3740e453d8975e79277ff1808ad34249e728e6075 not found: ID does not exist" containerID="5983eacb6eac1b48b214b4a3740e453d8975e79277ff1808ad34249e728e6075"
Sep 09 13:50:53.754969 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.754891    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5983eacb6eac1b48b214b4a3740e453d8975e79277ff1808ad34249e728e6075"} err="failed to get container status \"5983eacb6eac1b48b214b4a3740e453d8975e79277ff1808ad34249e728e6075\": rpc error: code = NotFound desc = could not find container \"5983eacb6eac1b48b214b4a3740e453d8975e79277ff1808ad34249e728e6075\": container with ID starting with 5983eacb6eac1b48b214b4a3740e453d8975e79277ff1808ad34249e728e6075 not found: ID does not exist"
Sep 09 13:50:53.754969 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.754909    2566 scope.go:117] "RemoveContainer" containerID="0f9ee376eb50a08f235d0e3c399f29ea618782a9fdc1ca75258cc91f28022f2e"
Sep 09 13:50:53.755163 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:50:53.755147    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f9ee376eb50a08f235d0e3c399f29ea618782a9fdc1ca75258cc91f28022f2e\": container with ID starting with 0f9ee376eb50a08f235d0e3c399f29ea618782a9fdc1ca75258cc91f28022f2e not found: ID does not exist" containerID="0f9ee376eb50a08f235d0e3c399f29ea618782a9fdc1ca75258cc91f28022f2e"
Sep 09 13:50:53.755221 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.755169    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f9ee376eb50a08f235d0e3c399f29ea618782a9fdc1ca75258cc91f28022f2e"} err="failed to get container status \"0f9ee376eb50a08f235d0e3c399f29ea618782a9fdc1ca75258cc91f28022f2e\": rpc error: code = NotFound desc = could not find container \"0f9ee376eb50a08f235d0e3c399f29ea618782a9fdc1ca75258cc91f28022f2e\": container with ID starting with 0f9ee376eb50a08f235d0e3c399f29ea618782a9fdc1ca75258cc91f28022f2e not found: ID does not exist"
Sep 09 13:50:53.755909 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.755885    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"]
Sep 09 13:50:53.758800 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:53.758780    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b5cba4-openshift-default-57fb6bbdb4-qzsvd"]
Sep 09 13:50:55.150022 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:55.149987    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4535ca39-b804-4428-8a3f-3f272f672519" path="/var/lib/kubelet/pods/4535ca39-b804-4428-8a3f-3f272f672519/volumes"
Sep 09 13:50:59.193279 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:59.193247    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6a7dbc-b8bwr"]
Sep 09 13:50:59.196159 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:50:59.196136    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6a7dbc-b8bwr"]
Sep 09 13:51:01.152953 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:01.151366    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4526231e-b71e-4dbb-acdd-cf4b5b99116f" path="/var/lib/kubelet/pods/4526231e-b71e-4dbb-acdd-cf4b5b99116f/volumes"
Sep 09 13:51:05.973940 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:05.973898    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"]
Sep 09 13:51:05.974456 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:05.974206    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm" podUID="83622cac-d0f5-4aa6-8502-aa9019e9429c" containerName="istio-proxy" containerID="cri-o://c1f6769dca07b6bc8c67425f8f52f7e566097ef200b507b5360f68ea7ecd13e7" gracePeriod=30
Sep 09 13:51:06.272158 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:06.272074    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 13:51:06.272158 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:06.272149    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm" podUID="83622cac-d0f5-4aa6-8502-aa9019e9429c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 13:51:06.525168 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:06.525080    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"]
Sep 09 13:51:07.203709 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:07.203678    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ftm96_af902e8b-d693-4341-9979-03cf69548cfb/console-operator/2.log"
Sep 09 13:51:07.208697 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:07.208677    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ftm96_af902e8b-d693-4341-9979-03cf69548cfb/console-operator/2.log"
Sep 09 13:51:07.456400 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:07.456315    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"]
Sep 09 13:51:11.228341 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.228314    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:51:11.302054 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.301956    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-istio-envoy\") pod \"83622cac-d0f5-4aa6-8502-aa9019e9429c\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") "
Sep 09 13:51:11.302054 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.302014    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83622cac-d0f5-4aa6-8502-aa9019e9429c-istiod-ca-cert\") pod \"83622cac-d0f5-4aa6-8502-aa9019e9429c\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") "
Sep 09 13:51:11.302054 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.302033    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-workload-socket\") pod \"83622cac-d0f5-4aa6-8502-aa9019e9429c\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") "
Sep 09 13:51:11.302366 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.302217    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-credential-socket\") pod \"83622cac-d0f5-4aa6-8502-aa9019e9429c\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") "
Sep 09 13:51:11.302366 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.302279    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-workload-certs\") pod \"83622cac-d0f5-4aa6-8502-aa9019e9429c\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") "
Sep 09 13:51:11.302366 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.302321    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "83622cac-d0f5-4aa6-8502-aa9019e9429c" (UID: "83622cac-d0f5-4aa6-8502-aa9019e9429c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:51:11.302366 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.302330    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-istio-data\") pod \"83622cac-d0f5-4aa6-8502-aa9019e9429c\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") "
Sep 09 13:51:11.302366 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.302365    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83622cac-d0f5-4aa6-8502-aa9019e9429c-istio-podinfo\") pod \"83622cac-d0f5-4aa6-8502-aa9019e9429c\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") "
Sep 09 13:51:11.302635 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.302410    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83622cac-d0f5-4aa6-8502-aa9019e9429c-istio-token\") pod \"83622cac-d0f5-4aa6-8502-aa9019e9429c\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") "
Sep 09 13:51:11.302635 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.302435    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfjts\" (UniqueName: \"kubernetes.io/projected/83622cac-d0f5-4aa6-8502-aa9019e9429c-kube-api-access-dfjts\") pod \"83622cac-d0f5-4aa6-8502-aa9019e9429c\" (UID: \"83622cac-d0f5-4aa6-8502-aa9019e9429c\") "
Sep 09 13:51:11.302746 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.302429    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83622cac-d0f5-4aa6-8502-aa9019e9429c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "83622cac-d0f5-4aa6-8502-aa9019e9429c" (UID: "83622cac-d0f5-4aa6-8502-aa9019e9429c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:51:11.302746 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.302561    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "83622cac-d0f5-4aa6-8502-aa9019e9429c" (UID: "83622cac-d0f5-4aa6-8502-aa9019e9429c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:51:11.302746 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.302671    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-istio-data" (OuterVolumeSpecName: "istio-data") pod "83622cac-d0f5-4aa6-8502-aa9019e9429c" (UID: "83622cac-d0f5-4aa6-8502-aa9019e9429c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:51:11.302746 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.302559    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "83622cac-d0f5-4aa6-8502-aa9019e9429c" (UID: "83622cac-d0f5-4aa6-8502-aa9019e9429c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:51:11.302990 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.302787    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:51:11.302990 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.302836    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:51:11.302990 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.302851    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83622cac-d0f5-4aa6-8502-aa9019e9429c-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:51:11.302990 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.302863    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:51:11.302990 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.302875    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:51:11.304380 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.304352    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "83622cac-d0f5-4aa6-8502-aa9019e9429c" (UID: "83622cac-d0f5-4aa6-8502-aa9019e9429c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:51:11.304547 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.304526    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83622cac-d0f5-4aa6-8502-aa9019e9429c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "83622cac-d0f5-4aa6-8502-aa9019e9429c" (UID: "83622cac-d0f5-4aa6-8502-aa9019e9429c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 13:51:11.304632 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.304555    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83622cac-d0f5-4aa6-8502-aa9019e9429c-istio-token" (OuterVolumeSpecName: "istio-token") pod "83622cac-d0f5-4aa6-8502-aa9019e9429c" (UID: "83622cac-d0f5-4aa6-8502-aa9019e9429c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:51:11.304684 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.304627    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83622cac-d0f5-4aa6-8502-aa9019e9429c-kube-api-access-dfjts" (OuterVolumeSpecName: "kube-api-access-dfjts") pod "83622cac-d0f5-4aa6-8502-aa9019e9429c" (UID: "83622cac-d0f5-4aa6-8502-aa9019e9429c"). InnerVolumeSpecName "kube-api-access-dfjts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:51:11.403436 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.403396    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83622cac-d0f5-4aa6-8502-aa9019e9429c-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:51:11.403436 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.403429    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83622cac-d0f5-4aa6-8502-aa9019e9429c-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:51:11.403436 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.403438    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfjts\" (UniqueName: \"kubernetes.io/projected/83622cac-d0f5-4aa6-8502-aa9019e9429c-kube-api-access-dfjts\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:51:11.403436 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.403446    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83622cac-d0f5-4aa6-8502-aa9019e9429c-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:51:11.835150 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.835116    2566 generic.go:356] "Generic (PLEG): container finished" podID="83622cac-d0f5-4aa6-8502-aa9019e9429c" containerID="c1f6769dca07b6bc8c67425f8f52f7e566097ef200b507b5360f68ea7ecd13e7" exitCode=0
Sep 09 13:51:11.835332 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.835165    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"
Sep 09 13:51:11.835332 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.835181    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm" event={"ID":"83622cac-d0f5-4aa6-8502-aa9019e9429c","Type":"ContainerDied","Data":"c1f6769dca07b6bc8c67425f8f52f7e566097ef200b507b5360f68ea7ecd13e7"}
Sep 09 13:51:11.835332 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.835226    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm" event={"ID":"83622cac-d0f5-4aa6-8502-aa9019e9429c","Type":"ContainerDied","Data":"df37145c651467d6c29d7921581094d58201087baf794ed348fb94f78e82f30d"}
Sep 09 13:51:11.835332 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.835248    2566 scope.go:117] "RemoveContainer" containerID="c1f6769dca07b6bc8c67425f8f52f7e566097ef200b507b5360f68ea7ecd13e7"
Sep 09 13:51:11.844686 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.844668    2566 scope.go:117] "RemoveContainer" containerID="857740802d67ea248bcbc5fba9640eedafaabf0a4fff422575ed410ac985c7f3"
Sep 09 13:51:11.852612 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.852593    2566 scope.go:117] "RemoveContainer" containerID="c1f6769dca07b6bc8c67425f8f52f7e566097ef200b507b5360f68ea7ecd13e7"
Sep 09 13:51:11.852911 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:51:11.852891    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1f6769dca07b6bc8c67425f8f52f7e566097ef200b507b5360f68ea7ecd13e7\": container with ID starting with c1f6769dca07b6bc8c67425f8f52f7e566097ef200b507b5360f68ea7ecd13e7 not found: ID does not exist" containerID="c1f6769dca07b6bc8c67425f8f52f7e566097ef200b507b5360f68ea7ecd13e7"
Sep 09 13:51:11.852976 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.852923    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1f6769dca07b6bc8c67425f8f52f7e566097ef200b507b5360f68ea7ecd13e7"} err="failed to get container status \"c1f6769dca07b6bc8c67425f8f52f7e566097ef200b507b5360f68ea7ecd13e7\": rpc error: code = NotFound desc = could not find container \"c1f6769dca07b6bc8c67425f8f52f7e566097ef200b507b5360f68ea7ecd13e7\": container with ID starting with c1f6769dca07b6bc8c67425f8f52f7e566097ef200b507b5360f68ea7ecd13e7 not found: ID does not exist"
Sep 09 13:51:11.852976 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.852940    2566 scope.go:117] "RemoveContainer" containerID="857740802d67ea248bcbc5fba9640eedafaabf0a4fff422575ed410ac985c7f3"
Sep 09 13:51:11.853176 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:51:11.853163    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"857740802d67ea248bcbc5fba9640eedafaabf0a4fff422575ed410ac985c7f3\": container with ID starting with 857740802d67ea248bcbc5fba9640eedafaabf0a4fff422575ed410ac985c7f3 not found: ID does not exist" containerID="857740802d67ea248bcbc5fba9640eedafaabf0a4fff422575ed410ac985c7f3"
Sep 09 13:51:11.853216 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.853179    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"857740802d67ea248bcbc5fba9640eedafaabf0a4fff422575ed410ac985c7f3"} err="failed to get container status \"857740802d67ea248bcbc5fba9640eedafaabf0a4fff422575ed410ac985c7f3\": rpc error: code = NotFound desc = could not find container \"857740802d67ea248bcbc5fba9640eedafaabf0a4fff422575ed410ac985c7f3\": container with ID starting with 857740802d67ea248bcbc5fba9640eedafaabf0a4fff422575ed410ac985c7f3 not found: ID does not exist"
Sep 09 13:51:11.856679 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.856657    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"]
Sep 09 13:51:11.861364 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:11.861345    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6a7dbc-openshift-default-5df4d6474-zwlsm"]
Sep 09 13:51:13.149945 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:13.149912    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83622cac-d0f5-4aa6-8502-aa9019e9429c" path="/var/lib/kubelet/pods/83622cac-d0f5-4aa6-8502-aa9019e9429c/volumes"
Sep 09 13:51:21.548629 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:21.548582    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx" podUID="884129c9-b984-4ed8-bcda-1464a001c9a8" containerName="main" containerID="cri-o://67d1469ffda9c80b2087814de6bc722cf822cdfaa2f23341ae1df98089742ed1" gracePeriod=45
Sep 09 13:51:21.791037 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:21.791014    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"
Sep 09 13:51:21.889364 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:21.889330    2566 generic.go:356] "Generic (PLEG): container finished" podID="884129c9-b984-4ed8-bcda-1464a001c9a8" containerID="67d1469ffda9c80b2087814de6bc722cf822cdfaa2f23341ae1df98089742ed1" exitCode=0
Sep 09 13:51:21.889578 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:21.889375    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"
Sep 09 13:51:21.889578 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:21.889389    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx" event={"ID":"884129c9-b984-4ed8-bcda-1464a001c9a8","Type":"ContainerDied","Data":"67d1469ffda9c80b2087814de6bc722cf822cdfaa2f23341ae1df98089742ed1"}
Sep 09 13:51:21.889578 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:21.889421    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx" event={"ID":"884129c9-b984-4ed8-bcda-1464a001c9a8","Type":"ContainerDied","Data":"3c9f6371f298dacbfd4963ec4c885e28ea0c1f396d2bacdea0a0e872b933a795"}
Sep 09 13:51:21.889578 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:21.889439    2566 scope.go:117] "RemoveContainer" containerID="67d1469ffda9c80b2087814de6bc722cf822cdfaa2f23341ae1df98089742ed1"
Sep 09 13:51:21.893992 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:21.893970    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/884129c9-b984-4ed8-bcda-1464a001c9a8-home\") pod \"884129c9-b984-4ed8-bcda-1464a001c9a8\" (UID: \"884129c9-b984-4ed8-bcda-1464a001c9a8\") "
Sep 09 13:51:21.894125 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:21.894037    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bc6p2\" (UniqueName: \"kubernetes.io/projected/884129c9-b984-4ed8-bcda-1464a001c9a8-kube-api-access-bc6p2\") pod \"884129c9-b984-4ed8-bcda-1464a001c9a8\" (UID: \"884129c9-b984-4ed8-bcda-1464a001c9a8\") "
Sep 09 13:51:21.894197 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:21.894120    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/884129c9-b984-4ed8-bcda-1464a001c9a8-model-cache\") pod \"884129c9-b984-4ed8-bcda-1464a001c9a8\" (UID: \"884129c9-b984-4ed8-bcda-1464a001c9a8\") "
Sep 09 13:51:21.894197 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:21.894177    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/884129c9-b984-4ed8-bcda-1464a001c9a8-tmp-dir\") pod \"884129c9-b984-4ed8-bcda-1464a001c9a8\" (UID: \"884129c9-b984-4ed8-bcda-1464a001c9a8\") "
Sep 09 13:51:21.894305 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:21.894211    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884129c9-b984-4ed8-bcda-1464a001c9a8-home" (OuterVolumeSpecName: "home") pod "884129c9-b984-4ed8-bcda-1464a001c9a8" (UID: "884129c9-b984-4ed8-bcda-1464a001c9a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:51:21.894305 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:21.894240    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/884129c9-b984-4ed8-bcda-1464a001c9a8-tls-certs\") pod \"884129c9-b984-4ed8-bcda-1464a001c9a8\" (UID: \"884129c9-b984-4ed8-bcda-1464a001c9a8\") "
Sep 09 13:51:21.894305 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:21.894273    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/884129c9-b984-4ed8-bcda-1464a001c9a8-dshm\") pod \"884129c9-b984-4ed8-bcda-1464a001c9a8\" (UID: \"884129c9-b984-4ed8-bcda-1464a001c9a8\") "
Sep 09 13:51:21.894456 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:21.894387    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884129c9-b984-4ed8-bcda-1464a001c9a8-model-cache" (OuterVolumeSpecName: "model-cache") pod "884129c9-b984-4ed8-bcda-1464a001c9a8" (UID: "884129c9-b984-4ed8-bcda-1464a001c9a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:51:21.894519 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:21.894486    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884129c9-b984-4ed8-bcda-1464a001c9a8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "884129c9-b984-4ed8-bcda-1464a001c9a8" (UID: "884129c9-b984-4ed8-bcda-1464a001c9a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:51:21.894637 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:21.894615    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/884129c9-b984-4ed8-bcda-1464a001c9a8-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:51:21.894637 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:21.894639    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/884129c9-b984-4ed8-bcda-1464a001c9a8-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:51:21.894842 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:21.894651    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/884129c9-b984-4ed8-bcda-1464a001c9a8-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:51:21.896215 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:21.896188    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884129c9-b984-4ed8-bcda-1464a001c9a8-kube-api-access-bc6p2" (OuterVolumeSpecName: "kube-api-access-bc6p2") pod "884129c9-b984-4ed8-bcda-1464a001c9a8" (UID: "884129c9-b984-4ed8-bcda-1464a001c9a8"). InnerVolumeSpecName "kube-api-access-bc6p2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:51:21.896348 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:21.896333    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884129c9-b984-4ed8-bcda-1464a001c9a8-dshm" (OuterVolumeSpecName: "dshm") pod "884129c9-b984-4ed8-bcda-1464a001c9a8" (UID: "884129c9-b984-4ed8-bcda-1464a001c9a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:51:21.896420 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:21.896385    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/884129c9-b984-4ed8-bcda-1464a001c9a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "884129c9-b984-4ed8-bcda-1464a001c9a8" (UID: "884129c9-b984-4ed8-bcda-1464a001c9a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:51:21.898951 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:21.898933    2566 scope.go:117] "RemoveContainer" containerID="67d1469ffda9c80b2087814de6bc722cf822cdfaa2f23341ae1df98089742ed1"
Sep 09 13:51:21.899249 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:51:21.899228    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67d1469ffda9c80b2087814de6bc722cf822cdfaa2f23341ae1df98089742ed1\": container with ID starting with 67d1469ffda9c80b2087814de6bc722cf822cdfaa2f23341ae1df98089742ed1 not found: ID does not exist" containerID="67d1469ffda9c80b2087814de6bc722cf822cdfaa2f23341ae1df98089742ed1"
Sep 09 13:51:21.899310 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:21.899259    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d1469ffda9c80b2087814de6bc722cf822cdfaa2f23341ae1df98089742ed1"} err="failed to get container status \"67d1469ffda9c80b2087814de6bc722cf822cdfaa2f23341ae1df98089742ed1\": rpc error: code = NotFound desc = could not find container \"67d1469ffda9c80b2087814de6bc722cf822cdfaa2f23341ae1df98089742ed1\": container with ID starting with 67d1469ffda9c80b2087814de6bc722cf822cdfaa2f23341ae1df98089742ed1 not found: ID does not exist"
Sep 09 13:51:21.996118 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:21.996075    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/884129c9-b984-4ed8-bcda-1464a001c9a8-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:51:21.996118 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:21.996107    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/884129c9-b984-4ed8-bcda-1464a001c9a8-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:51:21.996118 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:21.996116    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc6p2\" (UniqueName: \"kubernetes.io/projected/884129c9-b984-4ed8-bcda-1464a001c9a8-kube-api-access-bc6p2\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:51:22.212421 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.212384    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"]
Sep 09 13:51:22.216326 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.216296    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8f2df2/rate-test-model-8f2df2-kserve-57fc6bcdf5-nhqkx"]
Sep 09 13:51:22.476648 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.476541    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx" podUID="55d41a13-de5d-463f-bded-619051df6fdc" containerName="main" containerID="cri-o://ec3aa1dc4e3622284bbdc6ff3734dedad20ef1aa61de4c55383f819b64d80b08" gracePeriod=45
Sep 09 13:51:22.725400 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.725373    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"
Sep 09 13:51:22.804380 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.804297    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55d41a13-de5d-463f-bded-619051df6fdc-model-cache\") pod \"55d41a13-de5d-463f-bded-619051df6fdc\" (UID: \"55d41a13-de5d-463f-bded-619051df6fdc\") "
Sep 09 13:51:22.804380 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.804342    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55d41a13-de5d-463f-bded-619051df6fdc-dshm\") pod \"55d41a13-de5d-463f-bded-619051df6fdc\" (UID: \"55d41a13-de5d-463f-bded-619051df6fdc\") "
Sep 09 13:51:22.804380 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.804363    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55d41a13-de5d-463f-bded-619051df6fdc-tmp-dir\") pod \"55d41a13-de5d-463f-bded-619051df6fdc\" (UID: \"55d41a13-de5d-463f-bded-619051df6fdc\") "
Sep 09 13:51:22.804669 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.804428    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z54sd\" (UniqueName: \"kubernetes.io/projected/55d41a13-de5d-463f-bded-619051df6fdc-kube-api-access-z54sd\") pod \"55d41a13-de5d-463f-bded-619051df6fdc\" (UID: \"55d41a13-de5d-463f-bded-619051df6fdc\") "
Sep 09 13:51:22.804669 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.804454    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55d41a13-de5d-463f-bded-619051df6fdc-home\") pod \"55d41a13-de5d-463f-bded-619051df6fdc\" (UID: \"55d41a13-de5d-463f-bded-619051df6fdc\") "
Sep 09 13:51:22.804669 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.804520    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55d41a13-de5d-463f-bded-619051df6fdc-tls-certs\") pod \"55d41a13-de5d-463f-bded-619051df6fdc\" (UID: \"55d41a13-de5d-463f-bded-619051df6fdc\") "
Sep 09 13:51:22.804669 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.804624    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55d41a13-de5d-463f-bded-619051df6fdc-model-cache" (OuterVolumeSpecName: "model-cache") pod "55d41a13-de5d-463f-bded-619051df6fdc" (UID: "55d41a13-de5d-463f-bded-619051df6fdc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:51:22.804963 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.804663    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55d41a13-de5d-463f-bded-619051df6fdc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55d41a13-de5d-463f-bded-619051df6fdc" (UID: "55d41a13-de5d-463f-bded-619051df6fdc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:51:22.804963 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.804797    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55d41a13-de5d-463f-bded-619051df6fdc-home" (OuterVolumeSpecName: "home") pod "55d41a13-de5d-463f-bded-619051df6fdc" (UID: "55d41a13-de5d-463f-bded-619051df6fdc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:51:22.805063 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.804980    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55d41a13-de5d-463f-bded-619051df6fdc-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:51:22.805063 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.805001    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55d41a13-de5d-463f-bded-619051df6fdc-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:51:22.805063 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.805018    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55d41a13-de5d-463f-bded-619051df6fdc-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:51:22.806527 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.806506    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55d41a13-de5d-463f-bded-619051df6fdc-dshm" (OuterVolumeSpecName: "dshm") pod "55d41a13-de5d-463f-bded-619051df6fdc" (UID: "55d41a13-de5d-463f-bded-619051df6fdc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:51:22.806595 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.806575    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d41a13-de5d-463f-bded-619051df6fdc-kube-api-access-z54sd" (OuterVolumeSpecName: "kube-api-access-z54sd") pod "55d41a13-de5d-463f-bded-619051df6fdc" (UID: "55d41a13-de5d-463f-bded-619051df6fdc"). InnerVolumeSpecName "kube-api-access-z54sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:51:22.806632 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.806608    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55d41a13-de5d-463f-bded-619051df6fdc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55d41a13-de5d-463f-bded-619051df6fdc" (UID: "55d41a13-de5d-463f-bded-619051df6fdc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:51:22.896485 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.896453    2566 generic.go:356] "Generic (PLEG): container finished" podID="55d41a13-de5d-463f-bded-619051df6fdc" containerID="ec3aa1dc4e3622284bbdc6ff3734dedad20ef1aa61de4c55383f819b64d80b08" exitCode=0
Sep 09 13:51:22.896609 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.896501    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"
Sep 09 13:51:22.896609 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.896527    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx" event={"ID":"55d41a13-de5d-463f-bded-619051df6fdc","Type":"ContainerDied","Data":"ec3aa1dc4e3622284bbdc6ff3734dedad20ef1aa61de4c55383f819b64d80b08"}
Sep 09 13:51:22.896609 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.896563    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx" event={"ID":"55d41a13-de5d-463f-bded-619051df6fdc","Type":"ContainerDied","Data":"bc4b65aaee7edf69130fb28e3545bd7bdfbe2df3d07cf8d10bdf4a82f6f40940"}
Sep 09 13:51:22.896609 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.896584    2566 scope.go:117] "RemoveContainer" containerID="ec3aa1dc4e3622284bbdc6ff3734dedad20ef1aa61de4c55383f819b64d80b08"
Sep 09 13:51:22.905292 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.905275    2566 scope.go:117] "RemoveContainer" containerID="ec3aa1dc4e3622284bbdc6ff3734dedad20ef1aa61de4c55383f819b64d80b08"
Sep 09 13:51:22.905582 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:51:22.905562    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec3aa1dc4e3622284bbdc6ff3734dedad20ef1aa61de4c55383f819b64d80b08\": container with ID starting with ec3aa1dc4e3622284bbdc6ff3734dedad20ef1aa61de4c55383f819b64d80b08 not found: ID does not exist" containerID="ec3aa1dc4e3622284bbdc6ff3734dedad20ef1aa61de4c55383f819b64d80b08"
Sep 09 13:51:22.905649 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.905593    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec3aa1dc4e3622284bbdc6ff3734dedad20ef1aa61de4c55383f819b64d80b08"} err="failed to get container status \"ec3aa1dc4e3622284bbdc6ff3734dedad20ef1aa61de4c55383f819b64d80b08\": rpc error: code = NotFound desc = could not find container \"ec3aa1dc4e3622284bbdc6ff3734dedad20ef1aa61de4c55383f819b64d80b08\": container with ID starting with ec3aa1dc4e3622284bbdc6ff3734dedad20ef1aa61de4c55383f819b64d80b08 not found: ID does not exist"
Sep 09 13:51:22.905694 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.905660    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55d41a13-de5d-463f-bded-619051df6fdc-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:51:22.905694 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.905679    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z54sd\" (UniqueName: \"kubernetes.io/projected/55d41a13-de5d-463f-bded-619051df6fdc-kube-api-access-z54sd\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:51:22.905694 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.905689    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55d41a13-de5d-463f-bded-619051df6fdc-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:51:22.916879 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.916857    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"]
Sep 09 13:51:22.920713 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:22.920692    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-df48ae/rate-test-model-df48ae-kserve-84f65749ff-fcjxx"]
Sep 09 13:51:23.149702 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:23.149671    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55d41a13-de5d-463f-bded-619051df6fdc" path="/var/lib/kubelet/pods/55d41a13-de5d-463f-bded-619051df6fdc/volumes"
Sep 09 13:51:23.150080 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:23.150066    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="884129c9-b984-4ed8-bcda-1464a001c9a8" path="/var/lib/kubelet/pods/884129c9-b984-4ed8-bcda-1464a001c9a8/volumes"
Sep 09 13:51:37.488643 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.488559    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"]
Sep 09 13:51:37.489050 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.488928    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="884129c9-b984-4ed8-bcda-1464a001c9a8" containerName="main"
Sep 09 13:51:37.489050 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.488941    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="884129c9-b984-4ed8-bcda-1464a001c9a8" containerName="main"
Sep 09 13:51:37.489050 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.488955    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4535ca39-b804-4428-8a3f-3f272f672519" containerName="istio-proxy"
Sep 09 13:51:37.489050 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.488961    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4535ca39-b804-4428-8a3f-3f272f672519" containerName="istio-proxy"
Sep 09 13:51:37.489050 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.488974    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83622cac-d0f5-4aa6-8502-aa9019e9429c" containerName="istio-proxy"
Sep 09 13:51:37.489050 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.488982    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="83622cac-d0f5-4aa6-8502-aa9019e9429c" containerName="istio-proxy"
Sep 09 13:51:37.489050 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.488991    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4526231e-b71e-4dbb-acdd-cf4b5b99116f" containerName="revoke-keys"
Sep 09 13:51:37.489050 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.488996    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4526231e-b71e-4dbb-acdd-cf4b5b99116f" containerName="revoke-keys"
Sep 09 13:51:37.489050 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.489003    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55d41a13-de5d-463f-bded-619051df6fdc" containerName="main"
Sep 09 13:51:37.489050 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.489008    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="55d41a13-de5d-463f-bded-619051df6fdc" containerName="main"
Sep 09 13:51:37.489050 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.489018    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4535ca39-b804-4428-8a3f-3f272f672519" containerName="istio-proxy"
Sep 09 13:51:37.489050 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.489024    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4535ca39-b804-4428-8a3f-3f272f672519" containerName="istio-proxy"
Sep 09 13:51:37.489050 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.489034    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83622cac-d0f5-4aa6-8502-aa9019e9429c" containerName="istio-proxy"
Sep 09 13:51:37.489050 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.489039    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="83622cac-d0f5-4aa6-8502-aa9019e9429c" containerName="istio-proxy"
Sep 09 13:51:37.489518 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.489121    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4535ca39-b804-4428-8a3f-3f272f672519" containerName="istio-proxy"
Sep 09 13:51:37.489518 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.489134    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4535ca39-b804-4428-8a3f-3f272f672519" containerName="istio-proxy"
Sep 09 13:51:37.489518 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.489140    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="884129c9-b984-4ed8-bcda-1464a001c9a8" containerName="main"
Sep 09 13:51:37.489518 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.489147    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eda8b5c-04c2-42a4-83a6-3432587bbb33" containerName="istio-proxy"
Sep 09 13:51:37.489518 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.489155    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="83622cac-d0f5-4aa6-8502-aa9019e9429c" containerName="istio-proxy"
Sep 09 13:51:37.489518 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.489159    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="83622cac-d0f5-4aa6-8502-aa9019e9429c" containerName="istio-proxy"
Sep 09 13:51:37.489518 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.489164    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="55d41a13-de5d-463f-bded-619051df6fdc" containerName="main"
Sep 09 13:51:37.489518 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.489169    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4526231e-b71e-4dbb-acdd-cf4b5b99116f" containerName="revoke-keys"
Sep 09 13:51:37.493669 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.493649    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:37.496077 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.496052    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-3691cf-openshift-default-dockercfg-z9hng\""
Sep 09 13:51:37.504423 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.504401    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"]
Sep 09 13:51:37.639690 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.639654    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vz8m\" (UniqueName: \"kubernetes.io/projected/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-kube-api-access-4vz8m\") pod \"e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") " pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:37.639690 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.639692    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istio-data\") pod \"e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") " pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:37.639938 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.639716    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-workload-socket\") pod \"e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") " pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:37.639938 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.639755    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istio-token\") pod \"e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") " pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:37.639938 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.639799    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-credential-socket\") pod \"e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") " pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:37.639938 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.639885    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istio-envoy\") pod \"e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") " pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:37.640071 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.639942    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-workload-certs\") pod \"e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") " pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:37.640071 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.639965    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istio-podinfo\") pod \"e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") " pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:37.640071 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.640053    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istiod-ca-cert\") pod \"e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") " pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:37.740784 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.740710    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istio-envoy\") pod \"e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") " pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:37.740784 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.740751    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-workload-certs\") pod \"e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") " pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:37.740784 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.740766    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istio-podinfo\") pod \"e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") " pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:37.741056 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.740837    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istiod-ca-cert\") pod \"e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") " pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:37.741056 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.740867    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vz8m\" (UniqueName: \"kubernetes.io/projected/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-kube-api-access-4vz8m\") pod \"e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") " pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:37.741056 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.740883    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istio-data\") pod \"e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") " pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:37.741056 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.740913    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-workload-socket\") pod \"e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") " pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:37.741056 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.740940    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istio-token\") pod \"e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") " pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:37.741056 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.740979    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-credential-socket\") pod \"e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") " pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:37.741381 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.741244    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-workload-certs\") pod \"e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") " pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:37.741381 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.741326    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istio-data\") pod \"e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") " pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:37.741381 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.741368    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-credential-socket\") pod \"e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") " pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:37.741536 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.741449    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-workload-socket\") pod \"e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") " pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:37.741683 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.741661    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istiod-ca-cert\") pod \"e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") " pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:37.743149 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.743130    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istio-envoy\") pod \"e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") " pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:37.743273 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.743257    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istio-podinfo\") pod \"e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") " pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:37.749228 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.749190    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istio-token\") pod \"e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") " pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:37.749865 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.749847    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vz8m\" (UniqueName: \"kubernetes.io/projected/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-kube-api-access-4vz8m\") pod \"e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") " pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:37.803894 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:37.803874    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:38.077886 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:38.077862    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"]
Sep 09 13:51:38.079783 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:51:38.079752    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4352a0c7_cc5c_4572_a17c_f4bdc9775b5e.slice/crio-a34058a1827247b3cbf605f07fd003b310657d67cbe6adeebb2e579d784b79b8 WatchSource:0}: Error finding container a34058a1827247b3cbf605f07fd003b310657d67cbe6adeebb2e579d784b79b8: Status 404 returned error can't find the container with id a34058a1827247b3cbf605f07fd003b310657d67cbe6adeebb2e579d784b79b8
Sep 09 13:51:38.082000 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:38.081968    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:51:38.082106 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:38.082030    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:51:38.082106 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:38.082059    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:51:38.978895 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:38.978859    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf" event={"ID":"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e","Type":"ContainerStarted","Data":"9aa82d5c6b43173a726a3e7af9f6a8947eb320c2c70fd43211c0e47a0d1567e9"}
Sep 09 13:51:38.979362 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:38.978914    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf" event={"ID":"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e","Type":"ContainerStarted","Data":"a34058a1827247b3cbf605f07fd003b310657d67cbe6adeebb2e579d784b79b8"}
Sep 09 13:51:38.997788 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:38.997741    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf" podStartSLOduration=1.9977224 podStartE2EDuration="1.9977224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:51:38.997313342 +0000 UTC m=+1532.357163168" watchObservedRunningTime="2026-09-09 13:51:38.9977224 +0000 UTC m=+1532.357572222"
Sep 09 13:51:39.805097 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:39.805066    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:39.810073 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:39.810048    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:39.984850 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:39.984797    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:39.985724 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:39.985705    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:51:42.751634 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:42.751598    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv"]
Sep 09 13:51:42.755166 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:42.755150    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv"
Sep 09 13:51:42.758624 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:42.758599    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3691cf\""
Sep 09 13:51:42.758761 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:42.758642    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3691cf-dockercfg-d5r85\""
Sep 09 13:51:42.764752 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:42.764729    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv"]
Sep 09 13:51:42.777550 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:42.777527    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm"]
Sep 09 13:51:42.780831 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:42.780797    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm"
Sep 09 13:51:42.790999 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:42.790978    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm"]
Sep 09 13:51:42.890181 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:42.890147    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ab8d99d-c6e6-4394-b978-93d2a23152f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv\" (UID: \"8ab8d99d-c6e6-4394-b978-93d2a23152f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv"
Sep 09 13:51:42.890350 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:42.890209    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6svd\" (UniqueName: \"kubernetes.io/projected/8ab8d99d-c6e6-4394-b978-93d2a23152f3-kube-api-access-g6svd\") pod \"payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv\" (UID: \"8ab8d99d-c6e6-4394-b978-93d2a23152f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv"
Sep 09 13:51:42.890350 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:42.890261    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5r4r\" (UniqueName: \"kubernetes.io/projected/5e7b51de-bdfd-451c-a7f9-5230a947a1be-kube-api-access-p5r4r\") pod \"payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm\" (UID: \"5e7b51de-bdfd-451c-a7f9-5230a947a1be\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm"
Sep 09 13:51:42.890434 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:42.890337    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e7b51de-bdfd-451c-a7f9-5230a947a1be-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm\" (UID: \"5e7b51de-bdfd-451c-a7f9-5230a947a1be\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm"
Sep 09 13:51:42.991570 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:42.991534    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e7b51de-bdfd-451c-a7f9-5230a947a1be-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm\" (UID: \"5e7b51de-bdfd-451c-a7f9-5230a947a1be\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm"
Sep 09 13:51:42.991735 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:42.991635    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ab8d99d-c6e6-4394-b978-93d2a23152f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv\" (UID: \"8ab8d99d-c6e6-4394-b978-93d2a23152f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv"
Sep 09 13:51:42.991735 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:42.991694    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6svd\" (UniqueName: \"kubernetes.io/projected/8ab8d99d-c6e6-4394-b978-93d2a23152f3-kube-api-access-g6svd\") pod \"payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv\" (UID: \"8ab8d99d-c6e6-4394-b978-93d2a23152f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv"
Sep 09 13:51:42.991735 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:42.991718    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5r4r\" (UniqueName: \"kubernetes.io/projected/5e7b51de-bdfd-451c-a7f9-5230a947a1be-kube-api-access-p5r4r\") pod \"payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm\" (UID: \"5e7b51de-bdfd-451c-a7f9-5230a947a1be\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm"
Sep 09 13:51:42.992282 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:42.992259    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e7b51de-bdfd-451c-a7f9-5230a947a1be-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm\" (UID: \"5e7b51de-bdfd-451c-a7f9-5230a947a1be\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm"
Sep 09 13:51:42.992377 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:42.992358    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ab8d99d-c6e6-4394-b978-93d2a23152f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv\" (UID: \"8ab8d99d-c6e6-4394-b978-93d2a23152f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv"
Sep 09 13:51:42.999525 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:42.999505    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5r4r\" (UniqueName: \"kubernetes.io/projected/5e7b51de-bdfd-451c-a7f9-5230a947a1be-kube-api-access-p5r4r\") pod \"payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm\" (UID: \"5e7b51de-bdfd-451c-a7f9-5230a947a1be\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm"
Sep 09 13:51:42.999645 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:42.999564    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6svd\" (UniqueName: \"kubernetes.io/projected/8ab8d99d-c6e6-4394-b978-93d2a23152f3-kube-api-access-g6svd\") pod \"payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv\" (UID: \"8ab8d99d-c6e6-4394-b978-93d2a23152f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv"
Sep 09 13:51:43.068062 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:43.067994    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv"
Sep 09 13:51:43.090753 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:43.090726    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm"
Sep 09 13:51:43.342276 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:43.342217    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv"]
Sep 09 13:51:43.344971 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:51:43.344947    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab8d99d_c6e6_4394_b978_93d2a23152f3.slice/crio-c35a1d81229e04ae5edea39659e006f8c0d818ff739e04744b556d58bc78df9a WatchSource:0}: Error finding container c35a1d81229e04ae5edea39659e006f8c0d818ff739e04744b556d58bc78df9a: Status 404 returned error can't find the container with id c35a1d81229e04ae5edea39659e006f8c0d818ff739e04744b556d58bc78df9a
Sep 09 13:51:43.370739 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:43.370717    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm"]
Sep 09 13:51:43.374879 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:51:43.374852    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e7b51de_bdfd_451c_a7f9_5230a947a1be.slice/crio-4c24bbe96f0e9dd4db6977563290144bc9d4f9195b33fb0f50c037b1e1d02864 WatchSource:0}: Error finding container 4c24bbe96f0e9dd4db6977563290144bc9d4f9195b33fb0f50c037b1e1d02864: Status 404 returned error can't find the container with id 4c24bbe96f0e9dd4db6977563290144bc9d4f9195b33fb0f50c037b1e1d02864
Sep 09 13:51:44.014045 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:44.014012    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm" event={"ID":"5e7b51de-bdfd-451c-a7f9-5230a947a1be","Type":"ContainerStarted","Data":"743dba0333a414ab9d9d8feb86207d84c7b98fde3be586a7b30e3ff2cb779857"}
Sep 09 13:51:44.014584 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:44.014068    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm" event={"ID":"5e7b51de-bdfd-451c-a7f9-5230a947a1be","Type":"ContainerStarted","Data":"4c24bbe96f0e9dd4db6977563290144bc9d4f9195b33fb0f50c037b1e1d02864"}
Sep 09 13:51:44.014584 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:44.014103    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm"
Sep 09 13:51:44.015770 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:44.015742    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv" event={"ID":"8ab8d99d-c6e6-4394-b978-93d2a23152f3","Type":"ContainerStarted","Data":"664a46b6f4fba5dbf79d34255dafa6cf04355bfdbff739f725763f78e90570f3"}
Sep 09 13:51:44.015910 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:44.015790    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv" event={"ID":"8ab8d99d-c6e6-4394-b978-93d2a23152f3","Type":"ContainerStarted","Data":"c35a1d81229e04ae5edea39659e006f8c0d818ff739e04744b556d58bc78df9a"}
Sep 09 13:51:44.015910 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:44.015863    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv"
Sep 09 13:51:44.029365 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:44.029322    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm" podStartSLOduration=2.029307331 podStartE2EDuration="2.029307331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:51:44.027878546 +0000 UTC m=+1537.387728375" watchObservedRunningTime="2026-09-09 13:51:44.029307331 +0000 UTC m=+1537.389157158"
Sep 09 13:51:44.043162 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:44.043123    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv" podStartSLOduration=2.043112879 podStartE2EDuration="2.043112879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:51:44.041526451 +0000 UTC m=+1537.401376280" watchObservedRunningTime="2026-09-09 13:51:44.043112879 +0000 UTC m=+1537.402962705"
Sep 09 13:51:55.024008 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:55.023972    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv"
Sep 09 13:51:55.024664 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:55.024634    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm"
Sep 09 13:51:59.097733 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:59.097698    2566 generic.go:356] "Generic (PLEG): container finished" podID="4352a0c7-cc5c-4572-a17c-f4bdc9775b5e" containerID="9aa82d5c6b43173a726a3e7af9f6a8947eb320c2c70fd43211c0e47a0d1567e9" exitCode=0
Sep 09 13:51:59.097733 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:59.097736    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf" event={"ID":"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e","Type":"ContainerDied","Data":"9aa82d5c6b43173a726a3e7af9f6a8947eb320c2c70fd43211c0e47a0d1567e9"}
Sep 09 13:51:59.098233 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:59.098122    2566 scope.go:117] "RemoveContainer" containerID="9aa82d5c6b43173a726a3e7af9f6a8947eb320c2c70fd43211c0e47a0d1567e9"
Sep 09 13:51:59.099167 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:59.099137    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:51:59.099261 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:59.099206    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:51:59.099261 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:51:59.099238    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 13:52:00.110163 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:00.110125    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf" event={"ID":"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e","Type":"ContainerStarted","Data":"cdf0085da7e27ce7b5ed778fe4e4eaabfbe6cca05090c435c8e2a033a0eeeaf0"}
Sep 09 13:52:00.804413 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:00.804379    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:52:00.804908 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:00.804879    2566 patch_prober.go:28] interesting pod/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 09 13:52:00.805104 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:00.805076    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf" podUID="4352a0c7-cc5c-4572-a17c-f4bdc9775b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 09 13:52:01.805134 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:01.805083    2566 patch_prober.go:28] interesting pod/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 09 13:52:01.805651 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:01.805160    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf" podUID="4352a0c7-cc5c-4572-a17c-f4bdc9775b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 09 13:52:02.805203 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:02.805156    2566 patch_prober.go:28] interesting pod/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 09 13:52:02.805689 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:02.805226    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf" podUID="4352a0c7-cc5c-4572-a17c-f4bdc9775b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 09 13:52:03.804764 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:03.804712    2566 patch_prober.go:28] interesting pod/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 09 13:52:03.805008 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:03.804794    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf" podUID="4352a0c7-cc5c-4572-a17c-f4bdc9775b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 09 13:52:04.804479 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:04.804433    2566 patch_prober.go:28] interesting pod/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Sep 09 13:52:04.804982 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:04.804518    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf" podUID="4352a0c7-cc5c-4572-a17c-f4bdc9775b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Sep 09 13:52:05.808240 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:05.808210    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:52:05.808602 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:05.808567    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:52:05.809284 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:05.809268    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:52:07.488591 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:07.488557    2566 scope.go:117] "RemoveContainer" containerID="59e6b58abdbbb070d08819c587b64fc687488b82eb160441f8a6ec78963341bf"
Sep 09 13:52:10.939966 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:10.939931    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm_5e7b51de-bdfd-451c-a7f9-5230a947a1be/payload-processing/0.log"
Sep 09 13:52:59.778684 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:59.778605    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"]
Sep 09 13:52:59.782095 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:59.782080    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"
Sep 09 13:52:59.784545 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:59.784524    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-8349bf\"/\"kube-root-ca.crt\""
Sep 09 13:52:59.785058 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:59.785040    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-8349bf\"/\"ipp-route-8349bf-kserve-self-signed-certs\""
Sep 09 13:52:59.785149 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:59.785055    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-8349bf\"/\"openshift-service-ca.crt\""
Sep 09 13:52:59.785149 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:59.785127    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-8349bf\"/\"default-dockercfg-zj4z9\""
Sep 09 13:52:59.795412 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:59.795387    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"]
Sep 09 13:52:59.906360 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:59.906322    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-dshm\") pod \"ipp-route-8349bf-kserve-594fc859c5-cn5ns\" (UID: \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\") " pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"
Sep 09 13:52:59.906360 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:59.906363    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n8n4\" (UniqueName: \"kubernetes.io/projected/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-kube-api-access-8n8n4\") pod \"ipp-route-8349bf-kserve-594fc859c5-cn5ns\" (UID: \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\") " pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"
Sep 09 13:52:59.906556 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:59.906449    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-tls-certs\") pod \"ipp-route-8349bf-kserve-594fc859c5-cn5ns\" (UID: \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\") " pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"
Sep 09 13:52:59.906556 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:59.906489    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-tmp-dir\") pod \"ipp-route-8349bf-kserve-594fc859c5-cn5ns\" (UID: \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\") " pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"
Sep 09 13:52:59.906556 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:59.906549    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-home\") pod \"ipp-route-8349bf-kserve-594fc859c5-cn5ns\" (UID: \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\") " pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"
Sep 09 13:52:59.906660 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:52:59.906594    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-model-cache\") pod \"ipp-route-8349bf-kserve-594fc859c5-cn5ns\" (UID: \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\") " pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"
Sep 09 13:53:00.007723 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:00.007691    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-model-cache\") pod \"ipp-route-8349bf-kserve-594fc859c5-cn5ns\" (UID: \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\") " pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"
Sep 09 13:53:00.007915 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:00.007749    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-dshm\") pod \"ipp-route-8349bf-kserve-594fc859c5-cn5ns\" (UID: \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\") " pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"
Sep 09 13:53:00.007915 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:00.007776    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n8n4\" (UniqueName: \"kubernetes.io/projected/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-kube-api-access-8n8n4\") pod \"ipp-route-8349bf-kserve-594fc859c5-cn5ns\" (UID: \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\") " pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"
Sep 09 13:53:00.007915 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:00.007834    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-tls-certs\") pod \"ipp-route-8349bf-kserve-594fc859c5-cn5ns\" (UID: \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\") " pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"
Sep 09 13:53:00.007915 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:00.007867    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-tmp-dir\") pod \"ipp-route-8349bf-kserve-594fc859c5-cn5ns\" (UID: \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\") " pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"
Sep 09 13:53:00.007915 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:00.007910    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-home\") pod \"ipp-route-8349bf-kserve-594fc859c5-cn5ns\" (UID: \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\") " pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"
Sep 09 13:53:00.008213 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:00.008163    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-model-cache\") pod \"ipp-route-8349bf-kserve-594fc859c5-cn5ns\" (UID: \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\") " pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"
Sep 09 13:53:00.008279 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:00.008241    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-home\") pod \"ipp-route-8349bf-kserve-594fc859c5-cn5ns\" (UID: \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\") " pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"
Sep 09 13:53:00.008320 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:00.008290    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-tmp-dir\") pod \"ipp-route-8349bf-kserve-594fc859c5-cn5ns\" (UID: \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\") " pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"
Sep 09 13:53:00.009999 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:00.009977    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-dshm\") pod \"ipp-route-8349bf-kserve-594fc859c5-cn5ns\" (UID: \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\") " pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"
Sep 09 13:53:00.010400 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:00.010379    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-tls-certs\") pod \"ipp-route-8349bf-kserve-594fc859c5-cn5ns\" (UID: \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\") " pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"
Sep 09 13:53:00.017294 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:00.017267    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n8n4\" (UniqueName: \"kubernetes.io/projected/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-kube-api-access-8n8n4\") pod \"ipp-route-8349bf-kserve-594fc859c5-cn5ns\" (UID: \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\") " pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"
Sep 09 13:53:00.091185 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:00.091154    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"
Sep 09 13:53:00.361062 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:00.361031    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"]
Sep 09 13:53:00.363149 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:53:00.363116    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod473e8cf5_5d7e_4510_8744_5bfdbcd815f8.slice/crio-c2058651163aaebe385ba556cdc6604420f7c3d5fd5369d523b5c15384c1c9b3 WatchSource:0}: Error finding container c2058651163aaebe385ba556cdc6604420f7c3d5fd5369d523b5c15384c1c9b3: Status 404 returned error can't find the container with id c2058651163aaebe385ba556cdc6604420f7c3d5fd5369d523b5c15384c1c9b3
Sep 09 13:53:00.417247 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:00.417215    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns" event={"ID":"473e8cf5-5d7e-4510-8744-5bfdbcd815f8","Type":"ContainerStarted","Data":"c2058651163aaebe385ba556cdc6604420f7c3d5fd5369d523b5c15384c1c9b3"}
Sep 09 13:53:01.424058 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:01.424024    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns" event={"ID":"473e8cf5-5d7e-4510-8744-5bfdbcd815f8","Type":"ContainerStarted","Data":"acf11fc4eed1d649f071054f1a14f27149acaa994992b58216afe28e0981d261"}
Sep 09 13:53:01.440686 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:01.440636    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns" podStartSLOduration=2.440622252 podStartE2EDuration="2.440622252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:53:01.438969975 +0000 UTC m=+1614.798819814" watchObservedRunningTime="2026-09-09 13:53:01.440622252 +0000 UTC m=+1614.800472078"
Sep 09 13:53:07.521646 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:07.521610    2566 scope.go:117] "RemoveContainer" containerID="52bcc10af7d46d51141a9761bdebea516aaa720c4bf2d0c9826ed318939d300d"
Sep 09 13:53:10.091851 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:10.091800    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"
Sep 09 13:53:10.091851 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:10.091860    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"
Sep 09 13:53:10.104561 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:10.104537    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"
Sep 09 13:53:10.484166 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:10.484080    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"
Sep 09 13:53:38.022447 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:38.022417    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm_5e7b51de-bdfd-451c-a7f9-5230a947a1be/payload-processing/0.log"
Sep 09 13:53:43.161236 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:43.161200    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp"]
Sep 09 13:53:43.164778 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:43.164761    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp"
Sep 09 13:53:43.167214 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:43.167183    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-b2555b-dockercfg-xr2ct\""
Sep 09 13:53:43.167332 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:43.167221    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-b2555b\""
Sep 09 13:53:43.174765 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:43.174743    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp"]
Sep 09 13:53:43.302719 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:43.302688    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91153c22-047a-4224-8aea-4d1153a2ccd2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp\" (UID: \"91153c22-047a-4224-8aea-4d1153a2ccd2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp"
Sep 09 13:53:43.302719 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:43.302727    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzcwq\" (UniqueName: \"kubernetes.io/projected/91153c22-047a-4224-8aea-4d1153a2ccd2-kube-api-access-mzcwq\") pod \"payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp\" (UID: \"91153c22-047a-4224-8aea-4d1153a2ccd2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp"
Sep 09 13:53:43.404076 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:43.404040    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91153c22-047a-4224-8aea-4d1153a2ccd2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp\" (UID: \"91153c22-047a-4224-8aea-4d1153a2ccd2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp"
Sep 09 13:53:43.404076 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:43.404080    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzcwq\" (UniqueName: \"kubernetes.io/projected/91153c22-047a-4224-8aea-4d1153a2ccd2-kube-api-access-mzcwq\") pod \"payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp\" (UID: \"91153c22-047a-4224-8aea-4d1153a2ccd2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp"
Sep 09 13:53:43.404732 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:43.404710    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91153c22-047a-4224-8aea-4d1153a2ccd2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp\" (UID: \"91153c22-047a-4224-8aea-4d1153a2ccd2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp"
Sep 09 13:53:43.412355 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:43.412285    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzcwq\" (UniqueName: \"kubernetes.io/projected/91153c22-047a-4224-8aea-4d1153a2ccd2-kube-api-access-mzcwq\") pod \"payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp\" (UID: \"91153c22-047a-4224-8aea-4d1153a2ccd2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp"
Sep 09 13:53:43.475194 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:43.475171    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp"
Sep 09 13:53:43.747112 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:43.747089    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp"]
Sep 09 13:53:43.749073 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:53:43.749043    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91153c22_047a_4224_8aea_4d1153a2ccd2.slice/crio-45a2f040bdc44076dfd9279494045854fc181fb821a0bb8099a4a1f4ba1455c9 WatchSource:0}: Error finding container 45a2f040bdc44076dfd9279494045854fc181fb821a0bb8099a4a1f4ba1455c9: Status 404 returned error can't find the container with id 45a2f040bdc44076dfd9279494045854fc181fb821a0bb8099a4a1f4ba1455c9
Sep 09 13:53:44.643739 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:44.643698    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp" event={"ID":"91153c22-047a-4224-8aea-4d1153a2ccd2","Type":"ContainerStarted","Data":"8b040efb5242c4fd9af8525776409a424fa84acecfab979045be11b8b1490320"}
Sep 09 13:53:44.644138 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:44.643762    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp" event={"ID":"91153c22-047a-4224-8aea-4d1153a2ccd2","Type":"ContainerStarted","Data":"45a2f040bdc44076dfd9279494045854fc181fb821a0bb8099a4a1f4ba1455c9"}
Sep 09 13:53:44.644138 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:44.643792    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp"
Sep 09 13:53:44.660851 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:44.660787    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp" podStartSLOduration=1.660772699 podStartE2EDuration="1.660772699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:53:44.659707815 +0000 UTC m=+1658.019557639" watchObservedRunningTime="2026-09-09 13:53:44.660772699 +0000 UTC m=+1658.020622524"
Sep 09 13:53:55.651579 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:53:55.651549    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp"
Sep 09 13:54:07.006421 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:07.006382    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2555b-dss76"]
Sep 09 13:54:07.010220 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:07.010192    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2555b-dss76"
Sep 09 13:54:07.012181 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:07.012159    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 13:54:07.012331 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:07.012315    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lh487\""
Sep 09 13:54:07.017734 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:07.017544    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2555b-dss76"]
Sep 09 13:54:07.030396 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:07.030366    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/098b75fa-f4eb-464f-8cf4-49b77947f4be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-b2555b-dss76\" (UID: \"098b75fa-f4eb-464f-8cf4-49b77947f4be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2555b-dss76"
Sep 09 13:54:07.131236 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:07.131202    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/098b75fa-f4eb-464f-8cf4-49b77947f4be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-b2555b-dss76\" (UID: \"098b75fa-f4eb-464f-8cf4-49b77947f4be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2555b-dss76"
Sep 09 13:54:07.133295 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:07.133273    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 13:54:07.142546 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:07.142518    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/098b75fa-f4eb-464f-8cf4-49b77947f4be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-b2555b-dss76\" (UID: \"098b75fa-f4eb-464f-8cf4-49b77947f4be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2555b-dss76"
Sep 09 13:54:07.324341 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:07.324260    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lh487\""
Sep 09 13:54:07.333000 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:07.332972    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2555b-dss76"
Sep 09 13:54:07.605778 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:07.605750    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2555b-dss76"]
Sep 09 13:54:07.608124 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:54:07.608096    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098b75fa_f4eb_464f_8cf4_49b77947f4be.slice/crio-b8898f457c23d49451eaa0f4a561740de3954d0dade1d98f4ad71b9a3c5194e4 WatchSource:0}: Error finding container b8898f457c23d49451eaa0f4a561740de3954d0dade1d98f4ad71b9a3c5194e4: Status 404 returned error can't find the container with id b8898f457c23d49451eaa0f4a561740de3954d0dade1d98f4ad71b9a3c5194e4
Sep 09 13:54:07.768277 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:07.768245    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2555b-dss76" event={"ID":"098b75fa-f4eb-464f-8cf4-49b77947f4be","Type":"ContainerStarted","Data":"b8898f457c23d49451eaa0f4a561740de3954d0dade1d98f4ad71b9a3c5194e4"}
Sep 09 13:54:08.774503 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:08.774469    2566 generic.go:356] "Generic (PLEG): container finished" podID="098b75fa-f4eb-464f-8cf4-49b77947f4be" containerID="697474a3d31ec09d65d8d6c6ff2a803bb6a2c4cce1f6a790b872a448e62c9e1e" exitCode=0
Sep 09 13:54:08.775040 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:08.774525    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2555b-dss76" event={"ID":"098b75fa-f4eb-464f-8cf4-49b77947f4be","Type":"ContainerDied","Data":"697474a3d31ec09d65d8d6c6ff2a803bb6a2c4cce1f6a790b872a448e62c9e1e"}
Sep 09 13:54:09.909727 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:09.909699    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2555b-dss76"
Sep 09 13:54:09.950475 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:09.950441    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/098b75fa-f4eb-464f-8cf4-49b77947f4be-maas-api-service-ca\") pod \"098b75fa-f4eb-464f-8cf4-49b77947f4be\" (UID: \"098b75fa-f4eb-464f-8cf4-49b77947f4be\") "
Sep 09 13:54:09.950881 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:09.950861    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/098b75fa-f4eb-464f-8cf4-49b77947f4be-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "098b75fa-f4eb-464f-8cf4-49b77947f4be" (UID: "098b75fa-f4eb-464f-8cf4-49b77947f4be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:54:10.052294 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:10.052194    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/098b75fa-f4eb-464f-8cf4-49b77947f4be-maas-api-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:54:10.785922 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:10.785894    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2555b-dss76"
Sep 09 13:54:10.786087 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:10.785891    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2555b-dss76" event={"ID":"098b75fa-f4eb-464f-8cf4-49b77947f4be","Type":"ContainerDied","Data":"b8898f457c23d49451eaa0f4a561740de3954d0dade1d98f4ad71b9a3c5194e4"}
Sep 09 13:54:10.786087 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:10.786031    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8898f457c23d49451eaa0f4a561740de3954d0dade1d98f4ad71b9a3c5194e4"
Sep 09 13:54:13.142063 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:13.142027    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp"]
Sep 09 13:54:13.142550 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:13.142314    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp" podUID="91153c22-047a-4224-8aea-4d1153a2ccd2" containerName="payload-processing" containerID="cri-o://8b040efb5242c4fd9af8525776409a424fa84acecfab979045be11b8b1490320" gracePeriod=30
Sep 09 13:54:13.305151 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:13.305107    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2555b-dss76"]
Sep 09 13:54:13.307610 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:13.307580    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2555b-dss76"]
Sep 09 13:54:13.393395 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:13.393323    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp"
Sep 09 13:54:13.484209 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:13.484173    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91153c22-047a-4224-8aea-4d1153a2ccd2-plugins-config-volume\") pod \"91153c22-047a-4224-8aea-4d1153a2ccd2\" (UID: \"91153c22-047a-4224-8aea-4d1153a2ccd2\") "
Sep 09 13:54:13.484404 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:13.484282    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzcwq\" (UniqueName: \"kubernetes.io/projected/91153c22-047a-4224-8aea-4d1153a2ccd2-kube-api-access-mzcwq\") pod \"91153c22-047a-4224-8aea-4d1153a2ccd2\" (UID: \"91153c22-047a-4224-8aea-4d1153a2ccd2\") "
Sep 09 13:54:13.484673 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:13.484639    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91153c22-047a-4224-8aea-4d1153a2ccd2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "91153c22-047a-4224-8aea-4d1153a2ccd2" (UID: "91153c22-047a-4224-8aea-4d1153a2ccd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:54:13.486529 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:13.486490    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91153c22-047a-4224-8aea-4d1153a2ccd2-kube-api-access-mzcwq" (OuterVolumeSpecName: "kube-api-access-mzcwq") pod "91153c22-047a-4224-8aea-4d1153a2ccd2" (UID: "91153c22-047a-4224-8aea-4d1153a2ccd2"). InnerVolumeSpecName "kube-api-access-mzcwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:54:13.585988 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:13.585956    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzcwq\" (UniqueName: \"kubernetes.io/projected/91153c22-047a-4224-8aea-4d1153a2ccd2-kube-api-access-mzcwq\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:54:13.585988 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:13.585982    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91153c22-047a-4224-8aea-4d1153a2ccd2-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:54:13.804862 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:13.804744    2566 generic.go:356] "Generic (PLEG): container finished" podID="91153c22-047a-4224-8aea-4d1153a2ccd2" containerID="8b040efb5242c4fd9af8525776409a424fa84acecfab979045be11b8b1490320" exitCode=0
Sep 09 13:54:13.804862 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:13.804792    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp"
Sep 09 13:54:13.804862 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:13.804850    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp" event={"ID":"91153c22-047a-4224-8aea-4d1153a2ccd2","Type":"ContainerDied","Data":"8b040efb5242c4fd9af8525776409a424fa84acecfab979045be11b8b1490320"}
Sep 09 13:54:13.805126 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:13.804904    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp" event={"ID":"91153c22-047a-4224-8aea-4d1153a2ccd2","Type":"ContainerDied","Data":"45a2f040bdc44076dfd9279494045854fc181fb821a0bb8099a4a1f4ba1455c9"}
Sep 09 13:54:13.805126 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:13.804929    2566 scope.go:117] "RemoveContainer" containerID="8b040efb5242c4fd9af8525776409a424fa84acecfab979045be11b8b1490320"
Sep 09 13:54:13.814753 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:13.814733    2566 scope.go:117] "RemoveContainer" containerID="8b040efb5242c4fd9af8525776409a424fa84acecfab979045be11b8b1490320"
Sep 09 13:54:13.815044 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:54:13.815026    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b040efb5242c4fd9af8525776409a424fa84acecfab979045be11b8b1490320\": container with ID starting with 8b040efb5242c4fd9af8525776409a424fa84acecfab979045be11b8b1490320 not found: ID does not exist" containerID="8b040efb5242c4fd9af8525776409a424fa84acecfab979045be11b8b1490320"
Sep 09 13:54:13.815115 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:13.815057    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b040efb5242c4fd9af8525776409a424fa84acecfab979045be11b8b1490320"} err="failed to get container status \"8b040efb5242c4fd9af8525776409a424fa84acecfab979045be11b8b1490320\": rpc error: code = NotFound desc = could not find container \"8b040efb5242c4fd9af8525776409a424fa84acecfab979045be11b8b1490320\": container with ID starting with 8b040efb5242c4fd9af8525776409a424fa84acecfab979045be11b8b1490320 not found: ID does not exist"
Sep 09 13:54:13.826383 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:13.826359    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp"]
Sep 09 13:54:13.829975 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:13.829954    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b2555b-5dd7676f6f-7pwdp"]
Sep 09 13:54:15.149538 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:15.149506    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="098b75fa-f4eb-464f-8cf4-49b77947f4be" path="/var/lib/kubelet/pods/098b75fa-f4eb-464f-8cf4-49b77947f4be/volumes"
Sep 09 13:54:15.149929 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:15.149900    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91153c22-047a-4224-8aea-4d1153a2ccd2" path="/var/lib/kubelet/pods/91153c22-047a-4224-8aea-4d1153a2ccd2/volumes"
Sep 09 13:54:23.252065 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:23.252029    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8349bf-snfz4"]
Sep 09 13:54:23.252425 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:23.252377    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="098b75fa-f4eb-464f-8cf4-49b77947f4be" containerName="revoke-keys"
Sep 09 13:54:23.252425 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:23.252388    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="098b75fa-f4eb-464f-8cf4-49b77947f4be" containerName="revoke-keys"
Sep 09 13:54:23.252425 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:23.252403    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91153c22-047a-4224-8aea-4d1153a2ccd2" containerName="payload-processing"
Sep 09 13:54:23.252425 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:23.252408    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="91153c22-047a-4224-8aea-4d1153a2ccd2" containerName="payload-processing"
Sep 09 13:54:23.252570 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:23.252476    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="098b75fa-f4eb-464f-8cf4-49b77947f4be" containerName="revoke-keys"
Sep 09 13:54:23.252570 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:23.252489    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="91153c22-047a-4224-8aea-4d1153a2ccd2" containerName="payload-processing"
Sep 09 13:54:23.256866 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:23.256847    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8349bf-snfz4"
Sep 09 13:54:23.259732 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:23.259712    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 13:54:23.259864 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:23.259713    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lh487\""
Sep 09 13:54:23.263021 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:23.262999    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8349bf-snfz4"]
Sep 09 13:54:23.379875 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:23.379795    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4be8970c-dfbf-4bd4-950c-09ad89ab667b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8349bf-snfz4\" (UID: \"4be8970c-dfbf-4bd4-950c-09ad89ab667b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8349bf-snfz4"
Sep 09 13:54:23.480738 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:23.480694    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4be8970c-dfbf-4bd4-950c-09ad89ab667b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8349bf-snfz4\" (UID: \"4be8970c-dfbf-4bd4-950c-09ad89ab667b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8349bf-snfz4"
Sep 09 13:54:23.481364 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:23.481344    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4be8970c-dfbf-4bd4-950c-09ad89ab667b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8349bf-snfz4\" (UID: \"4be8970c-dfbf-4bd4-950c-09ad89ab667b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8349bf-snfz4"
Sep 09 13:54:23.568945 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:23.568835    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8349bf-snfz4"
Sep 09 13:54:23.842803 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:23.842617    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8349bf-snfz4"]
Sep 09 13:54:23.845642 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:54:23.845613    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be8970c_dfbf_4bd4_950c_09ad89ab667b.slice/crio-d191d760edb86d793f64fa921939fc2b4c8f01179a901bd2cd1966b7b99ef50b WatchSource:0}: Error finding container d191d760edb86d793f64fa921939fc2b4c8f01179a901bd2cd1966b7b99ef50b: Status 404 returned error can't find the container with id d191d760edb86d793f64fa921939fc2b4c8f01179a901bd2cd1966b7b99ef50b
Sep 09 13:54:23.858045 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:23.858021    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8349bf-snfz4" event={"ID":"4be8970c-dfbf-4bd4-950c-09ad89ab667b","Type":"ContainerStarted","Data":"d191d760edb86d793f64fa921939fc2b4c8f01179a901bd2cd1966b7b99ef50b"}
Sep 09 13:54:24.865903 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:24.865865    2566 generic.go:356] "Generic (PLEG): container finished" podID="4be8970c-dfbf-4bd4-950c-09ad89ab667b" containerID="2ffb5dcf4e0c60c2870f57969b62173e4ef2db5106b47cf1bf00c10fb7093c40" exitCode=0
Sep 09 13:54:24.866358 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:24.865948    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8349bf-snfz4" event={"ID":"4be8970c-dfbf-4bd4-950c-09ad89ab667b","Type":"ContainerDied","Data":"2ffb5dcf4e0c60c2870f57969b62173e4ef2db5106b47cf1bf00c10fb7093c40"}
Sep 09 13:54:26.008378 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:26.008353    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8349bf-snfz4"
Sep 09 13:54:26.104421 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:26.104384    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4be8970c-dfbf-4bd4-950c-09ad89ab667b-maas-api-service-ca\") pod \"4be8970c-dfbf-4bd4-950c-09ad89ab667b\" (UID: \"4be8970c-dfbf-4bd4-950c-09ad89ab667b\") "
Sep 09 13:54:26.104758 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:26.104733    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be8970c-dfbf-4bd4-950c-09ad89ab667b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4be8970c-dfbf-4bd4-950c-09ad89ab667b" (UID: "4be8970c-dfbf-4bd4-950c-09ad89ab667b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:54:26.205993 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:26.205904    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4be8970c-dfbf-4bd4-950c-09ad89ab667b-maas-api-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:54:26.876856 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:26.876829    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8349bf-snfz4"
Sep 09 13:54:26.877030 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:26.876804    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8349bf-snfz4" event={"ID":"4be8970c-dfbf-4bd4-950c-09ad89ab667b","Type":"ContainerDied","Data":"d191d760edb86d793f64fa921939fc2b4c8f01179a901bd2cd1966b7b99ef50b"}
Sep 09 13:54:26.877030 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:26.876957    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d191d760edb86d793f64fa921939fc2b4c8f01179a901bd2cd1966b7b99ef50b"
Sep 09 13:54:38.931453 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:38.931372    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8349bf-snfz4"]
Sep 09 13:54:38.934114 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:38.934087    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8349bf-snfz4"]
Sep 09 13:54:39.149202 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:39.149159    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4be8970c-dfbf-4bd4-950c-09ad89ab667b" path="/var/lib/kubelet/pods/4be8970c-dfbf-4bd4-950c-09ad89ab667b/volumes"
Sep 09 13:54:44.283017 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:44.282980    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"]
Sep 09 13:54:59.305962 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:59.305921    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns" podUID="473e8cf5-5d7e-4510-8744-5bfdbcd815f8" containerName="main" containerID="cri-o://acf11fc4eed1d649f071054f1a14f27149acaa994992b58216afe28e0981d261" gracePeriod=45
Sep 09 13:54:59.544980 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:59.544956    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"
Sep 09 13:54:59.614280 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:59.614254    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-dshm\") pod \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\" (UID: \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\") "
Sep 09 13:54:59.614444 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:59.614287    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8n8n4\" (UniqueName: \"kubernetes.io/projected/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-kube-api-access-8n8n4\") pod \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\" (UID: \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\") "
Sep 09 13:54:59.614444 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:59.614325    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-model-cache\") pod \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\" (UID: \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\") "
Sep 09 13:54:59.614444 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:59.614408    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-tls-certs\") pod \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\" (UID: \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\") "
Sep 09 13:54:59.614444 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:59.614430    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-home\") pod \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\" (UID: \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\") "
Sep 09 13:54:59.614638 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:59.614470    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-tmp-dir\") pod \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\" (UID: \"473e8cf5-5d7e-4510-8744-5bfdbcd815f8\") "
Sep 09 13:54:59.614707 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:59.614655    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-model-cache" (OuterVolumeSpecName: "model-cache") pod "473e8cf5-5d7e-4510-8744-5bfdbcd815f8" (UID: "473e8cf5-5d7e-4510-8744-5bfdbcd815f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:54:59.614831 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:59.614779    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-home" (OuterVolumeSpecName: "home") pod "473e8cf5-5d7e-4510-8744-5bfdbcd815f8" (UID: "473e8cf5-5d7e-4510-8744-5bfdbcd815f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:54:59.614948 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:59.614923    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "473e8cf5-5d7e-4510-8744-5bfdbcd815f8" (UID: "473e8cf5-5d7e-4510-8744-5bfdbcd815f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:54:59.615022 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:59.614999    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:54:59.615079 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:59.615021    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:54:59.616584 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:59.616553    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-kube-api-access-8n8n4" (OuterVolumeSpecName: "kube-api-access-8n8n4") pod "473e8cf5-5d7e-4510-8744-5bfdbcd815f8" (UID: "473e8cf5-5d7e-4510-8744-5bfdbcd815f8"). InnerVolumeSpecName "kube-api-access-8n8n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:54:59.616840 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:59.616798    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-dshm" (OuterVolumeSpecName: "dshm") pod "473e8cf5-5d7e-4510-8744-5bfdbcd815f8" (UID: "473e8cf5-5d7e-4510-8744-5bfdbcd815f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:54:59.616907 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:59.616850    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "473e8cf5-5d7e-4510-8744-5bfdbcd815f8" (UID: "473e8cf5-5d7e-4510-8744-5bfdbcd815f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 13:54:59.715985 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:59.715954    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:54:59.715985 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:59.715979    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:54:59.715985 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:59.715990    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:54:59.716196 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:54:59.716002    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n8n4\" (UniqueName: \"kubernetes.io/projected/473e8cf5-5d7e-4510-8744-5bfdbcd815f8-kube-api-access-8n8n4\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:55:00.044474 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:00.044377    2566 generic.go:356] "Generic (PLEG): container finished" podID="473e8cf5-5d7e-4510-8744-5bfdbcd815f8" containerID="acf11fc4eed1d649f071054f1a14f27149acaa994992b58216afe28e0981d261" exitCode=0
Sep 09 13:55:00.044474 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:00.044441    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"
Sep 09 13:55:00.044655 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:00.044466    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns" event={"ID":"473e8cf5-5d7e-4510-8744-5bfdbcd815f8","Type":"ContainerDied","Data":"acf11fc4eed1d649f071054f1a14f27149acaa994992b58216afe28e0981d261"}
Sep 09 13:55:00.044655 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:00.044515    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns" event={"ID":"473e8cf5-5d7e-4510-8744-5bfdbcd815f8","Type":"ContainerDied","Data":"c2058651163aaebe385ba556cdc6604420f7c3d5fd5369d523b5c15384c1c9b3"}
Sep 09 13:55:00.044655 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:00.044528    2566 scope.go:117] "RemoveContainer" containerID="acf11fc4eed1d649f071054f1a14f27149acaa994992b58216afe28e0981d261"
Sep 09 13:55:00.053775 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:00.053760    2566 scope.go:117] "RemoveContainer" containerID="acf11fc4eed1d649f071054f1a14f27149acaa994992b58216afe28e0981d261"
Sep 09 13:55:00.054048 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:55:00.054033    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acf11fc4eed1d649f071054f1a14f27149acaa994992b58216afe28e0981d261\": container with ID starting with acf11fc4eed1d649f071054f1a14f27149acaa994992b58216afe28e0981d261 not found: ID does not exist" containerID="acf11fc4eed1d649f071054f1a14f27149acaa994992b58216afe28e0981d261"
Sep 09 13:55:00.054094 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:00.054055    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acf11fc4eed1d649f071054f1a14f27149acaa994992b58216afe28e0981d261"} err="failed to get container status \"acf11fc4eed1d649f071054f1a14f27149acaa994992b58216afe28e0981d261\": rpc error: code = NotFound desc = could not find container \"acf11fc4eed1d649f071054f1a14f27149acaa994992b58216afe28e0981d261\": container with ID starting with acf11fc4eed1d649f071054f1a14f27149acaa994992b58216afe28e0981d261 not found: ID does not exist"
Sep 09 13:55:00.065152 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:00.065129    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"]
Sep 09 13:55:00.069036 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:00.069016    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-8349bf/ipp-route-8349bf-kserve-594fc859c5-cn5ns"]
Sep 09 13:55:01.149523 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:01.149489    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="473e8cf5-5d7e-4510-8744-5bfdbcd815f8" path="/var/lib/kubelet/pods/473e8cf5-5d7e-4510-8744-5bfdbcd815f8/volumes"
Sep 09 13:55:27.769560 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:27.769515    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3691cf-jcp6r"]
Sep 09 13:55:27.770436 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:27.770410    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="473e8cf5-5d7e-4510-8744-5bfdbcd815f8" containerName="main"
Sep 09 13:55:27.770436 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:27.770439    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="473e8cf5-5d7e-4510-8744-5bfdbcd815f8" containerName="main"
Sep 09 13:55:27.770588 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:27.770496    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4be8970c-dfbf-4bd4-950c-09ad89ab667b" containerName="revoke-keys"
Sep 09 13:55:27.770588 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:27.770504    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4be8970c-dfbf-4bd4-950c-09ad89ab667b" containerName="revoke-keys"
Sep 09 13:55:27.770691 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:27.770672    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4be8970c-dfbf-4bd4-950c-09ad89ab667b" containerName="revoke-keys"
Sep 09 13:55:27.770691 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:27.770687    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="473e8cf5-5d7e-4510-8744-5bfdbcd815f8" containerName="main"
Sep 09 13:55:27.774701 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:27.774677    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3691cf-jcp6r"
Sep 09 13:55:27.777523 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:27.777499    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 13:55:27.777824 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:27.777788    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lh487\""
Sep 09 13:55:27.780260 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:27.780237    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3691cf-jcp6r"]
Sep 09 13:55:27.876775 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:27.876743    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a38730-c8ef-4010-9980-826211aab422-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3691cf-jcp6r\" (UID: \"e8a38730-c8ef-4010-9980-826211aab422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3691cf-jcp6r"
Sep 09 13:55:27.977940 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:27.977905    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a38730-c8ef-4010-9980-826211aab422-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3691cf-jcp6r\" (UID: \"e8a38730-c8ef-4010-9980-826211aab422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3691cf-jcp6r"
Sep 09 13:55:27.978547 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:27.978528    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a38730-c8ef-4010-9980-826211aab422-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3691cf-jcp6r\" (UID: \"e8a38730-c8ef-4010-9980-826211aab422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3691cf-jcp6r"
Sep 09 13:55:28.087800 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:28.087777    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3691cf-jcp6r"
Sep 09 13:55:28.360863 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:28.360841    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3691cf-jcp6r"]
Sep 09 13:55:28.363366 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:55:28.363341    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a38730_c8ef_4010_9980_826211aab422.slice/crio-df850b028093e6f2074d10c78595e27e151834e49b9cab2a75725a88863a280f WatchSource:0}: Error finding container df850b028093e6f2074d10c78595e27e151834e49b9cab2a75725a88863a280f: Status 404 returned error can't find the container with id df850b028093e6f2074d10c78595e27e151834e49b9cab2a75725a88863a280f
Sep 09 13:55:29.195211 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:29.195179    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3691cf-jcp6r" event={"ID":"e8a38730-c8ef-4010-9980-826211aab422","Type":"ContainerStarted","Data":"16cb1b58403c52a526ca3549a487250d2ed31ebb57b18fc49f53de25b67488d0"}
Sep 09 13:55:29.195612 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:29.195240    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3691cf-jcp6r" event={"ID":"e8a38730-c8ef-4010-9980-826211aab422","Type":"ContainerStarted","Data":"df850b028093e6f2074d10c78595e27e151834e49b9cab2a75725a88863a280f"}
Sep 09 13:55:29.216358 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:29.216310    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3691cf-jcp6r" podStartSLOduration=2.216293794 podStartE2EDuration="2.216293794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:55:29.214065339 +0000 UTC m=+1762.573915166" watchObservedRunningTime="2026-09-09 13:55:29.216293794 +0000 UTC m=+1762.576143622"
Sep 09 13:55:30.201255 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:30.201218    2566 generic.go:356] "Generic (PLEG): container finished" podID="e8a38730-c8ef-4010-9980-826211aab422" containerID="16cb1b58403c52a526ca3549a487250d2ed31ebb57b18fc49f53de25b67488d0" exitCode=0
Sep 09 13:55:30.201692 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:30.201303    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3691cf-jcp6r" event={"ID":"e8a38730-c8ef-4010-9980-826211aab422","Type":"ContainerDied","Data":"16cb1b58403c52a526ca3549a487250d2ed31ebb57b18fc49f53de25b67488d0"}
Sep 09 13:55:31.335845 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:31.335806    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3691cf-jcp6r"
Sep 09 13:55:31.509261 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:31.509181    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a38730-c8ef-4010-9980-826211aab422-maas-api-service-ca\") pod \"e8a38730-c8ef-4010-9980-826211aab422\" (UID: \"e8a38730-c8ef-4010-9980-826211aab422\") "
Sep 09 13:55:31.509509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:31.509488    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a38730-c8ef-4010-9980-826211aab422-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e8a38730-c8ef-4010-9980-826211aab422" (UID: "e8a38730-c8ef-4010-9980-826211aab422"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:55:31.610273 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:31.610242    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a38730-c8ef-4010-9980-826211aab422-maas-api-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:55:32.212557 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:32.212528    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3691cf-jcp6r"
Sep 09 13:55:32.212557 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:32.212542    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3691cf-jcp6r" event={"ID":"e8a38730-c8ef-4010-9980-826211aab422","Type":"ContainerDied","Data":"df850b028093e6f2074d10c78595e27e151834e49b9cab2a75725a88863a280f"}
Sep 09 13:55:32.212846 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:32.212583    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df850b028093e6f2074d10c78595e27e151834e49b9cab2a75725a88863a280f"
Sep 09 13:55:38.346997 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:38.346960    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm"]
Sep 09 13:55:38.347476 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:38.347213    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm" podUID="5e7b51de-bdfd-451c-a7f9-5230a947a1be" containerName="payload-processing" containerID="cri-o://743dba0333a414ab9d9d8feb86207d84c7b98fde3be586a7b30e3ff2cb779857" gracePeriod=30
Sep 09 13:55:38.353447 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:38.353416    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv"]
Sep 09 13:55:38.353666 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:38.353635    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv" podUID="8ab8d99d-c6e6-4394-b978-93d2a23152f3" containerName="payload-pre-processing" containerID="cri-o://664a46b6f4fba5dbf79d34255dafa6cf04355bfdbff739f725763f78e90570f3" gracePeriod=30
Sep 09 13:55:38.522389 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:38.522324    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3691cf-jcp6r"]
Sep 09 13:55:38.526736 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:38.526701    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3691cf-jcp6r"]
Sep 09 13:55:38.624449 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:38.624424    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm"
Sep 09 13:55:38.627182 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:38.627164    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv"
Sep 09 13:55:38.774235 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:38.774199    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5r4r\" (UniqueName: \"kubernetes.io/projected/5e7b51de-bdfd-451c-a7f9-5230a947a1be-kube-api-access-p5r4r\") pod \"5e7b51de-bdfd-451c-a7f9-5230a947a1be\" (UID: \"5e7b51de-bdfd-451c-a7f9-5230a947a1be\") "
Sep 09 13:55:38.774423 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:38.774258    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ab8d99d-c6e6-4394-b978-93d2a23152f3-plugins-config-volume\") pod \"8ab8d99d-c6e6-4394-b978-93d2a23152f3\" (UID: \"8ab8d99d-c6e6-4394-b978-93d2a23152f3\") "
Sep 09 13:55:38.774423 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:38.774279    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6svd\" (UniqueName: \"kubernetes.io/projected/8ab8d99d-c6e6-4394-b978-93d2a23152f3-kube-api-access-g6svd\") pod \"8ab8d99d-c6e6-4394-b978-93d2a23152f3\" (UID: \"8ab8d99d-c6e6-4394-b978-93d2a23152f3\") "
Sep 09 13:55:38.774423 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:38.774303    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e7b51de-bdfd-451c-a7f9-5230a947a1be-plugins-config-volume\") pod \"5e7b51de-bdfd-451c-a7f9-5230a947a1be\" (UID: \"5e7b51de-bdfd-451c-a7f9-5230a947a1be\") "
Sep 09 13:55:38.774767 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:38.774731    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e7b51de-bdfd-451c-a7f9-5230a947a1be-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5e7b51de-bdfd-451c-a7f9-5230a947a1be" (UID: "5e7b51de-bdfd-451c-a7f9-5230a947a1be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:55:38.774767 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:38.774720    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab8d99d-c6e6-4394-b978-93d2a23152f3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8ab8d99d-c6e6-4394-b978-93d2a23152f3" (UID: "8ab8d99d-c6e6-4394-b978-93d2a23152f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:55:38.776499 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:38.776477    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab8d99d-c6e6-4394-b978-93d2a23152f3-kube-api-access-g6svd" (OuterVolumeSpecName: "kube-api-access-g6svd") pod "8ab8d99d-c6e6-4394-b978-93d2a23152f3" (UID: "8ab8d99d-c6e6-4394-b978-93d2a23152f3"). InnerVolumeSpecName "kube-api-access-g6svd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:55:38.776596 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:38.776510    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e7b51de-bdfd-451c-a7f9-5230a947a1be-kube-api-access-p5r4r" (OuterVolumeSpecName: "kube-api-access-p5r4r") pod "5e7b51de-bdfd-451c-a7f9-5230a947a1be" (UID: "5e7b51de-bdfd-451c-a7f9-5230a947a1be"). InnerVolumeSpecName "kube-api-access-p5r4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:55:38.875706 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:38.875622    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5r4r\" (UniqueName: \"kubernetes.io/projected/5e7b51de-bdfd-451c-a7f9-5230a947a1be-kube-api-access-p5r4r\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:55:38.875706 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:38.875651    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ab8d99d-c6e6-4394-b978-93d2a23152f3-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:55:38.875706 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:38.875665    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6svd\" (UniqueName: \"kubernetes.io/projected/8ab8d99d-c6e6-4394-b978-93d2a23152f3-kube-api-access-g6svd\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:55:38.875706 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:38.875677    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e7b51de-bdfd-451c-a7f9-5230a947a1be-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:55:39.149942 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:39.149855    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8a38730-c8ef-4010-9980-826211aab422" path="/var/lib/kubelet/pods/e8a38730-c8ef-4010-9980-826211aab422/volumes"
Sep 09 13:55:39.248641 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:39.248605    2566 generic.go:356] "Generic (PLEG): container finished" podID="5e7b51de-bdfd-451c-a7f9-5230a947a1be" containerID="743dba0333a414ab9d9d8feb86207d84c7b98fde3be586a7b30e3ff2cb779857" exitCode=0
Sep 09 13:55:39.248641 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:39.248641    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm"
Sep 09 13:55:39.248871 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:39.248684    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm" event={"ID":"5e7b51de-bdfd-451c-a7f9-5230a947a1be","Type":"ContainerDied","Data":"743dba0333a414ab9d9d8feb86207d84c7b98fde3be586a7b30e3ff2cb779857"}
Sep 09 13:55:39.248871 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:39.248731    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm" event={"ID":"5e7b51de-bdfd-451c-a7f9-5230a947a1be","Type":"ContainerDied","Data":"4c24bbe96f0e9dd4db6977563290144bc9d4f9195b33fb0f50c037b1e1d02864"}
Sep 09 13:55:39.248871 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:39.248754    2566 scope.go:117] "RemoveContainer" containerID="743dba0333a414ab9d9d8feb86207d84c7b98fde3be586a7b30e3ff2cb779857"
Sep 09 13:55:39.250287 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:39.250257    2566 generic.go:356] "Generic (PLEG): container finished" podID="8ab8d99d-c6e6-4394-b978-93d2a23152f3" containerID="664a46b6f4fba5dbf79d34255dafa6cf04355bfdbff739f725763f78e90570f3" exitCode=0
Sep 09 13:55:39.250392 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:39.250304    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv"
Sep 09 13:55:39.250392 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:39.250320    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv" event={"ID":"8ab8d99d-c6e6-4394-b978-93d2a23152f3","Type":"ContainerDied","Data":"664a46b6f4fba5dbf79d34255dafa6cf04355bfdbff739f725763f78e90570f3"}
Sep 09 13:55:39.250392 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:39.250357    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv" event={"ID":"8ab8d99d-c6e6-4394-b978-93d2a23152f3","Type":"ContainerDied","Data":"c35a1d81229e04ae5edea39659e006f8c0d818ff739e04744b556d58bc78df9a"}
Sep 09 13:55:39.257653 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:39.257637    2566 scope.go:117] "RemoveContainer" containerID="743dba0333a414ab9d9d8feb86207d84c7b98fde3be586a7b30e3ff2cb779857"
Sep 09 13:55:39.257917 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:55:39.257900    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"743dba0333a414ab9d9d8feb86207d84c7b98fde3be586a7b30e3ff2cb779857\": container with ID starting with 743dba0333a414ab9d9d8feb86207d84c7b98fde3be586a7b30e3ff2cb779857 not found: ID does not exist" containerID="743dba0333a414ab9d9d8feb86207d84c7b98fde3be586a7b30e3ff2cb779857"
Sep 09 13:55:39.257974 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:39.257926    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"743dba0333a414ab9d9d8feb86207d84c7b98fde3be586a7b30e3ff2cb779857"} err="failed to get container status \"743dba0333a414ab9d9d8feb86207d84c7b98fde3be586a7b30e3ff2cb779857\": rpc error: code = NotFound desc = could not find container \"743dba0333a414ab9d9d8feb86207d84c7b98fde3be586a7b30e3ff2cb779857\": container with ID starting with 743dba0333a414ab9d9d8feb86207d84c7b98fde3be586a7b30e3ff2cb779857 not found: ID does not exist"
Sep 09 13:55:39.257974 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:39.257943    2566 scope.go:117] "RemoveContainer" containerID="664a46b6f4fba5dbf79d34255dafa6cf04355bfdbff739f725763f78e90570f3"
Sep 09 13:55:39.265984 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:39.265965    2566 scope.go:117] "RemoveContainer" containerID="664a46b6f4fba5dbf79d34255dafa6cf04355bfdbff739f725763f78e90570f3"
Sep 09 13:55:39.266246 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:55:39.266228    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"664a46b6f4fba5dbf79d34255dafa6cf04355bfdbff739f725763f78e90570f3\": container with ID starting with 664a46b6f4fba5dbf79d34255dafa6cf04355bfdbff739f725763f78e90570f3 not found: ID does not exist" containerID="664a46b6f4fba5dbf79d34255dafa6cf04355bfdbff739f725763f78e90570f3"
Sep 09 13:55:39.266310 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:39.266253    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"664a46b6f4fba5dbf79d34255dafa6cf04355bfdbff739f725763f78e90570f3"} err="failed to get container status \"664a46b6f4fba5dbf79d34255dafa6cf04355bfdbff739f725763f78e90570f3\": rpc error: code = NotFound desc = could not find container \"664a46b6f4fba5dbf79d34255dafa6cf04355bfdbff739f725763f78e90570f3\": container with ID starting with 664a46b6f4fba5dbf79d34255dafa6cf04355bfdbff739f725763f78e90570f3 not found: ID does not exist"
Sep 09 13:55:39.266435 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:39.266420    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm"]
Sep 09 13:55:39.270404 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:39.270384    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3691cf-54b4d7895f-s6wnm"]
Sep 09 13:55:39.288052 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:39.288027    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv"]
Sep 09 13:55:39.291572 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:39.291554    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3691cf-6899df46d6-knfnv"]
Sep 09 13:55:41.149797 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:41.149764    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e7b51de-bdfd-451c-a7f9-5230a947a1be" path="/var/lib/kubelet/pods/5e7b51de-bdfd-451c-a7f9-5230a947a1be/volumes"
Sep 09 13:55:41.150198 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:41.150134    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ab8d99d-c6e6-4394-b978-93d2a23152f3" path="/var/lib/kubelet/pods/8ab8d99d-c6e6-4394-b978-93d2a23152f3/volumes"
Sep 09 13:55:44.988321 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:44.988279    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"]
Sep 09 13:55:44.988708 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:44.988481    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf" podUID="4352a0c7-cc5c-4572-a17c-f4bdc9775b5e" containerName="istio-proxy" containerID="cri-o://cdf0085da7e27ce7b5ed778fe4e4eaabfbe6cca05090c435c8e2a033a0eeeaf0" gracePeriod=30
Sep 09 13:55:45.289945 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:45.289869    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f2df2-v9fsx"]
Sep 09 13:55:45.290283 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:45.290246    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e7b51de-bdfd-451c-a7f9-5230a947a1be" containerName="payload-processing"
Sep 09 13:55:45.290283 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:45.290281    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e7b51de-bdfd-451c-a7f9-5230a947a1be" containerName="payload-processing"
Sep 09 13:55:45.290439 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:45.290296    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ab8d99d-c6e6-4394-b978-93d2a23152f3" containerName="payload-pre-processing"
Sep 09 13:55:45.290439 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:45.290302    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ab8d99d-c6e6-4394-b978-93d2a23152f3" containerName="payload-pre-processing"
Sep 09 13:55:45.290439 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:45.290322    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8a38730-c8ef-4010-9980-826211aab422" containerName="revoke-keys"
Sep 09 13:55:45.290439 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:45.290327    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8a38730-c8ef-4010-9980-826211aab422" containerName="revoke-keys"
Sep 09 13:55:45.290439 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:45.290380    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e7b51de-bdfd-451c-a7f9-5230a947a1be" containerName="payload-processing"
Sep 09 13:55:45.290439 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:45.290388    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ab8d99d-c6e6-4394-b978-93d2a23152f3" containerName="payload-pre-processing"
Sep 09 13:55:45.290439 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:45.290395    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8a38730-c8ef-4010-9980-826211aab422" containerName="revoke-keys"
Sep 09 13:55:45.293715 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:45.293696    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f2df2-v9fsx"
Sep 09 13:55:45.296064 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:45.296044    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 13:55:45.296225 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:45.296052    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lh487\""
Sep 09 13:55:45.298606 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:45.298584    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f2df2-v9fsx"]
Sep 09 13:55:45.331787 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:45.331751    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b162366-768e-4de7-82e6-0388560ed00a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8f2df2-v9fsx\" (UID: \"1b162366-768e-4de7-82e6-0388560ed00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f2df2-v9fsx"
Sep 09 13:55:45.433169 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:45.433137    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b162366-768e-4de7-82e6-0388560ed00a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8f2df2-v9fsx\" (UID: \"1b162366-768e-4de7-82e6-0388560ed00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f2df2-v9fsx"
Sep 09 13:55:45.433723 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:45.433705    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b162366-768e-4de7-82e6-0388560ed00a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8f2df2-v9fsx\" (UID: \"1b162366-768e-4de7-82e6-0388560ed00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f2df2-v9fsx"
Sep 09 13:55:45.606554 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:45.606525    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f2df2-v9fsx"
Sep 09 13:55:45.874467 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:45.874390    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f2df2-v9fsx"]
Sep 09 13:55:45.877564 ip-10-0-129-84 kubenswrapper[2566]: W0909 13:55:45.877538    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b162366_768e_4de7_82e6_0388560ed00a.slice/crio-5e97f24113fd70b20eb3a2cb5202bde33709455252ef8c0e4a101a41818a95f5 WatchSource:0}: Error finding container 5e97f24113fd70b20eb3a2cb5202bde33709455252ef8c0e4a101a41818a95f5: Status 404 returned error can't find the container with id 5e97f24113fd70b20eb3a2cb5202bde33709455252ef8c0e4a101a41818a95f5
Sep 09 13:55:46.297257 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:46.297224    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f2df2-v9fsx" event={"ID":"1b162366-768e-4de7-82e6-0388560ed00a","Type":"ContainerStarted","Data":"83b5025dcc49c7f65850ab29c674eb306f49d5867f683a85565ca87db96f8693"}
Sep 09 13:55:46.297631 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:46.297283    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f2df2-v9fsx" event={"ID":"1b162366-768e-4de7-82e6-0388560ed00a","Type":"ContainerStarted","Data":"5e97f24113fd70b20eb3a2cb5202bde33709455252ef8c0e4a101a41818a95f5"}
Sep 09 13:55:46.312019 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:46.311973    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f2df2-v9fsx" podStartSLOduration=1.311957346 podStartE2EDuration="1.311957346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 13:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 13:55:46.310316628 +0000 UTC m=+1779.670166453" watchObservedRunningTime="2026-09-09 13:55:46.311957346 +0000 UTC m=+1779.671807171"
Sep 09 13:55:47.303954 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:47.303910    2566 generic.go:356] "Generic (PLEG): container finished" podID="1b162366-768e-4de7-82e6-0388560ed00a" containerID="83b5025dcc49c7f65850ab29c674eb306f49d5867f683a85565ca87db96f8693" exitCode=0
Sep 09 13:55:47.304346 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:47.303990    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f2df2-v9fsx" event={"ID":"1b162366-768e-4de7-82e6-0388560ed00a","Type":"ContainerDied","Data":"83b5025dcc49c7f65850ab29c674eb306f49d5867f683a85565ca87db96f8693"}
Sep 09 13:55:48.444729 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:48.444698    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f2df2-v9fsx"
Sep 09 13:55:48.563217 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:48.563141    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b162366-768e-4de7-82e6-0388560ed00a-maas-api-service-ca\") pod \"1b162366-768e-4de7-82e6-0388560ed00a\" (UID: \"1b162366-768e-4de7-82e6-0388560ed00a\") "
Sep 09 13:55:48.563477 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:48.563456    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b162366-768e-4de7-82e6-0388560ed00a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1b162366-768e-4de7-82e6-0388560ed00a" (UID: "1b162366-768e-4de7-82e6-0388560ed00a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:55:48.664743 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:48.664710    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b162366-768e-4de7-82e6-0388560ed00a-maas-api-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:55:49.315149 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:49.315124    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f2df2-v9fsx"
Sep 09 13:55:49.315323 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:49.315163    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f2df2-v9fsx" event={"ID":"1b162366-768e-4de7-82e6-0388560ed00a","Type":"ContainerDied","Data":"5e97f24113fd70b20eb3a2cb5202bde33709455252ef8c0e4a101a41818a95f5"}
Sep 09 13:55:49.315323 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:49.315206    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e97f24113fd70b20eb3a2cb5202bde33709455252ef8c0e4a101a41818a95f5"
Sep 09 13:55:50.239515 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.239490    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:55:50.280583 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.280505    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istiod-ca-cert\") pod \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") "
Sep 09 13:55:50.280583 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.280547    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-credential-socket\") pod \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") "
Sep 09 13:55:50.280583 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.280580    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istio-podinfo\") pod \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") "
Sep 09 13:55:50.280841 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.280605    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istio-token\") pod \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") "
Sep 09 13:55:50.280841 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.280637    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-workload-certs\") pod \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") "
Sep 09 13:55:50.280841 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.280663    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istio-envoy\") pod \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") "
Sep 09 13:55:50.280841 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.280702    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vz8m\" (UniqueName: \"kubernetes.io/projected/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-kube-api-access-4vz8m\") pod \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") "
Sep 09 13:55:50.280841 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.280746    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istio-data\") pod \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") "
Sep 09 13:55:50.280841 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.280793    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-workload-socket\") pod \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\" (UID: \"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e\") "
Sep 09 13:55:50.281268 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.281230    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4352a0c7-cc5c-4572-a17c-f4bdc9775b5e" (UID: "4352a0c7-cc5c-4572-a17c-f4bdc9775b5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:55:50.281268 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.281249    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4352a0c7-cc5c-4572-a17c-f4bdc9775b5e" (UID: "4352a0c7-cc5c-4572-a17c-f4bdc9775b5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:55:50.281461 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.281350    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4352a0c7-cc5c-4572-a17c-f4bdc9775b5e" (UID: "4352a0c7-cc5c-4572-a17c-f4bdc9775b5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:55:50.281461 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.281416    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istio-data" (OuterVolumeSpecName: "istio-data") pod "4352a0c7-cc5c-4572-a17c-f4bdc9775b5e" (UID: "4352a0c7-cc5c-4572-a17c-f4bdc9775b5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:55:50.281461 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.281450    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4352a0c7-cc5c-4572-a17c-f4bdc9775b5e" (UID: "4352a0c7-cc5c-4572-a17c-f4bdc9775b5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:55:50.283073 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.283043    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istio-token" (OuterVolumeSpecName: "istio-token") pod "4352a0c7-cc5c-4572-a17c-f4bdc9775b5e" (UID: "4352a0c7-cc5c-4572-a17c-f4bdc9775b5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:55:50.283156 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.283122    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4352a0c7-cc5c-4572-a17c-f4bdc9775b5e" (UID: "4352a0c7-cc5c-4572-a17c-f4bdc9775b5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:55:50.283156 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.283126    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-kube-api-access-4vz8m" (OuterVolumeSpecName: "kube-api-access-4vz8m") pod "4352a0c7-cc5c-4572-a17c-f4bdc9775b5e" (UID: "4352a0c7-cc5c-4572-a17c-f4bdc9775b5e"). InnerVolumeSpecName "kube-api-access-4vz8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:55:50.283532 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.283513    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4352a0c7-cc5c-4572-a17c-f4bdc9775b5e" (UID: "4352a0c7-cc5c-4572-a17c-f4bdc9775b5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 13:55:50.322024 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.321997    2566 generic.go:356] "Generic (PLEG): container finished" podID="4352a0c7-cc5c-4572-a17c-f4bdc9775b5e" containerID="cdf0085da7e27ce7b5ed778fe4e4eaabfbe6cca05090c435c8e2a033a0eeeaf0" exitCode=0
Sep 09 13:55:50.322184 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.322045    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"
Sep 09 13:55:50.322184 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.322088    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf" event={"ID":"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e","Type":"ContainerDied","Data":"cdf0085da7e27ce7b5ed778fe4e4eaabfbe6cca05090c435c8e2a033a0eeeaf0"}
Sep 09 13:55:50.322184 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.322149    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf" event={"ID":"4352a0c7-cc5c-4572-a17c-f4bdc9775b5e","Type":"ContainerDied","Data":"a34058a1827247b3cbf605f07fd003b310657d67cbe6adeebb2e579d784b79b8"}
Sep 09 13:55:50.322184 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.322178    2566 scope.go:117] "RemoveContainer" containerID="cdf0085da7e27ce7b5ed778fe4e4eaabfbe6cca05090c435c8e2a033a0eeeaf0"
Sep 09 13:55:50.331726 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.331709    2566 scope.go:117] "RemoveContainer" containerID="9aa82d5c6b43173a726a3e7af9f6a8947eb320c2c70fd43211c0e47a0d1567e9"
Sep 09 13:55:50.339791 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.339774    2566 scope.go:117] "RemoveContainer" containerID="cdf0085da7e27ce7b5ed778fe4e4eaabfbe6cca05090c435c8e2a033a0eeeaf0"
Sep 09 13:55:50.340106 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:55:50.340084    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdf0085da7e27ce7b5ed778fe4e4eaabfbe6cca05090c435c8e2a033a0eeeaf0\": container with ID starting with cdf0085da7e27ce7b5ed778fe4e4eaabfbe6cca05090c435c8e2a033a0eeeaf0 not found: ID does not exist" containerID="cdf0085da7e27ce7b5ed778fe4e4eaabfbe6cca05090c435c8e2a033a0eeeaf0"
Sep 09 13:55:50.340190 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.340117    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf0085da7e27ce7b5ed778fe4e4eaabfbe6cca05090c435c8e2a033a0eeeaf0"} err="failed to get container status \"cdf0085da7e27ce7b5ed778fe4e4eaabfbe6cca05090c435c8e2a033a0eeeaf0\": rpc error: code = NotFound desc = could not find container \"cdf0085da7e27ce7b5ed778fe4e4eaabfbe6cca05090c435c8e2a033a0eeeaf0\": container with ID starting with cdf0085da7e27ce7b5ed778fe4e4eaabfbe6cca05090c435c8e2a033a0eeeaf0 not found: ID does not exist"
Sep 09 13:55:50.340190 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.340140    2566 scope.go:117] "RemoveContainer" containerID="9aa82d5c6b43173a726a3e7af9f6a8947eb320c2c70fd43211c0e47a0d1567e9"
Sep 09 13:55:50.340395 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:55:50.340377    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aa82d5c6b43173a726a3e7af9f6a8947eb320c2c70fd43211c0e47a0d1567e9\": container with ID starting with 9aa82d5c6b43173a726a3e7af9f6a8947eb320c2c70fd43211c0e47a0d1567e9 not found: ID does not exist" containerID="9aa82d5c6b43173a726a3e7af9f6a8947eb320c2c70fd43211c0e47a0d1567e9"
Sep 09 13:55:50.340431 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.340400    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aa82d5c6b43173a726a3e7af9f6a8947eb320c2c70fd43211c0e47a0d1567e9"} err="failed to get container status \"9aa82d5c6b43173a726a3e7af9f6a8947eb320c2c70fd43211c0e47a0d1567e9\": rpc error: code = NotFound desc = could not find container \"9aa82d5c6b43173a726a3e7af9f6a8947eb320c2c70fd43211c0e47a0d1567e9\": container with ID starting with 9aa82d5c6b43173a726a3e7af9f6a8947eb320c2c70fd43211c0e47a0d1567e9 not found: ID does not exist"
Sep 09 13:55:50.344295 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.344274    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"]
Sep 09 13:55:50.348005 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.347985    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3691cf-openshift-default-79bcf64c76-5qhqf"]
Sep 09 13:55:50.382570 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.382548    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:55:50.382570 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.382567    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:55:50.382690 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.382575    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:55:50.382690 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.382584    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vz8m\" (UniqueName: \"kubernetes.io/projected/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-kube-api-access-4vz8m\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:55:50.382690 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.382593    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:55:50.382690 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.382602    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:55:50.382690 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.382609    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:55:50.382690 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.382617    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:55:50.382690 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:50.382625    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:55:51.149458 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:51.149427    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4352a0c7-cc5c-4572-a17c-f4bdc9775b5e" path="/var/lib/kubelet/pods/4352a0c7-cc5c-4572-a17c-f4bdc9775b5e/volumes"
Sep 09 13:55:55.786831 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:55.786763    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf"]
Sep 09 13:55:55.787230 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:55.787044    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf" podUID="c7f39f3f-c016-4b65-a3a3-17aebb601d4e" containerName="payload-processing" containerID="cri-o://3f4cb56a907a4cf8f45dd736ee67c0b85acf178c05e5c405298b444c486a0e7e" gracePeriod=30
Sep 09 13:55:55.943556 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:55.943523    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f2df2-v9fsx"]
Sep 09 13:55:55.946522 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:55.946498    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8f2df2-v9fsx"]
Sep 09 13:55:56.040287 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:56.040218    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf"
Sep 09 13:55:56.132105 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:56.132077    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rds56\" (UniqueName: \"kubernetes.io/projected/c7f39f3f-c016-4b65-a3a3-17aebb601d4e-kube-api-access-rds56\") pod \"c7f39f3f-c016-4b65-a3a3-17aebb601d4e\" (UID: \"c7f39f3f-c016-4b65-a3a3-17aebb601d4e\") "
Sep 09 13:55:56.132262 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:56.132113    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7f39f3f-c016-4b65-a3a3-17aebb601d4e-plugins-config-volume\") pod \"c7f39f3f-c016-4b65-a3a3-17aebb601d4e\" (UID: \"c7f39f3f-c016-4b65-a3a3-17aebb601d4e\") "
Sep 09 13:55:56.132526 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:56.132504    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f39f3f-c016-4b65-a3a3-17aebb601d4e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c7f39f3f-c016-4b65-a3a3-17aebb601d4e" (UID: "c7f39f3f-c016-4b65-a3a3-17aebb601d4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:55:56.134155 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:56.134135    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f39f3f-c016-4b65-a3a3-17aebb601d4e-kube-api-access-rds56" (OuterVolumeSpecName: "kube-api-access-rds56") pod "c7f39f3f-c016-4b65-a3a3-17aebb601d4e" (UID: "c7f39f3f-c016-4b65-a3a3-17aebb601d4e"). InnerVolumeSpecName "kube-api-access-rds56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:55:56.233068 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:56.233033    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rds56\" (UniqueName: \"kubernetes.io/projected/c7f39f3f-c016-4b65-a3a3-17aebb601d4e-kube-api-access-rds56\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:55:56.233068 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:56.233060    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7f39f3f-c016-4b65-a3a3-17aebb601d4e-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:55:56.355079 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:56.355038    2566 generic.go:356] "Generic (PLEG): container finished" podID="c7f39f3f-c016-4b65-a3a3-17aebb601d4e" containerID="3f4cb56a907a4cf8f45dd736ee67c0b85acf178c05e5c405298b444c486a0e7e" exitCode=0
Sep 09 13:55:56.355271 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:56.355083    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf"
Sep 09 13:55:56.355271 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:56.355094    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf" event={"ID":"c7f39f3f-c016-4b65-a3a3-17aebb601d4e","Type":"ContainerDied","Data":"3f4cb56a907a4cf8f45dd736ee67c0b85acf178c05e5c405298b444c486a0e7e"}
Sep 09 13:55:56.355271 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:56.355149    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf" event={"ID":"c7f39f3f-c016-4b65-a3a3-17aebb601d4e","Type":"ContainerDied","Data":"c8c7132d5b77bf45be60842ee87b058103c552f5aa1164fa4af4e8e71ef1a88b"}
Sep 09 13:55:56.355271 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:56.355173    2566 scope.go:117] "RemoveContainer" containerID="3f4cb56a907a4cf8f45dd736ee67c0b85acf178c05e5c405298b444c486a0e7e"
Sep 09 13:55:56.364525 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:56.364505    2566 scope.go:117] "RemoveContainer" containerID="3f4cb56a907a4cf8f45dd736ee67c0b85acf178c05e5c405298b444c486a0e7e"
Sep 09 13:55:56.364775 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:55:56.364758    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f4cb56a907a4cf8f45dd736ee67c0b85acf178c05e5c405298b444c486a0e7e\": container with ID starting with 3f4cb56a907a4cf8f45dd736ee67c0b85acf178c05e5c405298b444c486a0e7e not found: ID does not exist" containerID="3f4cb56a907a4cf8f45dd736ee67c0b85acf178c05e5c405298b444c486a0e7e"
Sep 09 13:55:56.364905 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:56.364782    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f4cb56a907a4cf8f45dd736ee67c0b85acf178c05e5c405298b444c486a0e7e"} err="failed to get container status \"3f4cb56a907a4cf8f45dd736ee67c0b85acf178c05e5c405298b444c486a0e7e\": rpc error: code = NotFound desc = could not find container \"3f4cb56a907a4cf8f45dd736ee67c0b85acf178c05e5c405298b444c486a0e7e\": container with ID starting with 3f4cb56a907a4cf8f45dd736ee67c0b85acf178c05e5c405298b444c486a0e7e not found: ID does not exist"
Sep 09 13:55:56.376050 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:56.376029    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf"]
Sep 09 13:55:56.379625 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:56.379606    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8f2df2-7ff596d467-9dtrf"]
Sep 09 13:55:57.149426 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:57.149395    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b162366-768e-4de7-82e6-0388560ed00a" path="/var/lib/kubelet/pods/1b162366-768e-4de7-82e6-0388560ed00a/volumes"
Sep 09 13:55:57.149784 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:55:57.149743    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7f39f3f-c016-4b65-a3a3-17aebb601d4e" path="/var/lib/kubelet/pods/c7f39f3f-c016-4b65-a3a3-17aebb601d4e/volumes"
Sep 09 13:56:07.236015 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:07.235987    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ftm96_af902e8b-d693-4341-9979-03cf69548cfb/console-operator/2.log"
Sep 09 13:56:07.237572 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:07.237549    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ftm96_af902e8b-d693-4341-9979-03cf69548cfb/console-operator/2.log"
Sep 09 13:56:07.565336 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:07.565243    2566 scope.go:117] "RemoveContainer" containerID="6f05e0e0a283907592716917c65fd31f7ec69b3b3f2957e42a95fe20932ebaa5"
Sep 09 13:56:13.306464 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:13.306430    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-df48ae-29816025-chk84"]
Sep 09 13:56:13.309240 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:13.309213    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-df48ae-29816025-chk84"]
Sep 09 13:56:13.346120 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:13.346091    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc"]
Sep 09 13:56:13.346345 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:13.346314    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc" podUID="892980ec-4634-41a6-8b28-f80043e08558" containerName="payload-processing" containerID="cri-o://cd159c2ee842a458ef365381612b5b7db31976e097f19f96fe06dd1c53936915" gracePeriod=30
Sep 09 13:56:13.593617 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:13.593588    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc"
Sep 09 13:56:13.691948 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:13.691912    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/892980ec-4634-41a6-8b28-f80043e08558-plugins-config-volume\") pod \"892980ec-4634-41a6-8b28-f80043e08558\" (UID: \"892980ec-4634-41a6-8b28-f80043e08558\") "
Sep 09 13:56:13.692118 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:13.692015    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srzng\" (UniqueName: \"kubernetes.io/projected/892980ec-4634-41a6-8b28-f80043e08558-kube-api-access-srzng\") pod \"892980ec-4634-41a6-8b28-f80043e08558\" (UID: \"892980ec-4634-41a6-8b28-f80043e08558\") "
Sep 09 13:56:13.692306 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:13.692279    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/892980ec-4634-41a6-8b28-f80043e08558-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "892980ec-4634-41a6-8b28-f80043e08558" (UID: "892980ec-4634-41a6-8b28-f80043e08558"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:56:13.694135 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:13.694101    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892980ec-4634-41a6-8b28-f80043e08558-kube-api-access-srzng" (OuterVolumeSpecName: "kube-api-access-srzng") pod "892980ec-4634-41a6-8b28-f80043e08558" (UID: "892980ec-4634-41a6-8b28-f80043e08558"). InnerVolumeSpecName "kube-api-access-srzng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:56:13.793366 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:13.793334    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srzng\" (UniqueName: \"kubernetes.io/projected/892980ec-4634-41a6-8b28-f80043e08558-kube-api-access-srzng\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:56:13.793366 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:13.793363    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/892980ec-4634-41a6-8b28-f80043e08558-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:56:14.446508 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:14.446472    2566 generic.go:356] "Generic (PLEG): container finished" podID="892980ec-4634-41a6-8b28-f80043e08558" containerID="cd159c2ee842a458ef365381612b5b7db31976e097f19f96fe06dd1c53936915" exitCode=0
Sep 09 13:56:14.446979 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:14.446513    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc"
Sep 09 13:56:14.446979 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:14.446558    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc" event={"ID":"892980ec-4634-41a6-8b28-f80043e08558","Type":"ContainerDied","Data":"cd159c2ee842a458ef365381612b5b7db31976e097f19f96fe06dd1c53936915"}
Sep 09 13:56:14.446979 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:14.446622    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc" event={"ID":"892980ec-4634-41a6-8b28-f80043e08558","Type":"ContainerDied","Data":"6496d7f79620e24b340dacf8f92be57d50ee72164c2fc95aa605fa76538d3b0a"}
Sep 09 13:56:14.446979 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:14.446658    2566 scope.go:117] "RemoveContainer" containerID="cd159c2ee842a458ef365381612b5b7db31976e097f19f96fe06dd1c53936915"
Sep 09 13:56:14.456135 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:14.456119    2566 scope.go:117] "RemoveContainer" containerID="cd159c2ee842a458ef365381612b5b7db31976e097f19f96fe06dd1c53936915"
Sep 09 13:56:14.456387 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:56:14.456370    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd159c2ee842a458ef365381612b5b7db31976e097f19f96fe06dd1c53936915\": container with ID starting with cd159c2ee842a458ef365381612b5b7db31976e097f19f96fe06dd1c53936915 not found: ID does not exist" containerID="cd159c2ee842a458ef365381612b5b7db31976e097f19f96fe06dd1c53936915"
Sep 09 13:56:14.456432 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:14.456395    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd159c2ee842a458ef365381612b5b7db31976e097f19f96fe06dd1c53936915"} err="failed to get container status \"cd159c2ee842a458ef365381612b5b7db31976e097f19f96fe06dd1c53936915\": rpc error: code = NotFound desc = could not find container \"cd159c2ee842a458ef365381612b5b7db31976e097f19f96fe06dd1c53936915\": container with ID starting with cd159c2ee842a458ef365381612b5b7db31976e097f19f96fe06dd1c53936915 not found: ID does not exist"
Sep 09 13:56:14.472018 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:14.471995    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc"]
Sep 09 13:56:14.480836 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:14.480577    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-df48ae-6db675bc85-fgvdc"]
Sep 09 13:56:15.149135 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:15.149100    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14f8a9d3-4a2a-4b1c-a4c4-dbd562c488bc" path="/var/lib/kubelet/pods/14f8a9d3-4a2a-4b1c-a4c4-dbd562c488bc/volumes"
Sep 09 13:56:15.149492 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:15.149478    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="892980ec-4634-41a6-8b28-f80043e08558" path="/var/lib/kubelet/pods/892980ec-4634-41a6-8b28-f80043e08558/volumes"
Sep 09 13:56:20.416437 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:20.416398    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"]
Sep 09 13:56:20.416926 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:20.416629    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg" podUID="2f6020ae-98a0-4ea4-b95a-32e247ab0c32" containerName="istio-proxy" containerID="cri-o://a4d3b13a6352aff9309d7a225166a598ceacc4a364970f94517cc412bc1f6847" gracePeriod=30
Sep 09 13:56:25.671783 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:25.671757    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:56:25.700809 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:25.700733    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-workload-socket\") pod \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") "
Sep 09 13:56:25.700809 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:25.700776    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istio-data\") pod \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") "
Sep 09 13:56:25.700809 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:25.700800    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-workload-certs\") pod \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") "
Sep 09 13:56:25.701115 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:25.700865    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istiod-ca-cert\") pod \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") "
Sep 09 13:56:25.701115 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:25.700896    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-credential-socket\") pod \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") "
Sep 09 13:56:25.701115 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:25.700952    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p46bp\" (UniqueName: \"kubernetes.io/projected/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-kube-api-access-p46bp\") pod \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") "
Sep 09 13:56:25.701115 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:25.701065    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2f6020ae-98a0-4ea4-b95a-32e247ab0c32" (UID: "2f6020ae-98a0-4ea4-b95a-32e247ab0c32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:56:25.701115 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:25.701079    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istio-data" (OuterVolumeSpecName: "istio-data") pod "2f6020ae-98a0-4ea4-b95a-32e247ab0c32" (UID: "2f6020ae-98a0-4ea4-b95a-32e247ab0c32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:56:25.701115 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:25.701104    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istio-podinfo\") pod \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") "
Sep 09 13:56:25.701413 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:25.701172    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istio-envoy\") pod \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") "
Sep 09 13:56:25.701413 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:25.701232    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istio-token\") pod \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\" (UID: \"2f6020ae-98a0-4ea4-b95a-32e247ab0c32\") "
Sep 09 13:56:25.701413 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:25.701307    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2f6020ae-98a0-4ea4-b95a-32e247ab0c32" (UID: "2f6020ae-98a0-4ea4-b95a-32e247ab0c32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:56:25.701413 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:25.701314    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2f6020ae-98a0-4ea4-b95a-32e247ab0c32" (UID: "2f6020ae-98a0-4ea4-b95a-32e247ab0c32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:56:25.701618 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:25.701518    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:56:25.701618 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:25.701537    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:56:25.701618 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:25.701551    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:56:25.701618 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:25.701563    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:56:25.701834 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:25.701683    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2f6020ae-98a0-4ea4-b95a-32e247ab0c32" (UID: "2f6020ae-98a0-4ea4-b95a-32e247ab0c32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 13:56:25.703435 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:25.703405    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-kube-api-access-p46bp" (OuterVolumeSpecName: "kube-api-access-p46bp") pod "2f6020ae-98a0-4ea4-b95a-32e247ab0c32" (UID: "2f6020ae-98a0-4ea4-b95a-32e247ab0c32"). InnerVolumeSpecName "kube-api-access-p46bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:56:25.703635 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:25.703600    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2f6020ae-98a0-4ea4-b95a-32e247ab0c32" (UID: "2f6020ae-98a0-4ea4-b95a-32e247ab0c32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 13:56:25.703907 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:25.703886    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istio-token" (OuterVolumeSpecName: "istio-token") pod "2f6020ae-98a0-4ea4-b95a-32e247ab0c32" (UID: "2f6020ae-98a0-4ea4-b95a-32e247ab0c32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 13:56:25.703996 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:25.703940    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2f6020ae-98a0-4ea4-b95a-32e247ab0c32" (UID: "2f6020ae-98a0-4ea4-b95a-32e247ab0c32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 13:56:25.802109 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:25.802075    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:56:25.802109 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:25.802104    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:56:25.802314 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:25.802114    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p46bp\" (UniqueName: \"kubernetes.io/projected/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-kube-api-access-p46bp\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:56:25.802314 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:25.802143    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:56:25.802314 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:25.802154    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f6020ae-98a0-4ea4-b95a-32e247ab0c32-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 13:56:26.505868 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:26.505808    2566 generic.go:356] "Generic (PLEG): container finished" podID="2f6020ae-98a0-4ea4-b95a-32e247ab0c32" containerID="a4d3b13a6352aff9309d7a225166a598ceacc4a364970f94517cc412bc1f6847" exitCode=0
Sep 09 13:56:26.506049 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:26.505877    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"
Sep 09 13:56:26.506049 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:26.505910    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg" event={"ID":"2f6020ae-98a0-4ea4-b95a-32e247ab0c32","Type":"ContainerDied","Data":"a4d3b13a6352aff9309d7a225166a598ceacc4a364970f94517cc412bc1f6847"}
Sep 09 13:56:26.506049 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:26.505964    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg" event={"ID":"2f6020ae-98a0-4ea4-b95a-32e247ab0c32","Type":"ContainerDied","Data":"cbab4f2f4c8da61d8a3f1c429e9c039d7af003d26eff5e496794cb4f7d2c5edd"}
Sep 09 13:56:26.506049 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:26.505987    2566 scope.go:117] "RemoveContainer" containerID="a4d3b13a6352aff9309d7a225166a598ceacc4a364970f94517cc412bc1f6847"
Sep 09 13:56:26.515095 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:26.515077    2566 scope.go:117] "RemoveContainer" containerID="4d4c0f39e8fcbb9360b8ed4c30bf7663ae9c1da3f663692b015968acc2e0dbc1"
Sep 09 13:56:26.523181 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:26.523165    2566 scope.go:117] "RemoveContainer" containerID="a4d3b13a6352aff9309d7a225166a598ceacc4a364970f94517cc412bc1f6847"
Sep 09 13:56:26.523459 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:56:26.523438    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4d3b13a6352aff9309d7a225166a598ceacc4a364970f94517cc412bc1f6847\": container with ID starting with a4d3b13a6352aff9309d7a225166a598ceacc4a364970f94517cc412bc1f6847 not found: ID does not exist" containerID="a4d3b13a6352aff9309d7a225166a598ceacc4a364970f94517cc412bc1f6847"
Sep 09 13:56:26.523509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:26.523468    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4d3b13a6352aff9309d7a225166a598ceacc4a364970f94517cc412bc1f6847"} err="failed to get container status \"a4d3b13a6352aff9309d7a225166a598ceacc4a364970f94517cc412bc1f6847\": rpc error: code = NotFound desc = could not find container \"a4d3b13a6352aff9309d7a225166a598ceacc4a364970f94517cc412bc1f6847\": container with ID starting with a4d3b13a6352aff9309d7a225166a598ceacc4a364970f94517cc412bc1f6847 not found: ID does not exist"
Sep 09 13:56:26.523509 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:26.523485    2566 scope.go:117] "RemoveContainer" containerID="4d4c0f39e8fcbb9360b8ed4c30bf7663ae9c1da3f663692b015968acc2e0dbc1"
Sep 09 13:56:26.523710 ip-10-0-129-84 kubenswrapper[2566]: E0909 13:56:26.523692    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d4c0f39e8fcbb9360b8ed4c30bf7663ae9c1da3f663692b015968acc2e0dbc1\": container with ID starting with 4d4c0f39e8fcbb9360b8ed4c30bf7663ae9c1da3f663692b015968acc2e0dbc1 not found: ID does not exist" containerID="4d4c0f39e8fcbb9360b8ed4c30bf7663ae9c1da3f663692b015968acc2e0dbc1"
Sep 09 13:56:26.523751 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:26.523715    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d4c0f39e8fcbb9360b8ed4c30bf7663ae9c1da3f663692b015968acc2e0dbc1"} err="failed to get container status \"4d4c0f39e8fcbb9360b8ed4c30bf7663ae9c1da3f663692b015968acc2e0dbc1\": rpc error: code = NotFound desc = could not find container \"4d4c0f39e8fcbb9360b8ed4c30bf7663ae9c1da3f663692b015968acc2e0dbc1\": container with ID starting with 4d4c0f39e8fcbb9360b8ed4c30bf7663ae9c1da3f663692b015968acc2e0dbc1 not found: ID does not exist"
Sep 09 13:56:26.531013 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:26.530987    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"]
Sep 09 13:56:26.535328 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:26.535303    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-df48ae-openshift-default-6bf9cf46cd-hdpsg"]
Sep 09 13:56:27.156271 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:56:27.156233    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f6020ae-98a0-4ea4-b95a-32e247ab0c32" path="/var/lib/kubelet/pods/2f6020ae-98a0-4ea4-b95a-32e247ab0c32/volumes"
Sep 09 13:57:07.599884 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:57:07.599849    2566 scope.go:117] "RemoveContainer" containerID="9ee976955e67a7404a963d087339f920388b5e4c7ae770eabefeda5b78b44965"
Sep 09 13:57:07.608059 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:57:07.608038    2566 scope.go:117] "RemoveContainer" containerID="544eef58c858872b65a835154c94153008af759c72a366236bf7cf461de6fe55"
Sep 09 13:57:07.615309 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:57:07.615290    2566 scope.go:117] "RemoveContainer" containerID="b84f114f9f79b226d38b1601decccc65333c79999f08b9ee21b64009e98b23d4"
Sep 09 13:57:07.622201 ip-10-0-129-84 kubenswrapper[2566]: I0909 13:57:07.622184    2566 scope.go:117] "RemoveContainer" containerID="e468b87e0377a451caf339b85d141bc88b37f5f835933a2f90c3ff4e4c4c1744"
Sep 09 14:00:07.673984 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:07.673947    2566 scope.go:117] "RemoveContainer" containerID="697474a3d31ec09d65d8d6c6ff2a803bb6a2c4cce1f6a790b872a448e62c9e1e"
Sep 09 14:00:58.802923 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.802890    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"]
Sep 09 14:00:58.803368 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.803210    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7f39f3f-c016-4b65-a3a3-17aebb601d4e" containerName="payload-processing"
Sep 09 14:00:58.803368 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.803221    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7f39f3f-c016-4b65-a3a3-17aebb601d4e" containerName="payload-processing"
Sep 09 14:00:58.803368 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.803232    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="892980ec-4634-41a6-8b28-f80043e08558" containerName="payload-processing"
Sep 09 14:00:58.803368 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.803237    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="892980ec-4634-41a6-8b28-f80043e08558" containerName="payload-processing"
Sep 09 14:00:58.803368 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.803244    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4352a0c7-cc5c-4572-a17c-f4bdc9775b5e" containerName="istio-proxy"
Sep 09 14:00:58.803368 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.803251    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4352a0c7-cc5c-4572-a17c-f4bdc9775b5e" containerName="istio-proxy"
Sep 09 14:00:58.803368 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.803257    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4352a0c7-cc5c-4572-a17c-f4bdc9775b5e" containerName="istio-proxy"
Sep 09 14:00:58.803368 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.803261    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4352a0c7-cc5c-4572-a17c-f4bdc9775b5e" containerName="istio-proxy"
Sep 09 14:00:58.803368 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.803278    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b162366-768e-4de7-82e6-0388560ed00a" containerName="revoke-keys"
Sep 09 14:00:58.803368 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.803283    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b162366-768e-4de7-82e6-0388560ed00a" containerName="revoke-keys"
Sep 09 14:00:58.803368 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.803291    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f6020ae-98a0-4ea4-b95a-32e247ab0c32" containerName="istio-proxy"
Sep 09 14:00:58.803368 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.803296    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f6020ae-98a0-4ea4-b95a-32e247ab0c32" containerName="istio-proxy"
Sep 09 14:00:58.803368 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.803306    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f6020ae-98a0-4ea4-b95a-32e247ab0c32" containerName="istio-proxy"
Sep 09 14:00:58.803368 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.803311    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f6020ae-98a0-4ea4-b95a-32e247ab0c32" containerName="istio-proxy"
Sep 09 14:00:58.803368 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.803359    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4352a0c7-cc5c-4572-a17c-f4bdc9775b5e" containerName="istio-proxy"
Sep 09 14:00:58.803368 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.803369    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4352a0c7-cc5c-4572-a17c-f4bdc9775b5e" containerName="istio-proxy"
Sep 09 14:00:58.803368 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.803375    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b162366-768e-4de7-82e6-0388560ed00a" containerName="revoke-keys"
Sep 09 14:00:58.803925 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.803382    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f6020ae-98a0-4ea4-b95a-32e247ab0c32" containerName="istio-proxy"
Sep 09 14:00:58.803925 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.803387    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="892980ec-4634-41a6-8b28-f80043e08558" containerName="payload-processing"
Sep 09 14:00:58.803925 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.803392    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f6020ae-98a0-4ea4-b95a-32e247ab0c32" containerName="istio-proxy"
Sep 09 14:00:58.803925 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.803401    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7f39f3f-c016-4b65-a3a3-17aebb601d4e" containerName="payload-processing"
Sep 09 14:00:58.806248 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.806232    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"
Sep 09 14:00:58.823305 ip-10-0-129-84 kubenswrapper[2566]: E0909 14:00:58.823278    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n\" is forbidden: User \"system:node:ip-10-0-129-84.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"llm\": no relationship found between node 'ip-10-0-129-84.ec2.internal' and this object" podUID="51ba5c5b-7708-4c97-af37-62b1500cdc4c" pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"
Sep 09 14:00:58.823455 ip-10-0-129-84 kubenswrapper[2566]: E0909 14:00:58.823310    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"e2e-same-id-a-00f6feed-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-129-84.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"llm\": no relationship found between node 'ip-10-0-129-84.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"llm\"/\"e2e-same-id-a-00f6feed-kserve-self-signed-certs\"" type="*v1.Secret"
Sep 09 14:00:58.877029 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.877000    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"]
Sep 09 14:00:58.910597 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.910565    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51ba5c5b-7708-4c97-af37-62b1500cdc4c-tls-certs\") pod \"e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n\" (UID: \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\") " pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"
Sep 09 14:00:58.910720 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.910617    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n588p\" (UniqueName: \"kubernetes.io/projected/51ba5c5b-7708-4c97-af37-62b1500cdc4c-kube-api-access-n588p\") pod \"e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n\" (UID: \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\") " pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"
Sep 09 14:00:58.910720 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.910646    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51ba5c5b-7708-4c97-af37-62b1500cdc4c-dshm\") pod \"e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n\" (UID: \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\") " pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"
Sep 09 14:00:58.910838 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.910736    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51ba5c5b-7708-4c97-af37-62b1500cdc4c-tmp-dir\") pod \"e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n\" (UID: \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\") " pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"
Sep 09 14:00:58.910838 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.910802    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51ba5c5b-7708-4c97-af37-62b1500cdc4c-model-cache\") pod \"e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n\" (UID: \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\") " pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"
Sep 09 14:00:58.910922 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:58.910899    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51ba5c5b-7708-4c97-af37-62b1500cdc4c-home\") pod \"e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n\" (UID: \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\") " pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"
Sep 09 14:00:59.012060 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:59.012028    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n588p\" (UniqueName: \"kubernetes.io/projected/51ba5c5b-7708-4c97-af37-62b1500cdc4c-kube-api-access-n588p\") pod \"e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n\" (UID: \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\") " pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"
Sep 09 14:00:59.012060 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:59.012063    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51ba5c5b-7708-4c97-af37-62b1500cdc4c-dshm\") pod \"e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n\" (UID: \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\") " pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"
Sep 09 14:00:59.012270 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:59.012088    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51ba5c5b-7708-4c97-af37-62b1500cdc4c-tmp-dir\") pod \"e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n\" (UID: \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\") " pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"
Sep 09 14:00:59.012270 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:59.012113    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51ba5c5b-7708-4c97-af37-62b1500cdc4c-model-cache\") pod \"e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n\" (UID: \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\") " pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"
Sep 09 14:00:59.012270 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:59.012149    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51ba5c5b-7708-4c97-af37-62b1500cdc4c-home\") pod \"e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n\" (UID: \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\") " pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"
Sep 09 14:00:59.012270 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:59.012179    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51ba5c5b-7708-4c97-af37-62b1500cdc4c-tls-certs\") pod \"e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n\" (UID: \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\") " pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"
Sep 09 14:00:59.012505 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:59.012478    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51ba5c5b-7708-4c97-af37-62b1500cdc4c-tmp-dir\") pod \"e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n\" (UID: \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\") " pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"
Sep 09 14:00:59.012565 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:59.012489    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51ba5c5b-7708-4c97-af37-62b1500cdc4c-model-cache\") pod \"e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n\" (UID: \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\") " pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"
Sep 09 14:00:59.012617 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:59.012600    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51ba5c5b-7708-4c97-af37-62b1500cdc4c-home\") pod \"e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n\" (UID: \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\") " pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"
Sep 09 14:00:59.014368 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:59.014351    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51ba5c5b-7708-4c97-af37-62b1500cdc4c-dshm\") pod \"e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n\" (UID: \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\") " pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"
Sep 09 14:00:59.047353 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:59.047331    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n588p\" (UniqueName: \"kubernetes.io/projected/51ba5c5b-7708-4c97-af37-62b1500cdc4c-kube-api-access-n588p\") pod \"e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n\" (UID: \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\") " pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"
Sep 09 14:00:59.994400 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:00:59.994366    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-00f6feed-kserve-self-signed-certs\""
Sep 09 14:01:00.005534 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:00.005511    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51ba5c5b-7708-4c97-af37-62b1500cdc4c-tls-certs\") pod \"e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n\" (UID: \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\") " pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"
Sep 09 14:01:00.015038 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:00.015016    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"
Sep 09 14:01:00.347534 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:00.347502    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"]
Sep 09 14:01:00.781406 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:00.781313    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n" event={"ID":"51ba5c5b-7708-4c97-af37-62b1500cdc4c","Type":"ContainerStarted","Data":"c694793223869da604f05ef183e98a19e151aefb70867e4aa729af47792f62d5"}
Sep 09 14:01:00.781406 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:00.781358    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n" event={"ID":"51ba5c5b-7708-4c97-af37-62b1500cdc4c","Type":"ContainerStarted","Data":"9cef9643e149c0c403bd9e7300bd55155989d9ece72e09ca96c949f9954c73b3"}
Sep 09 14:01:00.819624 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:00.819576    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n" podStartSLOduration=2.819561169 podStartE2EDuration="2.819561169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:01:00.819341889 +0000 UTC m=+2094.179191716" watchObservedRunningTime="2026-09-09 14:01:00.819561169 +0000 UTC m=+2094.179411027"
Sep 09 14:01:07.272215 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:07.272183    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ftm96_af902e8b-d693-4341-9979-03cf69548cfb/console-operator/2.log"
Sep 09 14:01:07.272215 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:07.272209    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ftm96_af902e8b-d693-4341-9979-03cf69548cfb/console-operator/2.log"
Sep 09 14:01:07.691964 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:07.691934    2566 scope.go:117] "RemoveContainer" containerID="2ffb5dcf4e0c60c2870f57969b62173e4ef2db5106b47cf1bf00c10fb7093c40"
Sep 09 14:01:10.015445 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:10.015400    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"
Sep 09 14:01:10.015915 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:10.015463    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"
Sep 09 14:01:10.028398 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:10.028376    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"
Sep 09 14:01:10.837788 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:10.837747    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"
Sep 09 14:01:25.878542 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:25.878505    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"]
Sep 09 14:01:25.882033 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:25.882006    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"
Sep 09 14:01:25.885175 ip-10-0-129-84 kubenswrapper[2566]: E0909 14:01:25.885142    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj\" is forbidden: User \"system:node:ip-10-0-129-84.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"llm\": no relationship found between node 'ip-10-0-129-84.ec2.internal' and this object" podUID="73f02ede-023b-49f7-b856-ce447792b03f" pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"
Sep 09 14:01:25.885175 ip-10-0-129-84 kubenswrapper[2566]: E0909 14:01:25.885163    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"e2e-same-id-b-00f6feed-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-129-84.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"llm\": no relationship found between node 'ip-10-0-129-84.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"llm\"/\"e2e-same-id-b-00f6feed-kserve-self-signed-certs\"" type="*v1.Secret"
Sep 09 14:01:25.936272 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:25.936239    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"]
Sep 09 14:01:26.056007 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:26.055966    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73f02ede-023b-49f7-b856-ce447792b03f-model-cache\") pod \"e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj\" (UID: \"73f02ede-023b-49f7-b856-ce447792b03f\") " pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"
Sep 09 14:01:26.056227 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:26.056026    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf4ns\" (UniqueName: \"kubernetes.io/projected/73f02ede-023b-49f7-b856-ce447792b03f-kube-api-access-pf4ns\") pod \"e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj\" (UID: \"73f02ede-023b-49f7-b856-ce447792b03f\") " pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"
Sep 09 14:01:26.056227 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:26.056148    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73f02ede-023b-49f7-b856-ce447792b03f-dshm\") pod \"e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj\" (UID: \"73f02ede-023b-49f7-b856-ce447792b03f\") " pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"
Sep 09 14:01:26.056227 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:26.056190    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73f02ede-023b-49f7-b856-ce447792b03f-tmp-dir\") pod \"e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj\" (UID: \"73f02ede-023b-49f7-b856-ce447792b03f\") " pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"
Sep 09 14:01:26.056227 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:26.056218    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73f02ede-023b-49f7-b856-ce447792b03f-tls-certs\") pod \"e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj\" (UID: \"73f02ede-023b-49f7-b856-ce447792b03f\") " pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"
Sep 09 14:01:26.056406 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:26.056245    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73f02ede-023b-49f7-b856-ce447792b03f-home\") pod \"e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj\" (UID: \"73f02ede-023b-49f7-b856-ce447792b03f\") " pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"
Sep 09 14:01:26.157193 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:26.157089    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73f02ede-023b-49f7-b856-ce447792b03f-dshm\") pod \"e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj\" (UID: \"73f02ede-023b-49f7-b856-ce447792b03f\") " pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"
Sep 09 14:01:26.157193 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:26.157133    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73f02ede-023b-49f7-b856-ce447792b03f-tmp-dir\") pod \"e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj\" (UID: \"73f02ede-023b-49f7-b856-ce447792b03f\") " pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"
Sep 09 14:01:26.157193 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:26.157152    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73f02ede-023b-49f7-b856-ce447792b03f-tls-certs\") pod \"e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj\" (UID: \"73f02ede-023b-49f7-b856-ce447792b03f\") " pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"
Sep 09 14:01:26.157193 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:26.157169    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73f02ede-023b-49f7-b856-ce447792b03f-home\") pod \"e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj\" (UID: \"73f02ede-023b-49f7-b856-ce447792b03f\") " pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"
Sep 09 14:01:26.157535 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:26.157205    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73f02ede-023b-49f7-b856-ce447792b03f-model-cache\") pod \"e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj\" (UID: \"73f02ede-023b-49f7-b856-ce447792b03f\") " pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"
Sep 09 14:01:26.157535 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:26.157254    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf4ns\" (UniqueName: \"kubernetes.io/projected/73f02ede-023b-49f7-b856-ce447792b03f-kube-api-access-pf4ns\") pod \"e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj\" (UID: \"73f02ede-023b-49f7-b856-ce447792b03f\") " pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"
Sep 09 14:01:26.157669 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:26.157604    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73f02ede-023b-49f7-b856-ce447792b03f-tmp-dir\") pod \"e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj\" (UID: \"73f02ede-023b-49f7-b856-ce447792b03f\") " pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"
Sep 09 14:01:26.157669 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:26.157632    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73f02ede-023b-49f7-b856-ce447792b03f-home\") pod \"e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj\" (UID: \"73f02ede-023b-49f7-b856-ce447792b03f\") " pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"
Sep 09 14:01:26.157669 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:26.157646    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73f02ede-023b-49f7-b856-ce447792b03f-model-cache\") pod \"e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj\" (UID: \"73f02ede-023b-49f7-b856-ce447792b03f\") " pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"
Sep 09 14:01:26.159391 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:26.159372    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73f02ede-023b-49f7-b856-ce447792b03f-dshm\") pod \"e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj\" (UID: \"73f02ede-023b-49f7-b856-ce447792b03f\") " pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"
Sep 09 14:01:26.172548 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:26.172524    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf4ns\" (UniqueName: \"kubernetes.io/projected/73f02ede-023b-49f7-b856-ce447792b03f-kube-api-access-pf4ns\") pod \"e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj\" (UID: \"73f02ede-023b-49f7-b856-ce447792b03f\") " pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"
Sep 09 14:01:26.895217 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:26.895185    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-00f6feed-kserve-self-signed-certs\""
Sep 09 14:01:26.900262 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:26.900232    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73f02ede-023b-49f7-b856-ce447792b03f-tls-certs\") pod \"e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj\" (UID: \"73f02ede-023b-49f7-b856-ce447792b03f\") " pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"
Sep 09 14:01:27.090543 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:27.090492    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"
Sep 09 14:01:27.384621 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:27.384578    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"]
Sep 09 14:01:27.389205 ip-10-0-129-84 kubenswrapper[2566]: W0909 14:01:27.389171    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73f02ede_023b_49f7_b856_ce447792b03f.slice/crio-176a67212abe1969990fedf256672aadb1879e7610951001d37bf3bf0b9b17ed WatchSource:0}: Error finding container 176a67212abe1969990fedf256672aadb1879e7610951001d37bf3bf0b9b17ed: Status 404 returned error can't find the container with id 176a67212abe1969990fedf256672aadb1879e7610951001d37bf3bf0b9b17ed
Sep 09 14:01:27.909525 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:27.909486    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj" event={"ID":"73f02ede-023b-49f7-b856-ce447792b03f","Type":"ContainerStarted","Data":"54d3cbd4ddb862022c08c5d70de65a2f3c4f06b274d8d4e2dd59f00208152200"}
Sep 09 14:01:27.909525 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:27.909527    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj" event={"ID":"73f02ede-023b-49f7-b856-ce447792b03f","Type":"ContainerStarted","Data":"176a67212abe1969990fedf256672aadb1879e7610951001d37bf3bf0b9b17ed"}
Sep 09 14:01:37.091392 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:37.091360    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"
Sep 09 14:01:37.091939 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:37.091432    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"
Sep 09 14:01:37.103953 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:37.103929    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"
Sep 09 14:01:37.138888 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:37.138843    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj" podStartSLOduration=12.138829766 podStartE2EDuration="12.138829766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:01:27.947710601 +0000 UTC m=+2121.307560427" watchObservedRunningTime="2026-09-09 14:01:37.138829766 +0000 UTC m=+2130.498679586"
Sep 09 14:01:37.965125 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:37.965099    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"
Sep 09 14:01:48.653287 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:48.653252    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"]
Sep 09 14:01:50.289430 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:01:50.289390    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"]
Sep 09 14:02:03.672534 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:03.672420    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n" podUID="51ba5c5b-7708-4c97-af37-62b1500cdc4c" containerName="main" containerID="cri-o://c694793223869da604f05ef183e98a19e151aefb70867e4aa729af47792f62d5" gracePeriod=45
Sep 09 14:02:03.920216 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:03.920191    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"
Sep 09 14:02:03.996360 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:03.996279    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n588p\" (UniqueName: \"kubernetes.io/projected/51ba5c5b-7708-4c97-af37-62b1500cdc4c-kube-api-access-n588p\") pod \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\" (UID: \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\") "
Sep 09 14:02:03.996360 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:03.996334    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51ba5c5b-7708-4c97-af37-62b1500cdc4c-dshm\") pod \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\" (UID: \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\") "
Sep 09 14:02:03.996360 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:03.996358    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51ba5c5b-7708-4c97-af37-62b1500cdc4c-model-cache\") pod \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\" (UID: \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\") "
Sep 09 14:02:03.996962 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:03.996378    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51ba5c5b-7708-4c97-af37-62b1500cdc4c-home\") pod \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\" (UID: \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\") "
Sep 09 14:02:03.996962 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:03.996400    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51ba5c5b-7708-4c97-af37-62b1500cdc4c-tls-certs\") pod \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\" (UID: \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\") "
Sep 09 14:02:03.996962 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:03.996466    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51ba5c5b-7708-4c97-af37-62b1500cdc4c-tmp-dir\") pod \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\" (UID: \"51ba5c5b-7708-4c97-af37-62b1500cdc4c\") "
Sep 09 14:02:03.996962 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:03.996632    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ba5c5b-7708-4c97-af37-62b1500cdc4c-model-cache" (OuterVolumeSpecName: "model-cache") pod "51ba5c5b-7708-4c97-af37-62b1500cdc4c" (UID: "51ba5c5b-7708-4c97-af37-62b1500cdc4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:02:03.996962 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:03.996698    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ba5c5b-7708-4c97-af37-62b1500cdc4c-home" (OuterVolumeSpecName: "home") pod "51ba5c5b-7708-4c97-af37-62b1500cdc4c" (UID: "51ba5c5b-7708-4c97-af37-62b1500cdc4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:02:03.996962 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:03.996857    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ba5c5b-7708-4c97-af37-62b1500cdc4c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "51ba5c5b-7708-4c97-af37-62b1500cdc4c" (UID: "51ba5c5b-7708-4c97-af37-62b1500cdc4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:02:03.996962 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:03.996917    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51ba5c5b-7708-4c97-af37-62b1500cdc4c-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 14:02:03.996962 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:03.996934    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51ba5c5b-7708-4c97-af37-62b1500cdc4c-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 14:02:03.998568 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:03.998544    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ba5c5b-7708-4c97-af37-62b1500cdc4c-dshm" (OuterVolumeSpecName: "dshm") pod "51ba5c5b-7708-4c97-af37-62b1500cdc4c" (UID: "51ba5c5b-7708-4c97-af37-62b1500cdc4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:02:03.998661 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:03.998637    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ba5c5b-7708-4c97-af37-62b1500cdc4c-kube-api-access-n588p" (OuterVolumeSpecName: "kube-api-access-n588p") pod "51ba5c5b-7708-4c97-af37-62b1500cdc4c" (UID: "51ba5c5b-7708-4c97-af37-62b1500cdc4c"). InnerVolumeSpecName "kube-api-access-n588p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:02:03.998661 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:03.998635    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51ba5c5b-7708-4c97-af37-62b1500cdc4c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "51ba5c5b-7708-4c97-af37-62b1500cdc4c" (UID: "51ba5c5b-7708-4c97-af37-62b1500cdc4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:02:04.076562 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:04.076517    2566 generic.go:356] "Generic (PLEG): container finished" podID="51ba5c5b-7708-4c97-af37-62b1500cdc4c" containerID="c694793223869da604f05ef183e98a19e151aefb70867e4aa729af47792f62d5" exitCode=0
Sep 09 14:02:04.076730 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:04.076629    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n" event={"ID":"51ba5c5b-7708-4c97-af37-62b1500cdc4c","Type":"ContainerDied","Data":"c694793223869da604f05ef183e98a19e151aefb70867e4aa729af47792f62d5"}
Sep 09 14:02:04.076730 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:04.076670    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n" event={"ID":"51ba5c5b-7708-4c97-af37-62b1500cdc4c","Type":"ContainerDied","Data":"9cef9643e149c0c403bd9e7300bd55155989d9ece72e09ca96c949f9954c73b3"}
Sep 09 14:02:04.076730 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:04.076693    2566 scope.go:117] "RemoveContainer" containerID="c694793223869da604f05ef183e98a19e151aefb70867e4aa729af47792f62d5"
Sep 09 14:02:04.078211 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:04.078177    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"
Sep 09 14:02:04.087524 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:04.087504    2566 scope.go:117] "RemoveContainer" containerID="c694793223869da604f05ef183e98a19e151aefb70867e4aa729af47792f62d5"
Sep 09 14:02:04.087807 ip-10-0-129-84 kubenswrapper[2566]: E0909 14:02:04.087788    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c694793223869da604f05ef183e98a19e151aefb70867e4aa729af47792f62d5\": container with ID starting with c694793223869da604f05ef183e98a19e151aefb70867e4aa729af47792f62d5 not found: ID does not exist" containerID="c694793223869da604f05ef183e98a19e151aefb70867e4aa729af47792f62d5"
Sep 09 14:02:04.087882 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:04.087838    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c694793223869da604f05ef183e98a19e151aefb70867e4aa729af47792f62d5"} err="failed to get container status \"c694793223869da604f05ef183e98a19e151aefb70867e4aa729af47792f62d5\": rpc error: code = NotFound desc = could not find container \"c694793223869da604f05ef183e98a19e151aefb70867e4aa729af47792f62d5\": container with ID starting with c694793223869da604f05ef183e98a19e151aefb70867e4aa729af47792f62d5 not found: ID does not exist"
Sep 09 14:02:04.097957 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:04.097936    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51ba5c5b-7708-4c97-af37-62b1500cdc4c-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 14:02:04.098048 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:04.097959    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n588p\" (UniqueName: \"kubernetes.io/projected/51ba5c5b-7708-4c97-af37-62b1500cdc4c-kube-api-access-n588p\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 14:02:04.098048 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:04.097972    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51ba5c5b-7708-4c97-af37-62b1500cdc4c-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 14:02:04.098048 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:04.097986    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51ba5c5b-7708-4c97-af37-62b1500cdc4c-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 14:02:04.103154 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:04.103133    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"]
Sep 09 14:02:04.107718 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:04.107699    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-00f6feed-kserve-795cd7cdb7-p588n"]
Sep 09 14:02:05.150741 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:05.150710    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51ba5c5b-7708-4c97-af37-62b1500cdc4c" path="/var/lib/kubelet/pods/51ba5c5b-7708-4c97-af37-62b1500cdc4c/volumes"
Sep 09 14:02:05.308694 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:05.308658    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj" podUID="73f02ede-023b-49f7-b856-ce447792b03f" containerName="main" containerID="cri-o://54d3cbd4ddb862022c08c5d70de65a2f3c4f06b274d8d4e2dd59f00208152200" gracePeriod=45
Sep 09 14:02:05.544823 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:05.544791    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"
Sep 09 14:02:05.710361 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:05.710277    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pf4ns\" (UniqueName: \"kubernetes.io/projected/73f02ede-023b-49f7-b856-ce447792b03f-kube-api-access-pf4ns\") pod \"73f02ede-023b-49f7-b856-ce447792b03f\" (UID: \"73f02ede-023b-49f7-b856-ce447792b03f\") "
Sep 09 14:02:05.710361 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:05.710321    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73f02ede-023b-49f7-b856-ce447792b03f-tmp-dir\") pod \"73f02ede-023b-49f7-b856-ce447792b03f\" (UID: \"73f02ede-023b-49f7-b856-ce447792b03f\") "
Sep 09 14:02:05.710361 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:05.710360    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73f02ede-023b-49f7-b856-ce447792b03f-model-cache\") pod \"73f02ede-023b-49f7-b856-ce447792b03f\" (UID: \"73f02ede-023b-49f7-b856-ce447792b03f\") "
Sep 09 14:02:05.710649 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:05.710388    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73f02ede-023b-49f7-b856-ce447792b03f-home\") pod \"73f02ede-023b-49f7-b856-ce447792b03f\" (UID: \"73f02ede-023b-49f7-b856-ce447792b03f\") "
Sep 09 14:02:05.710649 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:05.710422    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73f02ede-023b-49f7-b856-ce447792b03f-tls-certs\") pod \"73f02ede-023b-49f7-b856-ce447792b03f\" (UID: \"73f02ede-023b-49f7-b856-ce447792b03f\") "
Sep 09 14:02:05.710649 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:05.710456    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73f02ede-023b-49f7-b856-ce447792b03f-dshm\") pod \"73f02ede-023b-49f7-b856-ce447792b03f\" (UID: \"73f02ede-023b-49f7-b856-ce447792b03f\") "
Sep 09 14:02:05.710839 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:05.710668    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73f02ede-023b-49f7-b856-ce447792b03f-model-cache" (OuterVolumeSpecName: "model-cache") pod "73f02ede-023b-49f7-b856-ce447792b03f" (UID: "73f02ede-023b-49f7-b856-ce447792b03f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:02:05.710839 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:05.710681    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73f02ede-023b-49f7-b856-ce447792b03f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "73f02ede-023b-49f7-b856-ce447792b03f" (UID: "73f02ede-023b-49f7-b856-ce447792b03f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:02:05.710839 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:05.710693    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73f02ede-023b-49f7-b856-ce447792b03f-home" (OuterVolumeSpecName: "home") pod "73f02ede-023b-49f7-b856-ce447792b03f" (UID: "73f02ede-023b-49f7-b856-ce447792b03f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:02:05.712546 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:05.712523    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73f02ede-023b-49f7-b856-ce447792b03f-dshm" (OuterVolumeSpecName: "dshm") pod "73f02ede-023b-49f7-b856-ce447792b03f" (UID: "73f02ede-023b-49f7-b856-ce447792b03f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 14:02:05.712546 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:05.712527    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f02ede-023b-49f7-b856-ce447792b03f-kube-api-access-pf4ns" (OuterVolumeSpecName: "kube-api-access-pf4ns") pod "73f02ede-023b-49f7-b856-ce447792b03f" (UID: "73f02ede-023b-49f7-b856-ce447792b03f"). InnerVolumeSpecName "kube-api-access-pf4ns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 14:02:05.712676 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:05.712596    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73f02ede-023b-49f7-b856-ce447792b03f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "73f02ede-023b-49f7-b856-ce447792b03f" (UID: "73f02ede-023b-49f7-b856-ce447792b03f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 14:02:05.811514 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:05.811486    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73f02ede-023b-49f7-b856-ce447792b03f-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 14:02:05.811514 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:05.811512    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73f02ede-023b-49f7-b856-ce447792b03f-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 14:02:05.811514 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:05.811521    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73f02ede-023b-49f7-b856-ce447792b03f-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 14:02:05.811717 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:05.811529    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf4ns\" (UniqueName: \"kubernetes.io/projected/73f02ede-023b-49f7-b856-ce447792b03f-kube-api-access-pf4ns\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 14:02:05.811717 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:05.811538    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73f02ede-023b-49f7-b856-ce447792b03f-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 14:02:05.811717 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:05.811546    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73f02ede-023b-49f7-b856-ce447792b03f-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Sep 09 14:02:06.089261 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:06.089222    2566 generic.go:356] "Generic (PLEG): container finished" podID="73f02ede-023b-49f7-b856-ce447792b03f" containerID="54d3cbd4ddb862022c08c5d70de65a2f3c4f06b274d8d4e2dd59f00208152200" exitCode=0
Sep 09 14:02:06.089451 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:06.089269    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"
Sep 09 14:02:06.089451 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:06.089306    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj" event={"ID":"73f02ede-023b-49f7-b856-ce447792b03f","Type":"ContainerDied","Data":"54d3cbd4ddb862022c08c5d70de65a2f3c4f06b274d8d4e2dd59f00208152200"}
Sep 09 14:02:06.089451 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:06.089358    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj" event={"ID":"73f02ede-023b-49f7-b856-ce447792b03f","Type":"ContainerDied","Data":"176a67212abe1969990fedf256672aadb1879e7610951001d37bf3bf0b9b17ed"}
Sep 09 14:02:06.089451 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:06.089383    2566 scope.go:117] "RemoveContainer" containerID="54d3cbd4ddb862022c08c5d70de65a2f3c4f06b274d8d4e2dd59f00208152200"
Sep 09 14:02:06.098086 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:06.098067    2566 scope.go:117] "RemoveContainer" containerID="54d3cbd4ddb862022c08c5d70de65a2f3c4f06b274d8d4e2dd59f00208152200"
Sep 09 14:02:06.098331 ip-10-0-129-84 kubenswrapper[2566]: E0909 14:02:06.098312    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54d3cbd4ddb862022c08c5d70de65a2f3c4f06b274d8d4e2dd59f00208152200\": container with ID starting with 54d3cbd4ddb862022c08c5d70de65a2f3c4f06b274d8d4e2dd59f00208152200 not found: ID does not exist" containerID="54d3cbd4ddb862022c08c5d70de65a2f3c4f06b274d8d4e2dd59f00208152200"
Sep 09 14:02:06.098385 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:06.098341    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d3cbd4ddb862022c08c5d70de65a2f3c4f06b274d8d4e2dd59f00208152200"} err="failed to get container status \"54d3cbd4ddb862022c08c5d70de65a2f3c4f06b274d8d4e2dd59f00208152200\": rpc error: code = NotFound desc = could not find container \"54d3cbd4ddb862022c08c5d70de65a2f3c4f06b274d8d4e2dd59f00208152200\": container with ID starting with 54d3cbd4ddb862022c08c5d70de65a2f3c4f06b274d8d4e2dd59f00208152200 not found: ID does not exist"
Sep 09 14:02:06.112026 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:06.112000    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"]
Sep 09 14:02:06.120318 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:06.120298    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-00f6feed-kserve-6d8c9b7999-lpjxj"]
Sep 09 14:02:07.156072 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:07.156038    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73f02ede-023b-49f7-b856-ce447792b03f" path="/var/lib/kubelet/pods/73f02ede-023b-49f7-b856-ce447792b03f/volumes"
Sep 09 14:02:07.710963 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:07.710843    2566 scope.go:117] "RemoveContainer" containerID="16cb1b58403c52a526ca3549a487250d2ed31ebb57b18fc49f53de25b67488d0"
Sep 09 14:02:07.733013 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:07.719075    2566 scope.go:117] "RemoveContainer" containerID="83b5025dcc49c7f65850ab29c674eb306f49d5867f683a85565ca87db96f8693"
Sep 09 14:02:42.655286 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:42.655257    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8469c9486d-jqtgr_bff62eb2-b9c8-4142-8d12-4b968be9de07/copy-manifests/0.log"
Sep 09 14:02:42.680483 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:42.680443    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8469c9486d-jqtgr_bff62eb2-b9c8-4142-8d12-4b968be9de07/manager/0.log"
Sep 09 14:02:43.365443 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:43.365411    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29816025-844hc_c2bb2921-5f53-426b-b458-d4f92008e571/cleanup/0.log"
Sep 09 14:02:45.409572 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:45.409532    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn_766fc6d1-54c1-4a76-844a-f828adb2442d/istio-proxy/0.log"
Sep 09 14:02:45.743478 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:45.743356    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-595c6d67d9-zbpjd_10f201f3-a37d-454a-a0ba-e0230efcbcfd/kube-auth-proxy/0.log"
Sep 09 14:02:46.194997 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:46.194961    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5fb8d9b79d-nxqx6_6210e253-1f3a-480f-84de-2cc61e20c535/router/0.log"
Sep 09 14:02:46.660579 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:46.660544    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-5t22z_366979d9-163d-42be-aa7a-258f9ad1b098/main/0.log"
Sep 09 14:02:46.888005 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:46.887959    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-h784p_ef9d42fc-968c-411e-9981-b24e72db2c42/main/0.log"
Sep 09 14:02:47.222941 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:47.222911    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-r4s24_e49c0e6c-c8e4-4356-be97-b1d89e8384d5/main/0.log"
Sep 09 14:02:52.522145 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:52.522110    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nqbm9/must-gather-j8v2g"]
Sep 09 14:02:52.522633 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:52.522465    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73f02ede-023b-49f7-b856-ce447792b03f" containerName="main"
Sep 09 14:02:52.522633 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:52.522476    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="73f02ede-023b-49f7-b856-ce447792b03f" containerName="main"
Sep 09 14:02:52.522633 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:52.522494    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51ba5c5b-7708-4c97-af37-62b1500cdc4c" containerName="main"
Sep 09 14:02:52.522633 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:52.522500    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="51ba5c5b-7708-4c97-af37-62b1500cdc4c" containerName="main"
Sep 09 14:02:52.522633 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:52.522560    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="51ba5c5b-7708-4c97-af37-62b1500cdc4c" containerName="main"
Sep 09 14:02:52.522633 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:52.522599    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="73f02ede-023b-49f7-b856-ce447792b03f" containerName="main"
Sep 09 14:02:52.525571 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:52.525555    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nqbm9/must-gather-j8v2g"
Sep 09 14:02:52.528809 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:52.528786    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nqbm9\"/\"default-dockercfg-gttcx\""
Sep 09 14:02:52.528918 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:52.528899    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nqbm9\"/\"kube-root-ca.crt\""
Sep 09 14:02:52.529503 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:52.529483    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nqbm9\"/\"openshift-service-ca.crt\""
Sep 09 14:02:52.547115 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:52.547084    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nqbm9/must-gather-j8v2g"]
Sep 09 14:02:52.617342 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:52.617300    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br8tm\" (UniqueName: \"kubernetes.io/projected/546348da-31f9-4d67-8666-bc0bad2d756d-kube-api-access-br8tm\") pod \"must-gather-j8v2g\" (UID: \"546348da-31f9-4d67-8666-bc0bad2d756d\") " pod="openshift-must-gather-nqbm9/must-gather-j8v2g"
Sep 09 14:02:52.617342 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:52.617343    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/546348da-31f9-4d67-8666-bc0bad2d756d-must-gather-output\") pod \"must-gather-j8v2g\" (UID: \"546348da-31f9-4d67-8666-bc0bad2d756d\") " pod="openshift-must-gather-nqbm9/must-gather-j8v2g"
Sep 09 14:02:52.718522 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:52.718482    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br8tm\" (UniqueName: \"kubernetes.io/projected/546348da-31f9-4d67-8666-bc0bad2d756d-kube-api-access-br8tm\") pod \"must-gather-j8v2g\" (UID: \"546348da-31f9-4d67-8666-bc0bad2d756d\") " pod="openshift-must-gather-nqbm9/must-gather-j8v2g"
Sep 09 14:02:52.718522 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:52.718523    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/546348da-31f9-4d67-8666-bc0bad2d756d-must-gather-output\") pod \"must-gather-j8v2g\" (UID: \"546348da-31f9-4d67-8666-bc0bad2d756d\") " pod="openshift-must-gather-nqbm9/must-gather-j8v2g"
Sep 09 14:02:52.718914 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:52.718896    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/546348da-31f9-4d67-8666-bc0bad2d756d-must-gather-output\") pod \"must-gather-j8v2g\" (UID: \"546348da-31f9-4d67-8666-bc0bad2d756d\") " pod="openshift-must-gather-nqbm9/must-gather-j8v2g"
Sep 09 14:02:52.736703 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:52.736679    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br8tm\" (UniqueName: \"kubernetes.io/projected/546348da-31f9-4d67-8666-bc0bad2d756d-kube-api-access-br8tm\") pod \"must-gather-j8v2g\" (UID: \"546348da-31f9-4d67-8666-bc0bad2d756d\") " pod="openshift-must-gather-nqbm9/must-gather-j8v2g"
Sep 09 14:02:52.834270 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:52.834240    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nqbm9/must-gather-j8v2g"
Sep 09 14:02:53.111508 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:53.111314    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nqbm9/must-gather-j8v2g"]
Sep 09 14:02:53.114497 ip-10-0-129-84 kubenswrapper[2566]: W0909 14:02:53.114472    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod546348da_31f9_4d67_8666_bc0bad2d756d.slice/crio-2ac6cd14519bb71a4d481c652ce14e768fc38f6adabe2f0b441e9c09f9d01263 WatchSource:0}: Error finding container 2ac6cd14519bb71a4d481c652ce14e768fc38f6adabe2f0b441e9c09f9d01263: Status 404 returned error can't find the container with id 2ac6cd14519bb71a4d481c652ce14e768fc38f6adabe2f0b441e9c09f9d01263
Sep 09 14:02:53.116778 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:53.116759    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 14:02:53.298116 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:53.298086    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nqbm9/must-gather-j8v2g" event={"ID":"546348da-31f9-4d67-8666-bc0bad2d756d","Type":"ContainerStarted","Data":"2ac6cd14519bb71a4d481c652ce14e768fc38f6adabe2f0b441e9c09f9d01263"}
Sep 09 14:02:54.304952 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:54.304918    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nqbm9/must-gather-j8v2g" event={"ID":"546348da-31f9-4d67-8666-bc0bad2d756d","Type":"ContainerStarted","Data":"418ffc6a46efee8d7a11104fce4eabfdda172d9ab9397c4d5b4195eac0f25347"}
Sep 09 14:02:54.304952 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:54.304959    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nqbm9/must-gather-j8v2g" event={"ID":"546348da-31f9-4d67-8666-bc0bad2d756d","Type":"ContainerStarted","Data":"4f505496a110f66bbc91080d74c7d4610a85cee5a0aca7e052da66cc7172a39b"}
Sep 09 14:02:55.478101 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:55.478068    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-plsz4_0ecc5e12-af64-4305-83ed-37552f51b76c/global-pull-secret-syncer/0.log"
Sep 09 14:02:55.620606 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:55.620565    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-q5njn_ab687c9c-b837-4387-a696-7f24afd7711f/konnectivity-agent/0.log"
Sep 09 14:02:55.675510 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:02:55.675475    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-84.ec2.internal_bad615a00a972079824050bf66b78c89/haproxy/0.log"
Sep 09 14:03:02.300550 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:02.300517    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-wdm7n_bbe27bd0-1d07-43fe-8bdc-11439590bbab/cluster-monitoring-operator/0.log"
Sep 09 14:03:02.475371 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:02.475321    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bkbmm_b0b37b18-368c-4506-83a3-f55bece72293/node-exporter/0.log"
Sep 09 14:03:02.499476 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:02.499411    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bkbmm_b0b37b18-368c-4506-83a3-f55bece72293/kube-rbac-proxy/0.log"
Sep 09 14:03:02.522462 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:02.522373    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bkbmm_b0b37b18-368c-4506-83a3-f55bece72293/init-textfile/0.log"
Sep 09 14:03:04.871096 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:04.871060    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-shzt9_6a2933d5-93f7-4cc3-ae7c-30f4c9444b3c/networking-console-plugin/0.log"
Sep 09 14:03:05.494173 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:05.494140    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ftm96_af902e8b-d693-4341-9979-03cf69548cfb/console-operator/2.log"
Sep 09 14:03:05.501459 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:05.501411    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ftm96_af902e8b-d693-4341-9979-03cf69548cfb/console-operator/3.log"
Sep 09 14:03:06.144404 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:06.144334    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nqbm9/must-gather-j8v2g" podStartSLOduration=13.489212362 podStartE2EDuration="14.144310642s" totalImagesPullingTime="655.09828ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:02:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:02:54.323524602 +0000 UTC m=+2207.683374428" watchObservedRunningTime="2026-09-09 14:03:06.144310642 +0000 UTC m=+2219.504160469"
Sep 09 14:03:06.145506 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:06.145470    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nqbm9/perf-node-gather-daemonset-jxfl9"]
Sep 09 14:03:06.150744 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:06.150712    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nqbm9/perf-node-gather-daemonset-jxfl9"
Sep 09 14:03:06.157939 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:06.157642    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nqbm9/perf-node-gather-daemonset-jxfl9"]
Sep 09 14:03:06.256309 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:06.255974    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/80779997-c77e-4b69-85b6-a12373cf2e2d-lib-modules\") pod \"perf-node-gather-daemonset-jxfl9\" (UID: \"80779997-c77e-4b69-85b6-a12373cf2e2d\") " pod="openshift-must-gather-nqbm9/perf-node-gather-daemonset-jxfl9"
Sep 09 14:03:06.256309 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:06.256040    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/80779997-c77e-4b69-85b6-a12373cf2e2d-podres\") pod \"perf-node-gather-daemonset-jxfl9\" (UID: \"80779997-c77e-4b69-85b6-a12373cf2e2d\") " pod="openshift-must-gather-nqbm9/perf-node-gather-daemonset-jxfl9"
Sep 09 14:03:06.256309 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:06.256084    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80779997-c77e-4b69-85b6-a12373cf2e2d-sys\") pod \"perf-node-gather-daemonset-jxfl9\" (UID: \"80779997-c77e-4b69-85b6-a12373cf2e2d\") " pod="openshift-must-gather-nqbm9/perf-node-gather-daemonset-jxfl9"
Sep 09 14:03:06.256309 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:06.256121    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/80779997-c77e-4b69-85b6-a12373cf2e2d-proc\") pod \"perf-node-gather-daemonset-jxfl9\" (UID: \"80779997-c77e-4b69-85b6-a12373cf2e2d\") " pod="openshift-must-gather-nqbm9/perf-node-gather-daemonset-jxfl9"
Sep 09 14:03:06.256309 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:06.256185    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl6t5\" (UniqueName: \"kubernetes.io/projected/80779997-c77e-4b69-85b6-a12373cf2e2d-kube-api-access-vl6t5\") pod \"perf-node-gather-daemonset-jxfl9\" (UID: \"80779997-c77e-4b69-85b6-a12373cf2e2d\") " pod="openshift-must-gather-nqbm9/perf-node-gather-daemonset-jxfl9"
Sep 09 14:03:06.357485 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:06.357451    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl6t5\" (UniqueName: \"kubernetes.io/projected/80779997-c77e-4b69-85b6-a12373cf2e2d-kube-api-access-vl6t5\") pod \"perf-node-gather-daemonset-jxfl9\" (UID: \"80779997-c77e-4b69-85b6-a12373cf2e2d\") " pod="openshift-must-gather-nqbm9/perf-node-gather-daemonset-jxfl9"
Sep 09 14:03:06.357660 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:06.357523    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/80779997-c77e-4b69-85b6-a12373cf2e2d-lib-modules\") pod \"perf-node-gather-daemonset-jxfl9\" (UID: \"80779997-c77e-4b69-85b6-a12373cf2e2d\") " pod="openshift-must-gather-nqbm9/perf-node-gather-daemonset-jxfl9"
Sep 09 14:03:06.357660 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:06.357564    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/80779997-c77e-4b69-85b6-a12373cf2e2d-podres\") pod \"perf-node-gather-daemonset-jxfl9\" (UID: \"80779997-c77e-4b69-85b6-a12373cf2e2d\") " pod="openshift-must-gather-nqbm9/perf-node-gather-daemonset-jxfl9"
Sep 09 14:03:06.357747 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:06.357706    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/80779997-c77e-4b69-85b6-a12373cf2e2d-podres\") pod \"perf-node-gather-daemonset-jxfl9\" (UID: \"80779997-c77e-4b69-85b6-a12373cf2e2d\") " pod="openshift-must-gather-nqbm9/perf-node-gather-daemonset-jxfl9"
Sep 09 14:03:06.357747 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:06.357708    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80779997-c77e-4b69-85b6-a12373cf2e2d-sys\") pod \"perf-node-gather-daemonset-jxfl9\" (UID: \"80779997-c77e-4b69-85b6-a12373cf2e2d\") " pod="openshift-must-gather-nqbm9/perf-node-gather-daemonset-jxfl9"
Sep 09 14:03:06.357747 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:06.357726    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/80779997-c77e-4b69-85b6-a12373cf2e2d-lib-modules\") pod \"perf-node-gather-daemonset-jxfl9\" (UID: \"80779997-c77e-4b69-85b6-a12373cf2e2d\") " pod="openshift-must-gather-nqbm9/perf-node-gather-daemonset-jxfl9"
Sep 09 14:03:06.357889 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:06.357764    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/80779997-c77e-4b69-85b6-a12373cf2e2d-proc\") pod \"perf-node-gather-daemonset-jxfl9\" (UID: \"80779997-c77e-4b69-85b6-a12373cf2e2d\") " pod="openshift-must-gather-nqbm9/perf-node-gather-daemonset-jxfl9"
Sep 09 14:03:06.357889 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:06.357803    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80779997-c77e-4b69-85b6-a12373cf2e2d-sys\") pod \"perf-node-gather-daemonset-jxfl9\" (UID: \"80779997-c77e-4b69-85b6-a12373cf2e2d\") " pod="openshift-must-gather-nqbm9/perf-node-gather-daemonset-jxfl9"
Sep 09 14:03:06.357889 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:06.357874    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/80779997-c77e-4b69-85b6-a12373cf2e2d-proc\") pod \"perf-node-gather-daemonset-jxfl9\" (UID: \"80779997-c77e-4b69-85b6-a12373cf2e2d\") " pod="openshift-must-gather-nqbm9/perf-node-gather-daemonset-jxfl9"
Sep 09 14:03:06.372494 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:06.372466    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl6t5\" (UniqueName: \"kubernetes.io/projected/80779997-c77e-4b69-85b6-a12373cf2e2d-kube-api-access-vl6t5\") pod \"perf-node-gather-daemonset-jxfl9\" (UID: \"80779997-c77e-4b69-85b6-a12373cf2e2d\") " pod="openshift-must-gather-nqbm9/perf-node-gather-daemonset-jxfl9"
Sep 09 14:03:06.465316 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:06.465221    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nqbm9/perf-node-gather-daemonset-jxfl9"
Sep 09 14:03:06.773393 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:06.773295    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nqbm9/perf-node-gather-daemonset-jxfl9"]
Sep 09 14:03:06.779173 ip-10-0-129-84 kubenswrapper[2566]: W0909 14:03:06.779146    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod80779997_c77e_4b69_85b6_a12373cf2e2d.slice/crio-4296b5bed7a9356af858ab68ab29c7bf658a2cb88773e420473341cdac300f07 WatchSource:0}: Error finding container 4296b5bed7a9356af858ab68ab29c7bf658a2cb88773e420473341cdac300f07: Status 404 returned error can't find the container with id 4296b5bed7a9356af858ab68ab29c7bf658a2cb88773e420473341cdac300f07
Sep 09 14:03:06.932230 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:06.932195    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-d6qbv_97bffee4-6cb2-4047-8a04-e92e83309608/volume-data-source-validator/0.log"
Sep 09 14:03:07.390698 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:07.390633    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nqbm9/perf-node-gather-daemonset-jxfl9" event={"ID":"80779997-c77e-4b69-85b6-a12373cf2e2d","Type":"ContainerStarted","Data":"22e600452bd230e872542cd8ddb7674b463c8326076503a5f44e3dad3e02fb14"}
Sep 09 14:03:07.391351 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:07.390711    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nqbm9/perf-node-gather-daemonset-jxfl9" event={"ID":"80779997-c77e-4b69-85b6-a12373cf2e2d","Type":"ContainerStarted","Data":"4296b5bed7a9356af858ab68ab29c7bf658a2cb88773e420473341cdac300f07"}
Sep 09 14:03:07.391621 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:07.391597    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nqbm9/perf-node-gather-daemonset-jxfl9"
Sep 09 14:03:07.408909 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:07.408839    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nqbm9/perf-node-gather-daemonset-jxfl9" podStartSLOduration=1.40880285 podStartE2EDuration="1.40880285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:03:07.407077915 +0000 UTC m=+2220.766927751" watchObservedRunningTime="2026-09-09 14:03:07.40880285 +0000 UTC m=+2220.768652680"
Sep 09 14:03:08.017920 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:08.017876    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4dnj2_be3c3076-7d51-4348-ba53-1d350b9f69b6/dns/0.log"
Sep 09 14:03:08.039802 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:08.039774    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4dnj2_be3c3076-7d51-4348-ba53-1d350b9f69b6/kube-rbac-proxy/0.log"
Sep 09 14:03:08.208903 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:08.208877    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sw9qv_f06384ff-80d7-4e3a-a9bd-4925d55d6824/dns-node-resolver/0.log"
Sep 09 14:03:08.859628 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:08.859602    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qrq28_e61e15eb-3d0d-4a92-a91a-0987c9fd4eeb/node-ca/0.log"
Sep 09 14:03:09.830953 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:09.830925    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn_766fc6d1-54c1-4a76-844a-f828adb2442d/istio-proxy/0.log"
Sep 09 14:03:10.322850 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:10.322754    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-595c6d67d9-zbpjd_10f201f3-a37d-454a-a0ba-e0230efcbcfd/kube-auth-proxy/0.log"
Sep 09 14:03:10.524769 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:10.524741    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5fb8d9b79d-nxqx6_6210e253-1f3a-480f-84de-2cc61e20c535/router/0.log"
Sep 09 14:03:11.521947 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:11.521919    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-thhvt_fee30cf9-8a3a-466d-8460-8ec8b363ff6d/serve-healthcheck-canary/0.log"
Sep 09 14:03:12.035955 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:12.035917    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-z7s72_e2fed7f4-5c70-4dc5-a61d-880e4c65492f/insights-operator/0.log"
Sep 09 14:03:12.038359 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:12.038329    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-z7s72_e2fed7f4-5c70-4dc5-a61d-880e4c65492f/insights-operator/1.log"
Sep 09 14:03:12.143669 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:12.143642    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sckf8_403adf37-3fd7-4d5b-bda9-0b28e7623e08/kube-rbac-proxy/0.log"
Sep 09 14:03:12.168670 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:12.168642    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sckf8_403adf37-3fd7-4d5b-bda9-0b28e7623e08/exporter/0.log"
Sep 09 14:03:12.193529 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:12.193493    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sckf8_403adf37-3fd7-4d5b-bda9-0b28e7623e08/extractor/0.log"
Sep 09 14:03:14.413874 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:14.413846    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nqbm9/perf-node-gather-daemonset-jxfl9"
Sep 09 14:03:15.015612 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:15.015573    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8469c9486d-jqtgr_bff62eb2-b9c8-4142-8d12-4b968be9de07/manager/0.log"
Sep 09 14:03:15.033634 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:15.033601    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8469c9486d-jqtgr_bff62eb2-b9c8-4142-8d12-4b968be9de07/copy-manifests/0.log"
Sep 09 14:03:16.058954 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:16.058921    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn_766fc6d1-54c1-4a76-844a-f828adb2442d/istio-proxy/0.log"
Sep 09 14:03:16.451561 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:16.451532    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-68df89f6f5-kknpr_5d616539-8e2a-4076-9abf-ea8dd8f8cbc3/manager/0.log"
Sep 09 14:03:16.507191 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:16.507157    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-595c6d67d9-zbpjd_10f201f3-a37d-454a-a0ba-e0230efcbcfd/kube-auth-proxy/0.log"
Sep 09 14:03:16.612826 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:16.612750    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5fb8d9b79d-nxqx6_6210e253-1f3a-480f-84de-2cc61e20c535/router/0.log"
Sep 09 14:03:18.125940 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:18.125906    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-vtvmn_766fc6d1-54c1-4a76-844a-f828adb2442d/istio-proxy/0.log"
Sep 09 14:03:18.587117 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:18.587084    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-595c6d67d9-zbpjd_10f201f3-a37d-454a-a0ba-e0230efcbcfd/kube-auth-proxy/0.log"
Sep 09 14:03:18.688012 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:18.687979    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5fb8d9b79d-nxqx6_6210e253-1f3a-480f-84de-2cc61e20c535/router/0.log"
Sep 09 14:03:22.200592 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:22.200554    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-5hgw6_613994fe-2f81-47d4-81f3-1b6a63682f5b/kube-storage-version-migrator-operator/1.log"
Sep 09 14:03:22.201465 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:22.201444    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-5hgw6_613994fe-2f81-47d4-81f3-1b6a63682f5b/kube-storage-version-migrator-operator/0.log"
Sep 09 14:03:23.418161 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:23.418131    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q6cft_6dde766b-3e64-4762-ab24-25b30b1b4465/kube-multus-additional-cni-plugins/0.log"
Sep 09 14:03:23.441943 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:23.441908    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q6cft_6dde766b-3e64-4762-ab24-25b30b1b4465/egress-router-binary-copy/0.log"
Sep 09 14:03:23.463499 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:23.463467    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q6cft_6dde766b-3e64-4762-ab24-25b30b1b4465/cni-plugins/0.log"
Sep 09 14:03:23.488687 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:23.488657    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q6cft_6dde766b-3e64-4762-ab24-25b30b1b4465/bond-cni-plugin/0.log"
Sep 09 14:03:23.526099 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:23.526065    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q6cft_6dde766b-3e64-4762-ab24-25b30b1b4465/routeoverride-cni/0.log"
Sep 09 14:03:23.552879 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:23.552840    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q6cft_6dde766b-3e64-4762-ab24-25b30b1b4465/whereabouts-cni-bincopy/0.log"
Sep 09 14:03:23.577282 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:23.577254    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q6cft_6dde766b-3e64-4762-ab24-25b30b1b4465/whereabouts-cni/0.log"
Sep 09 14:03:23.818999 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:23.818925    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-gwqft_a455ac3b-27e3-42ce-b4c3-c0fe4b5f460d/kube-multus/0.log"
Sep 09 14:03:23.995993 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:23.995965    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-q5dxg_9ddda0bf-9853-4015-a006-c9e44d47acd5/network-metrics-daemon/0.log"
Sep 09 14:03:24.020482 ip-10-0-129-84 kubenswrapper[2566]: I0909 14:03:24.020447    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-q5dxg_9ddda0bf-9853-4015-a006-c9e44d47acd5/kube-rbac-proxy/0.log"