Sep 16 14:14:11.291731 ip-10-0-132-217 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 14:14:11.638552 ip-10-0-132-217 kubenswrapper[2587]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 14:14:11.638552 ip-10-0-132-217 kubenswrapper[2587]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 14:14:11.638552 ip-10-0-132-217 kubenswrapper[2587]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 14:14:11.638552 ip-10-0-132-217 kubenswrapper[2587]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 14:14:11.638552 ip-10-0-132-217 kubenswrapper[2587]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 14:14:11.639346 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.639255    2587 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 14:14:11.643086 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643066    2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 14:14:11.643086 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643085    2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 14:14:11.643086 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643089    2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 14:14:11.643191 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643093    2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 14:14:11.643191 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643096    2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 14:14:11.643191 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643100    2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 14:14:11.643191 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643103    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 14:14:11.643191 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643106    2587 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 14:14:11.643191 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643109    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 14:14:11.643191 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643112    2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 14:14:11.643191 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643115    2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 14:14:11.643191 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643117    2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 14:14:11.643191 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643120    2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 14:14:11.643191 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643123    2587 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 14:14:11.643191 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643125    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 14:14:11.643191 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643128    2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 14:14:11.643191 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643131    2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 14:14:11.643191 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643134    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 14:14:11.643191 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643136    2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 14:14:11.643191 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643145    2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 14:14:11.643191 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643148    2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 14:14:11.643191 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643151    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 14:14:11.643191 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643154    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 14:14:11.643697 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643158    2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 14:14:11.643697 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643162    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 14:14:11.643697 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643165    2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 14:14:11.643697 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643168    2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 14:14:11.643697 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643171    2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 14:14:11.643697 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643174    2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 14:14:11.643697 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643177    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 14:14:11.643697 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643179    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 14:14:11.643697 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643182    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 14:14:11.643697 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643185    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 14:14:11.643697 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643188    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 14:14:11.643697 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643191    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 14:14:11.643697 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643195    2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 14:14:11.643697 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643198    2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 14:14:11.643697 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643202    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 14:14:11.643697 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643205    2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 14:14:11.643697 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643208    2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 14:14:11.643697 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643210    2587 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 14:14:11.643697 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643213    2587 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 14:14:11.644159 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643215    2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 14:14:11.644159 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643218    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 14:14:11.644159 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643221    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 14:14:11.644159 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643223    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 14:14:11.644159 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643227    2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 14:14:11.644159 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643229    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 14:14:11.644159 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643232    2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 14:14:11.644159 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643234    2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 14:14:11.644159 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643239    2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 14:14:11.644159 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643242    2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 14:14:11.644159 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643248    2587 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 14:14:11.644159 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643250    2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 14:14:11.644159 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643253    2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 14:14:11.644159 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643255    2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 14:14:11.644159 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643258    2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 14:14:11.644159 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643261    2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 14:14:11.644159 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643264    2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 14:14:11.644159 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643266    2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 14:14:11.644159 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643269    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 14:14:11.644636 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643271    2587 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 14:14:11.644636 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643274    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 14:14:11.644636 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643276    2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 14:14:11.644636 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643278    2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 14:14:11.644636 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643281    2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 14:14:11.644636 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643283    2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 14:14:11.644636 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643287    2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 14:14:11.644636 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643290    2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 14:14:11.644636 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643292    2587 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 14:14:11.644636 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643295    2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 14:14:11.644636 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643298    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 14:14:11.644636 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643301    2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 14:14:11.644636 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643303    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 14:14:11.644636 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643306    2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 14:14:11.644636 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643308    2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 14:14:11.644636 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643311    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 14:14:11.644636 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643314    2587 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 14:14:11.644636 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643316    2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 14:14:11.644636 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643319    2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 14:14:11.644636 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643321    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 14:14:11.645112 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643324    2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 14:14:11.645112 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643327    2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 14:14:11.645112 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643331    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 14:14:11.645112 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643333    2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 14:14:11.645112 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643336    2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 14:14:11.645112 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643339    2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 14:14:11.645112 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643342    2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 14:14:11.645112 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643344    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 14:14:11.645112 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643347    2587 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 14:14:11.645112 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643350    2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 14:14:11.645112 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643352    2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 14:14:11.645112 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643355    2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 14:14:11.645112 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643357    2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 14:14:11.645112 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643360    2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 14:14:11.645112 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643362    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 14:14:11.645112 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643365    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 14:14:11.645112 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643367    2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 14:14:11.645112 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.643372    2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 14:14:11.645112 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644150    2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 14:14:11.645112 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644157    2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 14:14:11.645613 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644161    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 14:14:11.645613 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644164    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 14:14:11.645613 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644168    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 14:14:11.645613 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644171    2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 14:14:11.645613 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644174    2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 14:14:11.645613 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644177    2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 14:14:11.645613 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644181    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 14:14:11.645613 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644183    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 14:14:11.645613 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644186    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 14:14:11.645613 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644189    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 14:14:11.645613 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644191    2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 14:14:11.645613 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644194    2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 14:14:11.645613 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644197    2587 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 14:14:11.645613 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644199    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 14:14:11.645613 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644202    2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 14:14:11.645613 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644204    2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 14:14:11.645613 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644207    2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 14:14:11.645613 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644210    2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 14:14:11.645613 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644213    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 14:14:11.645613 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644216    2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 14:14:11.646125 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644218    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 14:14:11.646125 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644221    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 14:14:11.646125 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644223    2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 14:14:11.646125 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644226    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 14:14:11.646125 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644229    2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 14:14:11.646125 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644231    2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 14:14:11.646125 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644234    2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 14:14:11.646125 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644236    2587 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 14:14:11.646125 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644239    2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 14:14:11.646125 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644241    2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 14:14:11.646125 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644245    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 14:14:11.646125 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644248    2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 14:14:11.646125 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644250    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 14:14:11.646125 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644253    2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 14:14:11.646125 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644255    2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 14:14:11.646125 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644258    2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 14:14:11.646125 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644261    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 14:14:11.646125 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644263    2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 14:14:11.646125 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644266    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 14:14:11.646125 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644269    2587 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 14:14:11.646623 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644271    2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 14:14:11.646623 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644274    2587 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 14:14:11.646623 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644276    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 14:14:11.646623 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644279    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 14:14:11.646623 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644281    2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 14:14:11.646623 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644284    2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 14:14:11.646623 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644287    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 14:14:11.646623 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644291    2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 14:14:11.646623 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644295    2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 14:14:11.646623 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644297    2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 14:14:11.646623 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644300    2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 14:14:11.646623 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644305    2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 14:14:11.646623 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644308    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 14:14:11.646623 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644310    2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 14:14:11.646623 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644313    2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 14:14:11.646623 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644315    2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 14:14:11.646623 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644318    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 14:14:11.646623 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644321    2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 14:14:11.646623 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644323    2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 14:14:11.646623 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644326    2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 14:14:11.647105 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644329    2587 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 14:14:11.647105 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644331    2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 14:14:11.647105 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644334    2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 14:14:11.647105 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644337    2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 14:14:11.647105 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644340    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 14:14:11.647105 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644342    2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 14:14:11.647105 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644344    2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 14:14:11.647105 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644347    2587 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 14:14:11.647105 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644350    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 14:14:11.647105 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644352    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 14:14:11.647105 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644355    2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 14:14:11.647105 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644359    2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 14:14:11.647105 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644362    2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 14:14:11.647105 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644365    2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 14:14:11.647105 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644368    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 14:14:11.647105 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644371    2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 14:14:11.647105 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644374    2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 14:14:11.647105 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644376    2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 14:14:11.647105 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644379    2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 14:14:11.647591 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644383    2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 14:14:11.647591 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644403    2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 14:14:11.647591 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644407    2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 14:14:11.647591 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644409    2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 14:14:11.647591 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644413    2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 14:14:11.647591 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644417    2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 14:14:11.647591 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644421    2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 14:14:11.647591 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644425    2587 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 14:14:11.647591 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644428    2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 14:14:11.647591 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644430    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 14:14:11.647591 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644433    2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 14:14:11.647591 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644436    2587 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 14:14:11.647591 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644439    2587 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 14:14:11.647591 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644441    2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 14:14:11.647591 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644444    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 14:14:11.647591 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644447    2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 14:14:11.647591 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644450    2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 14:14:11.647591 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644452    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 14:14:11.647591 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644849    2587 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 14:14:11.647591 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644854    2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 14:14:11.648108 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644857    2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 14:14:11.648108 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644860    2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 14:14:11.648108 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644863    2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 14:14:11.648108 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644866    2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 14:14:11.648108 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644868    2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 14:14:11.648108 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644871    2587 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 14:14:11.648108 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644874    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 14:14:11.648108 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644876    2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 14:14:11.648108 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644879    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 14:14:11.648108 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644881    2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 14:14:11.648108 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644884    2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 14:14:11.648108 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644886    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 14:14:11.648108 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644889    2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 14:14:11.648108 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644891    2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 14:14:11.648108 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644894    2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 14:14:11.648108 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644897    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 14:14:11.648108 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644901    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 14:14:11.648108 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644903    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 14:14:11.648108 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644906    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 14:14:11.648108 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644909    2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 14:14:11.648618 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644911    2587 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 14:14:11.648618 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644914    2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 14:14:11.648618 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644916    2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 14:14:11.648618 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644919    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 14:14:11.648618 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644922    2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 14:14:11.648618 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644925    2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 14:14:11.648618 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644927    2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 14:14:11.648618 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644930    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 14:14:11.648618 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644933    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 14:14:11.648618 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644935    2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 14:14:11.648618 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644938    2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 14:14:11.648618 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644940    2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 14:14:11.648618 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644943    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 14:14:11.648618 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644945    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 14:14:11.648618 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644949    2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 14:14:11.648618 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644953    2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 14:14:11.648618 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644956    2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 14:14:11.648618 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644958    2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 14:14:11.648618 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644961    2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 14:14:11.649070 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644964    2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 14:14:11.649070 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644967    2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 14:14:11.649070 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644971    2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 14:14:11.649070 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644974    2587 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 14:14:11.649070 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644978    2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 14:14:11.649070 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644981    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 14:14:11.649070 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644984    2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 14:14:11.649070 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644987    2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 14:14:11.649070 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644990    2587 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 14:14:11.649070 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644993    2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 14:14:11.649070 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644997    2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 14:14:11.649070 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.644999    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 14:14:11.649070 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645002    2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 14:14:11.649070 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645005    2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 14:14:11.649070 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645007    2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 14:14:11.649070 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645010    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 14:14:11.649070 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645013    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 14:14:11.649070 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645015    2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 14:14:11.649070 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645018    2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 14:14:11.649585 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645020    2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 14:14:11.649585 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645023    2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 14:14:11.649585 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645025    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 14:14:11.649585 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645028    2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 14:14:11.649585 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645030    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 14:14:11.649585 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645033    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 14:14:11.649585 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645035    2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 14:14:11.649585 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645037    2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 14:14:11.649585 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645040    2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 14:14:11.649585 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645043    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 14:14:11.649585 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645045    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 14:14:11.649585 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645048    2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 14:14:11.649585 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645050    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 14:14:11.649585 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645053    2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 14:14:11.649585 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645056    2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 14:14:11.649585 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645058    2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 14:14:11.649585 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645061    2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 14:14:11.649585 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645063    2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 14:14:11.649585 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645065    2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 14:14:11.650034 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645068    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 14:14:11.650034 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645070    2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 14:14:11.650034 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645073    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 14:14:11.650034 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645075    2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 14:14:11.650034 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645078    2587 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 14:14:11.650034 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645083    2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 14:14:11.650034 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645086    2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 14:14:11.650034 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645088    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 14:14:11.650034 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645091    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 14:14:11.650034 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645093    2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 14:14:11.650034 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645096    2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 14:14:11.650034 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645098    2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 14:14:11.650034 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645101    2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 14:14:11.650034 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645103    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 14:14:11.650034 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645106    2587 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 14:14:11.650034 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645108    2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 14:14:11.650034 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645111    2587 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 14:14:11.650034 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645114    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 14:14:11.650034 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645117    2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 14:14:11.650034 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645119    2587 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 14:14:11.650034 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645210    2587 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 14:14:11.650568 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645250    2587 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 14:14:11.650568 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645259    2587 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 14:14:11.650568 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645263    2587 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 14:14:11.650568 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645268    2587 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 14:14:11.650568 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645271    2587 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 14:14:11.650568 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645275    2587 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 14:14:11.650568 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645280    2587 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 14:14:11.650568 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645283    2587 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 14:14:11.650568 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645286    2587 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 14:14:11.650568 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645290    2587 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 14:14:11.650568 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645294    2587 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 14:14:11.650568 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645297    2587 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 14:14:11.650568 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645300    2587 flags.go:64] FLAG: --cgroup-root=""
Sep 16 14:14:11.650568 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645303    2587 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 14:14:11.650568 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645306    2587 flags.go:64] FLAG: --client-ca-file=""
Sep 16 14:14:11.650568 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645309    2587 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 14:14:11.650568 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645314    2587 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 14:14:11.650568 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645320    2587 flags.go:64] FLAG: --cluster-domain=""
Sep 16 14:14:11.650568 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645323    2587 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 14:14:11.650568 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645327    2587 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 14:14:11.650568 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645330    2587 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 14:14:11.650568 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645333    2587 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 14:14:11.650568 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645337    2587 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 14:14:11.650568 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645340    2587 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 14:14:11.651187 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645343    2587 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 14:14:11.651187 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645346    2587 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 14:14:11.651187 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645350    2587 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 14:14:11.651187 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645352    2587 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 14:14:11.651187 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645355    2587 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 14:14:11.651187 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645358    2587 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 14:14:11.651187 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645361    2587 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 14:14:11.651187 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645366    2587 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 14:14:11.651187 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645369    2587 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 14:14:11.651187 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645372    2587 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 14:14:11.651187 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645374    2587 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 14:14:11.651187 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645378    2587 flags.go:64] FLAG: --enable-server="true"
Sep 16 14:14:11.651187 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645381    2587 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 14:14:11.651187 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645398    2587 flags.go:64] FLAG: --event-burst="100"
Sep 16 14:14:11.651187 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645402    2587 flags.go:64] FLAG: --event-qps="50"
Sep 16 14:14:11.651187 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645405    2587 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 14:14:11.651187 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645408    2587 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 14:14:11.651187 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645411    2587 flags.go:64] FLAG: --eviction-hard=""
Sep 16 14:14:11.651187 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645415    2587 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 14:14:11.651187 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645418    2587 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 14:14:11.651187 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645421    2587 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 14:14:11.651187 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645424    2587 flags.go:64] FLAG: --eviction-soft=""
Sep 16 14:14:11.651187 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645428    2587 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 14:14:11.651187 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645431    2587 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 14:14:11.651187 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645438    2587 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 14:14:11.651821 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645441    2587 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 14:14:11.651821 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645444    2587 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 14:14:11.651821 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645447    2587 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 14:14:11.651821 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645450    2587 flags.go:64] FLAG: --feature-gates=""
Sep 16 14:14:11.651821 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645454    2587 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 14:14:11.651821 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645457    2587 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 14:14:11.651821 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645461    2587 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 14:14:11.651821 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645464    2587 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 14:14:11.651821 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645467    2587 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 14:14:11.651821 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645470    2587 flags.go:64] FLAG: --help="false"
Sep 16 14:14:11.651821 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645473    2587 flags.go:64] FLAG: --hostname-override="ip-10-0-132-217.ec2.internal"
Sep 16 14:14:11.651821 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645476    2587 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 14:14:11.651821 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645479    2587 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 14:14:11.651821 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645482    2587 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 14:14:11.651821 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645485    2587 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 14:14:11.651821 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645488    2587 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 14:14:11.651821 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645491    2587 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 14:14:11.651821 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645494    2587 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 14:14:11.651821 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645496    2587 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 14:14:11.651821 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645500    2587 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 14:14:11.651821 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645503    2587 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 14:14:11.651821 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645506    2587 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 14:14:11.651821 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645508    2587 flags.go:64] FLAG: --kube-reserved=""
Sep 16 14:14:11.651821 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645511    2587 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 14:14:11.652444 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645514    2587 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 14:14:11.652444 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645517    2587 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 14:14:11.652444 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645520    2587 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 14:14:11.652444 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645523    2587 flags.go:64] FLAG: --lock-file=""
Sep 16 14:14:11.652444 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645526    2587 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 14:14:11.652444 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645531    2587 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 14:14:11.652444 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645534    2587 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 14:14:11.652444 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645540    2587 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 14:14:11.652444 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645543    2587 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 14:14:11.652444 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645546    2587 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 14:14:11.652444 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645549    2587 flags.go:64] FLAG: --logging-format="text"
Sep 16 14:14:11.652444 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645553    2587 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 14:14:11.652444 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645556    2587 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 14:14:11.652444 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645559    2587 flags.go:64] FLAG: --manifest-url=""
Sep 16 14:14:11.652444 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645562    2587 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 14:14:11.652444 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645566    2587 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 14:14:11.652444 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645569    2587 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 14:14:11.652444 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645573    2587 flags.go:64] FLAG: --max-pods="110"
Sep 16 14:14:11.652444 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645577    2587 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 14:14:11.652444 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645580    2587 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 14:14:11.652444 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645583    2587 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 14:14:11.652444 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645586    2587 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 14:14:11.652444 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645589    2587 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 14:14:11.652444 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645592    2587 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 14:14:11.652444 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645595    2587 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 14:14:11.653073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645603    2587 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 14:14:11.653073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645606    2587 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 14:14:11.653073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645609    2587 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 14:14:11.653073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645612    2587 flags.go:64] FLAG: --pod-cidr=""
Sep 16 14:14:11.653073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645616    2587 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 14:14:11.653073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645622    2587 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 14:14:11.653073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645626    2587 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 14:14:11.653073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645629    2587 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 14:14:11.653073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645632    2587 flags.go:64] FLAG: --port="10250"
Sep 16 14:14:11.653073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645635    2587 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 14:14:11.653073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645638    2587 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c9782ef41264ec7f"
Sep 16 14:14:11.653073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645641    2587 flags.go:64] FLAG: --qos-reserved=""
Sep 16 14:14:11.653073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645646    2587 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 14:14:11.653073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645649    2587 flags.go:64] FLAG: --register-node="true"
Sep 16 14:14:11.653073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645651    2587 flags.go:64] FLAG: --register-with-taints=""
Sep 16 14:14:11.653073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645656    2587 flags.go:64] FLAG: --registry-burst="10"
Sep 16 14:14:11.653073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645659    2587 flags.go:64] FLAG: --registry-qps="5"
Sep 16 14:14:11.653073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645662    2587 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 14:14:11.653073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645665    2587 flags.go:64] FLAG: --reserved-memory=""
Sep 16 14:14:11.653073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645687    2587 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 14:14:11.653073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645690    2587 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 14:14:11.653073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645693    2587 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 14:14:11.653073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645696    2587 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 14:14:11.653073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645699    2587 flags.go:64] FLAG: --runonce="false"
Sep 16 14:14:11.653073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645702    2587 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 14:14:11.653730 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645705    2587 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 14:14:11.653730 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645708    2587 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 14:14:11.653730 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645711    2587 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 14:14:11.653730 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645714    2587 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 14:14:11.653730 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645717    2587 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 14:14:11.653730 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645720    2587 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 14:14:11.653730 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645723    2587 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 14:14:11.653730 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645726    2587 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 14:14:11.653730 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645729    2587 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 14:14:11.653730 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645732    2587 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 14:14:11.653730 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645735    2587 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 14:14:11.653730 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645738    2587 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 14:14:11.653730 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645741    2587 flags.go:64] FLAG: --system-cgroups=""
Sep 16 14:14:11.653730 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645744    2587 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 14:14:11.653730 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645750    2587 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 14:14:11.653730 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645752    2587 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 14:14:11.653730 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645755    2587 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 14:14:11.653730 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645759    2587 flags.go:64] FLAG: --tls-min-version=""
Sep 16 14:14:11.653730 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645763    2587 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 14:14:11.653730 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645766    2587 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 14:14:11.653730 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645769    2587 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 14:14:11.653730 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645772    2587 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 14:14:11.653730 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645776    2587 flags.go:64] FLAG: --v="2"
Sep 16 14:14:11.653730 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645781    2587 flags.go:64] FLAG: --version="false"
Sep 16 14:14:11.653730 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645785    2587 flags.go:64] FLAG: --vmodule=""
Sep 16 14:14:11.654835 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645789    2587 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 14:14:11.654835 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.645793    2587 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 14:14:11.654835 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645917    2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 14:14:11.654835 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645921    2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 14:14:11.654835 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645924    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 14:14:11.654835 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645926    2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 14:14:11.654835 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645929    2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 14:14:11.654835 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645932    2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 14:14:11.654835 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645934    2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 14:14:11.654835 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645937    2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 14:14:11.654835 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645940    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 14:14:11.654835 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645942    2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 14:14:11.654835 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645945    2587 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 14:14:11.654835 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645947    2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 14:14:11.654835 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645950    2587 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 14:14:11.654835 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645953    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 14:14:11.654835 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645955    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 14:14:11.654835 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645958    2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 14:14:11.654835 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645961    2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 14:14:11.654835 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645965    2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 14:14:11.655435 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645967    2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 14:14:11.655435 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645970    2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 14:14:11.655435 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645973    2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 14:14:11.655435 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645975    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 14:14:11.655435 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645978    2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 14:14:11.655435 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645980    2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 14:14:11.655435 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645984    2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 14:14:11.655435 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645987    2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 14:14:11.655435 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645989    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 14:14:11.655435 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645994    2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 14:14:11.655435 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.645998    2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 14:14:11.655435 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646002    2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 14:14:11.655435 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646006    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 14:14:11.655435 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646010    2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 14:14:11.655435 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646013    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 14:14:11.655435 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646016    2587 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 14:14:11.655435 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646019    2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 14:14:11.655435 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646022    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 14:14:11.655435 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646024    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 14:14:11.655435 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646027    2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 14:14:11.655932 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646029    2587 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 14:14:11.655932 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646032    2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 14:14:11.655932 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646034    2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 14:14:11.655932 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646048    2587 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 14:14:11.655932 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646051    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 14:14:11.655932 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646054    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 14:14:11.655932 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646056    2587 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 14:14:11.655932 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646059    2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 14:14:11.655932 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646062    2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 14:14:11.655932 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646064    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 14:14:11.655932 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646067    2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 14:14:11.655932 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646070    2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 14:14:11.655932 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646072    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 14:14:11.655932 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646075    2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 14:14:11.655932 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646077    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 14:14:11.655932 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646080    2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 14:14:11.655932 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646084    2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 14:14:11.655932 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646088    2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 14:14:11.655932 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646093    2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 14:14:11.655932 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646095    2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 14:14:11.656546 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646098    2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 14:14:11.656546 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646102    2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 14:14:11.656546 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646105    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 14:14:11.656546 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646107    2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 14:14:11.656546 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646110    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 14:14:11.656546 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646113    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 14:14:11.656546 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646116    2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 14:14:11.656546 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646119    2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 14:14:11.656546 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646122    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 14:14:11.656546 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646124    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 14:14:11.656546 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646127    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 14:14:11.656546 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646130    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 14:14:11.656546 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646133    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 14:14:11.656546 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646136    2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 14:14:11.656546 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646139    2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 14:14:11.656546 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646142    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 14:14:11.656546 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646145    2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 14:14:11.656546 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646147    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 14:14:11.656546 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646149    2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 14:14:11.656546 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646152    2587 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 14:14:11.657051 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646155    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 14:14:11.657051 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646157    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 14:14:11.657051 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646160    2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 14:14:11.657051 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646162    2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 14:14:11.657051 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646165    2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 14:14:11.657051 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646167    2587 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 14:14:11.657051 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646170    2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 14:14:11.657051 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646173    2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 14:14:11.657051 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646176    2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 14:14:11.657051 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646180    2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 14:14:11.657051 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646185    2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 14:14:11.657051 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646189    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 14:14:11.657051 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646193    2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 14:14:11.657051 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646196    2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 14:14:11.657051 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646199    2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 14:14:11.657051 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646202    2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 14:14:11.657051 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646205    2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 14:14:11.657051 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646207    2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 14:14:11.657051 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646210    2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 14:14:11.657532 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646212    2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 14:14:11.657532 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.646216    2587 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 14:14:11.657532 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.646832    2587 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 14:14:11.657532 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.656070    2587 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 14:14:11.657532 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.656092    2587 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 14:14:11.657532 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.656754    2587 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 14:14:11.657532 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.656762    2587 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 14:14:11.657532 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656825    2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 14:14:11.657532 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656830    2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 14:14:11.657532 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656834    2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 14:14:11.657532 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656838    2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 14:14:11.657532 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656841    2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 14:14:11.657532 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656844    2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 14:14:11.657532 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656846    2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 14:14:11.657882 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656849    2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 14:14:11.657882 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656852    2587 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 14:14:11.657882 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656855    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 14:14:11.657882 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656857    2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 14:14:11.657882 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656860    2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 14:14:11.657882 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656862    2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 14:14:11.657882 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656865    2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 14:14:11.657882 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656868    2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 14:14:11.657882 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656871    2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 14:14:11.657882 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656873    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 14:14:11.657882 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656876    2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 14:14:11.657882 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656878    2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 14:14:11.657882 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656882    2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 14:14:11.657882 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656885    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 14:14:11.657882 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656888    2587 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 14:14:11.657882 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656890    2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 14:14:11.657882 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656893    2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 14:14:11.657882 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656895    2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 14:14:11.657882 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656898    2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 14:14:11.657882 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656901    2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 14:14:11.658372 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656909    2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 14:14:11.658372 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656912    2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 14:14:11.658372 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656915    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 14:14:11.658372 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656918    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 14:14:11.658372 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656921    2587 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 14:14:11.658372 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656924    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 14:14:11.658372 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656926    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 14:14:11.658372 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656929    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 14:14:11.658372 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656932    2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 14:14:11.658372 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656934    2587 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 14:14:11.658372 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656937    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 14:14:11.658372 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656940    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 14:14:11.658372 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656942    2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 14:14:11.658372 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656945    2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 14:14:11.658372 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656947    2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 14:14:11.658372 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656950    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 14:14:11.658372 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656952    2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 14:14:11.658372 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656955    2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 14:14:11.658372 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656957    2587 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 14:14:11.658372 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656960    2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 14:14:11.658911 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656962    2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 14:14:11.658911 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656965    2587 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 14:14:11.658911 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656968    2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 14:14:11.658911 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656971    2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 14:14:11.658911 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656974    2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 14:14:11.658911 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656976    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 14:14:11.658911 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656979    2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 14:14:11.658911 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656981    2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 14:14:11.658911 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656984    2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 14:14:11.658911 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656987    2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 14:14:11.658911 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656989    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 14:14:11.658911 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656992    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 14:14:11.658911 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656994    2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 14:14:11.658911 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.656997    2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 14:14:11.658911 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657001    2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 14:14:11.658911 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657004    2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 14:14:11.658911 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657007    2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 14:14:11.658911 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657009    2587 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 14:14:11.658911 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657012    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 14:14:11.658911 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657015    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 14:14:11.659437 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657017    2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 14:14:11.659437 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657020    2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 14:14:11.659437 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657024    2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 14:14:11.659437 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657029    2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 14:14:11.659437 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657032    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 14:14:11.659437 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657035    2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 14:14:11.659437 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657038    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 14:14:11.659437 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657041    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 14:14:11.659437 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657043    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 14:14:11.659437 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657046    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 14:14:11.659437 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657049    2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 14:14:11.659437 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657052    2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 14:14:11.659437 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657056    2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 14:14:11.659437 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657059    2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 14:14:11.659437 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657063    2587 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 14:14:11.659437 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657066    2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 14:14:11.659437 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657070    2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 14:14:11.659437 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657073    2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 14:14:11.659437 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657077    2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 14:14:11.659899 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657080    2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 14:14:11.659899 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657082    2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 14:14:11.659899 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657085    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 14:14:11.659899 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657088    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 14:14:11.659899 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657091    2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 14:14:11.659899 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657094    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 14:14:11.659899 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657097    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 14:14:11.659899 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657100    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 14:14:11.659899 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657103    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 14:14:11.659899 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657105    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 14:14:11.659899 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657109    2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 14:14:11.659899 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657114    2587 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 14:14:11.659899 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657117    2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 14:14:11.659899 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.657122    2587 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 14:14:11.659899 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657244    2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 14:14:11.660282 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657250    2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 14:14:11.660282 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657253    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 14:14:11.660282 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657256    2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 14:14:11.660282 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657259    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 14:14:11.660282 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657262    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 14:14:11.660282 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657265    2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 14:14:11.660282 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657267    2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 14:14:11.660282 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657271    2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 14:14:11.660282 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657274    2587 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 14:14:11.660282 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657277    2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 14:14:11.660282 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657279    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 14:14:11.660282 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657283    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 14:14:11.660282 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657286    2587 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 14:14:11.660282 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657289    2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 14:14:11.660282 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657291    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 14:14:11.660282 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657296    2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 14:14:11.660282 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657299    2587 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 14:14:11.660282 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657303    2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 14:14:11.660282 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657305    2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 14:14:11.660825 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657308    2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 14:14:11.660825 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657311    2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 14:14:11.660825 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657313    2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 14:14:11.660825 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657316    2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 14:14:11.660825 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657318    2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 14:14:11.660825 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657321    2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 14:14:11.660825 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657324    2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 14:14:11.660825 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657326    2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 14:14:11.660825 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657329    2587 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 14:14:11.660825 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657331    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 14:14:11.660825 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657334    2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 14:14:11.660825 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657337    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 14:14:11.660825 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657340    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 14:14:11.660825 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657343    2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 14:14:11.660825 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657345    2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 14:14:11.660825 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657348    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 14:14:11.660825 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657351    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 14:14:11.660825 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657353    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 14:14:11.660825 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657355    2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 14:14:11.660825 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657358    2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 14:14:11.661293 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657360    2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 14:14:11.661293 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657363    2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 14:14:11.661293 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657365    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 14:14:11.661293 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657368    2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 14:14:11.661293 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657370    2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 14:14:11.661293 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657374    2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 14:14:11.661293 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657376    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 14:14:11.661293 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657379    2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 14:14:11.661293 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657382    2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 14:14:11.661293 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657384    2587 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 14:14:11.661293 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657401    2587 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 14:14:11.661293 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657404    2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 14:14:11.661293 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657407    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 14:14:11.661293 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657409    2587 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 14:14:11.661293 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657412    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 14:14:11.661293 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657415    2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 14:14:11.661293 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657418    2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 14:14:11.661293 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657421    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 14:14:11.661293 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657423    2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 14:14:11.661293 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657426    2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 14:14:11.661791 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657428    2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 14:14:11.661791 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657431    2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 14:14:11.661791 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657434    2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 14:14:11.661791 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657437    2587 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 14:14:11.661791 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657440    2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 14:14:11.661791 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657442    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 14:14:11.661791 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657445    2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 14:14:11.661791 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657447    2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 14:14:11.661791 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657450    2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 14:14:11.661791 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657452    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 14:14:11.661791 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657455    2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 14:14:11.661791 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657459    2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 14:14:11.661791 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657462    2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 14:14:11.661791 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657465    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 14:14:11.661791 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657468    2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 14:14:11.661791 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657471    2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 14:14:11.661791 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657473    2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 14:14:11.661791 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657475    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 14:14:11.661791 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657479    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 14:14:11.661791 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657482    2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 14:14:11.662286 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657484    2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 14:14:11.662286 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657486    2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 14:14:11.662286 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657489    2587 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 14:14:11.662286 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657491    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 14:14:11.662286 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657494    2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 14:14:11.662286 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657496    2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 14:14:11.662286 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657501    2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 14:14:11.662286 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657505    2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 14:14:11.662286 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657507    2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 14:14:11.662286 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657510    2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 14:14:11.662286 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657513    2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 14:14:11.662286 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657515    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 14:14:11.662286 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657518    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 14:14:11.662286 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657520    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 14:14:11.662286 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657523    2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 14:14:11.662286 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657525    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 14:14:11.662286 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657528    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 14:14:11.662286 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657530    2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 14:14:11.662286 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:11.657533    2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 14:14:11.662753 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.657539    2587 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 14:14:11.662753 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.657671    2587 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 14:14:11.662753 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.661115    2587 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 14:14:11.662753 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.661955    2587 server.go:1019] "Starting client certificate rotation"
Sep 16 14:14:11.662753 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.662062    2587 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 14:14:11.662753 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.662104    2587 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 14:14:11.664339 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.664323    2587 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 14:14:11.671671 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.671527    2587 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 14:14:11.681107 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.681086    2587 log.go:25] "Validated CRI v1 runtime API"
Sep 16 14:14:11.685498 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.685377    2587 log.go:25] "Validated CRI v1 image API"
Sep 16 14:14:11.687076 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.687049    2587 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 14:14:11.690000 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.689979    2587 fs.go:135] Filesystem UUIDs: map[55b74645-3a7c-43fe-8ece-012863b9abc3:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8a155614-f478-42d1-a2b2-79a98b918740:/dev/nvme0n1p3]
Sep 16 14:14:11.690075 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.690000    2587 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 14:14:11.694993 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.694960    2587 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 14:14:11.697404 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.697252    2587 manager.go:216] Machine: {Timestamp:2026-09-16 14:14:11.6956419 +0000 UTC m=+0.299266924 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099391 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26895ad760b14d66d8f84893245570 SystemUUID:ec26895a-d760-b14d-66d8-f84893245570 BootID:5316216b-204e-4d40-9992-32d02603618c Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f9:8e:8e:c1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f9:8e:8e:c1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:0b:bb:97:aa:61 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 14:14:11.697404 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.697372    2587 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 14:14:11.697592 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.697517    2587 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 14:14:11.700434 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.700383    2587 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 14:14:11.700616 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.700442    2587 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-217.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 14:14:11.701185 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.701171    2587 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 14:14:11.701236 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.701191    2587 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 14:14:11.701236 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.701203    2587 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 14:14:11.701852 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.701839    2587 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 14:14:11.702810 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.702796    2587 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 14:14:11.706846 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.706829    2587 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 14:14:11.707005 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.706993    2587 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 14:14:11.707069 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.707034    2587 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 14:14:11.707069 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.707050    2587 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 14:14:11.707148 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.707073    2587 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 14:14:11.707148 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.707088    2587 kubelet.go:398] "Adding apiserver pod source"
Sep 16 14:14:11.707148 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.707101    2587 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 14:14:11.707527 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.707513    2587 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 14:14:11.707606 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.707537    2587 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 14:14:11.709711 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.709687    2587 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 14:14:11.712330 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.712314    2587 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 14:14:11.712412 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.712346    2587 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 14:14:11.714580 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.714567    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 14:14:11.714636 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.714586    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 14:14:11.714636 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.714592    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 14:14:11.714636 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.714598    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 14:14:11.714636 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.714606    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 14:14:11.714636 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.714611    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 14:14:11.714636 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.714624    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 14:14:11.714636 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.714629    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 14:14:11.714636 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.714637    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 14:14:11.714841 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.714643    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 14:14:11.714841 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.714652    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 14:14:11.714841 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.714657    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 14:14:11.715437 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.715416    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 14:14:11.715437 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.715427    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 14:14:11.718218 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.718190    2587 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-217.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 14:14:11.718218 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:11.718196    2587 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-217.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 14:14:11.718323 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:11.718207    2587 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 14:14:11.719243 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.719231    2587 server.go:1268] "Started kubelet"
Sep 16 14:14:11.719381 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.719319    2587 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 14:14:11.719495 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.719420    2587 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 14:14:11.719495 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.719326    2587 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 14:14:11.720096 ip-10-0-132-217 systemd[1]: Started Kubernetes Kubelet.
Sep 16 14:14:11.720729 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.720684    2587 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 14:14:11.722289 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.722266    2587 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 14:14:11.724150 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:11.724132    2587 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 14:14:11.726303 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.726285    2587 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 14:14:11.726879 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.726864    2587 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 14:14:11.727136 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:11.727114    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-217.ec2.internal\" not found"
Sep 16 14:14:11.727560 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.727541    2587 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 14:14:11.727809 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.727790    2587 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 14:14:11.727886 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.727811    2587 factory.go:55] Registering systemd factory
Sep 16 14:14:11.727886 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.727821    2587 factory.go:223] Registration of the systemd container factory successfully
Sep 16 14:14:11.728084 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.728071    2587 factory.go:153] Registering CRI-O factory
Sep 16 14:14:11.728154 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.728088    2587 factory.go:223] Registration of the crio container factory successfully
Sep 16 14:14:11.728154 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.728106    2587 factory.go:103] Registering Raw factory
Sep 16 14:14:11.728154 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.728122    2587 manager.go:1196] Started watching for new ooms in manager
Sep 16 14:14:11.729527 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.728604    2587 manager.go:349] Starting recovery of all containers
Sep 16 14:14:11.729886 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.729837    2587 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 14:14:11.729886 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.729865    2587 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 14:14:11.730030 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.730014    2587 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 14:14:11.730030 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.730026    2587 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 14:14:11.732605 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:11.731013    2587 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-217.ec2.internal.18d5d2a9581c4ce8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-217.ec2.internal,UID:ip-10-0-132-217.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-217.ec2.internal,},FirstTimestamp:2026-09-16 14:14:11.71919588 +0000 UTC m=+0.322820969,LastTimestamp:2026-09-16 14:14:11.71919588 +0000 UTC m=+0.322820969,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-217.ec2.internal,}"
Sep 16 14:14:11.732605 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:11.731922    2587 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-217.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 14:14:11.732605 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:11.732036    2587 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 14:14:11.742186 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.742159    2587 manager.go:354] Recovery completed
Sep 16 14:14:11.746840 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.746824    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 14:14:11.749134 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.749096    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-217.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 14:14:11.749236 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.749144    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-217.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 14:14:11.749236 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.749159    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-217.ec2.internal" event="NodeHasSufficientPID"
Sep 16 14:14:11.749849 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.749831    2587 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 14:14:11.749926 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.749853    2587 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 14:14:11.749926 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.749871    2587 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 14:14:11.749992 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.749835    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 14:14:11.750734 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.750717    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-217.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 14:14:11.750785 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.750751    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-217.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 14:14:11.750785 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.750761    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-217.ec2.internal" event="NodeHasSufficientPID"
Sep 16 14:14:11.751725 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:11.751650    2587 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-217.ec2.internal.18d5d2a959e50852  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-217.ec2.internal,UID:ip-10-0-132-217.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-217.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-217.ec2.internal,},FirstTimestamp:2026-09-16 14:14:11.749128274 +0000 UTC m=+0.352753297,LastTimestamp:2026-09-16 14:14:11.749128274 +0000 UTC m=+0.352753297,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-217.ec2.internal,}"
Sep 16 14:14:11.752152 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.752141    2587 policy_none.go:49] "None policy: Start"
Sep 16 14:14:11.752196 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.752157    2587 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 14:14:11.752196 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.752167    2587 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 14:14:11.754110 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.754099    2587 policy_none.go:47] "Start"
Sep 16 14:14:11.763107 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:11.763035    2587 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-217.ec2.internal.18d5d2a959e560f7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-217.ec2.internal,UID:ip-10-0-132-217.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-217.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-217.ec2.internal,},FirstTimestamp:2026-09-16 14:14:11.749150967 +0000 UTC m=+0.352775992,LastTimestamp:2026-09-16 14:14:11.749150967 +0000 UTC m=+0.352775992,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-217.ec2.internal,}"
Sep 16 14:14:11.772081 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:11.772007    2587 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-217.ec2.internal.18d5d2a959e594a1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-217.ec2.internal,UID:ip-10-0-132-217.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-217.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-217.ec2.internal,},FirstTimestamp:2026-09-16 14:14:11.749164193 +0000 UTC m=+0.352789216,LastTimestamp:2026-09-16 14:14:11.749164193 +0000 UTC m=+0.352789216,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-217.ec2.internal,}"
Sep 16 14:14:11.776983 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.776961    2587 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2r9zl"
Sep 16 14:14:11.782307 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:11.782231    2587 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-217.ec2.internal.18d5d2a959e50852\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-217.ec2.internal.18d5d2a959e50852  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-217.ec2.internal,UID:ip-10-0-132-217.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-217.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-217.ec2.internal,},FirstTimestamp:2026-09-16 14:14:11.749128274 +0000 UTC m=+0.352753297,LastTimestamp:2026-09-16 14:14:11.75073699 +0000 UTC m=+0.354362016,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-217.ec2.internal,}"
Sep 16 14:14:11.786146 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.786123    2587 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2r9zl"
Sep 16 14:14:11.792108 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.792091    2587 manager.go:337] "Starting Device Plugin manager"
Sep 16 14:14:11.792219 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:11.792136    2587 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 14:14:11.792219 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.792150    2587 server.go:87] "Starting device plugin registration server"
Sep 16 14:14:11.792435 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.792420    2587 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 14:14:11.792501 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.792437    2587 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 14:14:11.792552 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.792516    2587 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 14:14:11.792617 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.792604    2587 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 14:14:11.792617 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.792617    2587 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 14:14:11.793134 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:11.793118    2587 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 14:14:11.793234 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:11.793169    2587 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-217.ec2.internal\" not found"
Sep 16 14:14:11.893474 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.893358    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 14:14:11.893474 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.893362    2587 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 14:14:11.895844 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.894338    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-217.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 14:14:11.895844 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.894371    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-217.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 14:14:11.895844 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.894406    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-217.ec2.internal" event="NodeHasSufficientPID"
Sep 16 14:14:11.895844 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.894444    2587 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-217.ec2.internal"
Sep 16 14:14:11.895844 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.895785    2587 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 14:14:11.895844 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.895835    2587 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 14:14:11.896132 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.895860    2587 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 14:14:11.896132 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:11.895905    2587 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 14:14:11.899219 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.899194    2587 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 14:14:11.903301 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.903284    2587 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-217.ec2.internal"
Sep 16 14:14:11.903356 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:11.903309    2587 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-217.ec2.internal\": node \"ip-10-0-132-217.ec2.internal\" not found"
Sep 16 14:14:11.932350 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:11.932320    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-217.ec2.internal\" not found"
Sep 16 14:14:11.996598 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.996564    2587 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-217.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-217.ec2.internal"]
Sep 16 14:14:11.996711 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.996668    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 14:14:11.997647 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.997631    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-217.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 14:14:11.997720 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.997662    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-217.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 14:14:11.997720 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:11.997674    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-217.ec2.internal" event="NodeHasSufficientPID"
Sep 16 14:14:12.000075 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.000061    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 14:14:12.000690 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.000674    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-217.ec2.internal"
Sep 16 14:14:12.000742 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.000710    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 14:14:12.000848 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.000824    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-217.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 14:14:12.000994 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.000857    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-217.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 14:14:12.000994 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.000870    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-217.ec2.internal" event="NodeHasSufficientPID"
Sep 16 14:14:12.003331 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.003310    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-217.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 14:14:12.003487 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.003343    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-217.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 14:14:12.003487 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.003353    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-217.ec2.internal" event="NodeHasSufficientPID"
Sep 16 14:14:12.004097 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.004080    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-217.ec2.internal"
Sep 16 14:14:12.004144 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.004117    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 14:14:12.004853 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.004837    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-217.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 14:14:12.004944 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.004859    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-217.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 14:14:12.004944 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.004869    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-217.ec2.internal" event="NodeHasSufficientPID"
Sep 16 14:14:12.032067 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.032031    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a33e9dba3903cb84af35554e9153b83a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-217.ec2.internal\" (UID: \"a33e9dba3903cb84af35554e9153b83a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-217.ec2.internal"
Sep 16 14:14:12.032244 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.032090    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a33e9dba3903cb84af35554e9153b83a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-217.ec2.internal\" (UID: \"a33e9dba3903cb84af35554e9153b83a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-217.ec2.internal"
Sep 16 14:14:12.032244 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.032109    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e37d493554911b356e79c3add3a4b130-config\") pod \"kube-apiserver-proxy-ip-10-0-132-217.ec2.internal\" (UID: \"e37d493554911b356e79c3add3a4b130\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-217.ec2.internal"
Sep 16 14:14:12.032783 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:12.032764    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-217.ec2.internal\" not found"
Sep 16 14:14:12.034224 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:12.034205    2587 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-217.ec2.internal\" not found" node="ip-10-0-132-217.ec2.internal"
Sep 16 14:14:12.038589 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:12.038568    2587 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-217.ec2.internal\" not found" node="ip-10-0-132-217.ec2.internal"
Sep 16 14:14:12.132823 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.132784    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a33e9dba3903cb84af35554e9153b83a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-217.ec2.internal\" (UID: \"a33e9dba3903cb84af35554e9153b83a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-217.ec2.internal"
Sep 16 14:14:12.132992 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.132838    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a33e9dba3903cb84af35554e9153b83a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-217.ec2.internal\" (UID: \"a33e9dba3903cb84af35554e9153b83a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-217.ec2.internal"
Sep 16 14:14:12.132992 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.132860    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e37d493554911b356e79c3add3a4b130-config\") pod \"kube-apiserver-proxy-ip-10-0-132-217.ec2.internal\" (UID: \"e37d493554911b356e79c3add3a4b130\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-217.ec2.internal"
Sep 16 14:14:12.132992 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:12.132882    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-217.ec2.internal\" not found"
Sep 16 14:14:12.132992 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.132919    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a33e9dba3903cb84af35554e9153b83a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-217.ec2.internal\" (UID: \"a33e9dba3903cb84af35554e9153b83a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-217.ec2.internal"
Sep 16 14:14:12.132992 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.132931    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e37d493554911b356e79c3add3a4b130-config\") pod \"kube-apiserver-proxy-ip-10-0-132-217.ec2.internal\" (UID: \"e37d493554911b356e79c3add3a4b130\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-217.ec2.internal"
Sep 16 14:14:12.132992 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.132939    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a33e9dba3903cb84af35554e9153b83a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-217.ec2.internal\" (UID: \"a33e9dba3903cb84af35554e9153b83a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-217.ec2.internal"
Sep 16 14:14:12.233913 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:12.233827    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-217.ec2.internal\" not found"
Sep 16 14:14:12.334240 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:12.334200    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-217.ec2.internal\" not found"
Sep 16 14:14:12.337367 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.337351    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-217.ec2.internal"
Sep 16 14:14:12.342434 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.342413    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-217.ec2.internal"
Sep 16 14:14:12.434839 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:12.434788    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-217.ec2.internal\" not found"
Sep 16 14:14:12.535285 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:12.535216    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-217.ec2.internal\" not found"
Sep 16 14:14:12.635756 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:12.635719    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-217.ec2.internal\" not found"
Sep 16 14:14:12.661897 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.661865    2587 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 14:14:12.662273 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.662050    2587 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 14:14:12.736146 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:12.736116    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-217.ec2.internal\" not found"
Sep 16 14:14:12.789973 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.789888    2587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 14:09:11 +0000 UTC" deadline="2028-06-18 11:22:12.688705634 +0000 UTC"
Sep 16 14:14:12.789973 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.789935    2587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15381h7m59.898780426s"
Sep 16 14:14:12.836447 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:12.836413    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-217.ec2.internal\" not found"
Sep 16 14:14:12.872646 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.872624    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 14:14:12.874068 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.874044    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-217.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 14:14:12.874189 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.874081    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-217.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 14:14:12.874189 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:12.874098    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-217.ec2.internal" event="NodeHasSufficientPID"
Sep 16 14:14:12.937309 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:12.937276    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-217.ec2.internal\" not found"
Sep 16 14:14:13.012059 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:13.012026    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33e9dba3903cb84af35554e9153b83a.slice/crio-7d09826054c5919f26aa2414dc6057fcd2390cd4fa808363017048135f40e92a WatchSource:0}: Error finding container 7d09826054c5919f26aa2414dc6057fcd2390cd4fa808363017048135f40e92a: Status 404 returned error can't find the container with id 7d09826054c5919f26aa2414dc6057fcd2390cd4fa808363017048135f40e92a
Sep 16 14:14:13.015430 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.015412    2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 14:14:13.038255 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:13.038209    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-217.ec2.internal\" not found"
Sep 16 14:14:13.106027 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.105935    2587 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 14:14:13.108805 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.108786    2587 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 14:14:13.138950 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:13.138926    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-217.ec2.internal\" not found"
Sep 16 14:14:13.194879 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.194843    2587 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 14:14:13.211601 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:13.211565    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode37d493554911b356e79c3add3a4b130.slice/crio-baf8049edad7e85f938fa8f582d01e72427fa39ff9226bab98a9ef6eaeb9a116 WatchSource:0}: Error finding container baf8049edad7e85f938fa8f582d01e72427fa39ff9226bab98a9ef6eaeb9a116: Status 404 returned error can't find the container with id baf8049edad7e85f938fa8f582d01e72427fa39ff9226bab98a9ef6eaeb9a116
Sep 16 14:14:13.227497 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.227463    2587 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-217.ec2.internal"
Sep 16 14:14:13.241181 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.241157    2587 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 14:14:13.242143 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.242128    2587 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-217.ec2.internal"
Sep 16 14:14:13.251796 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.251776    2587 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 14:14:13.709350 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.709315    2587 apiserver.go:52] "Watching apiserver"
Sep 16 14:14:13.717852 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.717821    2587 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 14:14:13.718249 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.718226    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-vtckf","openshift-network-operator/iptables-alerter-2dl6w","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t","openshift-cluster-node-tuning-operator/tuned-mjxbc","openshift-dns/node-resolver-trl7p","openshift-multus/network-metrics-daemon-6qpvt","openshift-network-diagnostics/network-check-target-h7q7r","openshift-ovn-kubernetes/ovnkube-node-zbsc4","kube-system/global-pull-secret-syncer-x59fz","kube-system/konnectivity-agent-vk8ld","kube-system/kube-apiserver-proxy-ip-10-0-132-217.ec2.internal","openshift-image-registry/node-ca-drp7m","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-217.ec2.internal","openshift-multus/multus-4d7hc"]
Sep 16 14:14:13.720907 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.720883    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:14:13.721036 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:13.720965    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7q7r" podUID="4e774454-f78b-40ba-ae21-ed6a11caaea5"
Sep 16 14:14:13.723422 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.723384    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.726252 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.726147    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t"
Sep 16 14:14:13.727387 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.726950    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 14:14:13.727387 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.727093    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 14:14:13.727387 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.727214    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 14:14:13.727387 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.727243    2587 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 14:14:13.727994 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.727972    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ppct7\""
Sep 16 14:14:13.728213 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.728176    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 14:14:13.728313 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.728290    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 14:14:13.728533 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.728516    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 14:14:13.729649 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.728850    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 14:14:13.729649 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.728862    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q9gn8\""
Sep 16 14:14:13.729649 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.728983    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 14:14:13.729649 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.729157    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.730883 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.730855    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 14:14:13.731712 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.731640    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 14:14:13.731812 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.731739    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 14:14:13.732297 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.732038    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2f6v9\""
Sep 16 14:14:13.732297 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.732053    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-trl7p"
Sep 16 14:14:13.734418 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.734377    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:13.734522 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:13.734482    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qpvt" podUID="e10f563c-16f8-4699-a7d4-52e6bfba29a5"
Sep 16 14:14:13.735267 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.735241    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9qxnn\""
Sep 16 14:14:13.735494 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.735475    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 14:14:13.735565 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.735548    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 14:14:13.736942 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.736924    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:13.737052 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:13.737012    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x59fz" podUID="6f12cf92-2a09-4fe6-a73e-5692728769b3"
Sep 16 14:14:13.739503 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.739483    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vtckf"
Sep 16 14:14:13.741856 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.741809    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-run-ovn\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.741856 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.741839    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-host-cni-bin\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.741988 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.741865    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/60bd4ded-155c-4351-af05-3e2be2bfc03c-ovnkube-script-lib\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.741988 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.741891    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxch6\" (UniqueName: \"kubernetes.io/projected/60bd4ded-155c-4351-af05-3e2be2bfc03c-kube-api-access-kxch6\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.741988 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.741918    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5a76ffd0-362f-4576-8836-483741c783d0-device-dir\") pod \"aws-ebs-csi-driver-node-6xf9t\" (UID: \"5a76ffd0-362f-4576-8836-483741c783d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t"
Sep 16 14:14:13.741988 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.741941    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-host-run-ovn-kubernetes\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.741988 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.741965    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5a76ffd0-362f-4576-8836-483741c783d0-socket-dir\") pod \"aws-ebs-csi-driver-node-6xf9t\" (UID: \"5a76ffd0-362f-4576-8836-483741c783d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t"
Sep 16 14:14:13.742240 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.741989    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-etc-sysconfig\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.742240 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742016    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-etc-kubernetes\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.742240 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742041    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-sys\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.742240 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742064    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/60bd4ded-155c-4351-af05-3e2be2bfc03c-ovnkube-config\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.742240 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742091    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfwbq\" (UniqueName: \"kubernetes.io/projected/4e774454-f78b-40ba-ae21-ed6a11caaea5-kube-api-access-hfwbq\") pod \"network-check-target-h7q7r\" (UID: \"4e774454-f78b-40ba-ae21-ed6a11caaea5\") " pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:14:13.742240 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742139    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-systemd-units\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.742240 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742165    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-host-run-netns\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.742240 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742202    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-etc-openvswitch\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.742240 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742227    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-run-openvswitch\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.742737 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742257    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-lib-modules\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.742737 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742284    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b4f78308-a546-426b-8863-898b414f6c29-etc-tuned\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.742737 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742311    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-node-log\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.742737 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742356    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.742737 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742387    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/60bd4ded-155c-4351-af05-3e2be2bfc03c-ovn-node-metrics-cert\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.742737 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742412    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5a76ffd0-362f-4576-8836-483741c783d0-sys-fs\") pod \"aws-ebs-csi-driver-node-6xf9t\" (UID: \"5a76ffd0-362f-4576-8836-483741c783d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t"
Sep 16 14:14:13.742737 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742438    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-etc-modprobe-d\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.742737 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742512    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-etc-systemd\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.742737 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742557    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 14:14:13.742737 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742574    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4f78308-a546-426b-8863-898b414f6c29-tmp\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.742737 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742642    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/60bd4ded-155c-4351-af05-3e2be2bfc03c-env-overrides\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.742737 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742659    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-etc-sysctl-conf\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.742737 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742674    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-run\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.742737 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742689    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-var-lib-kubelet\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.742737 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742704    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-host\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.742737 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742719    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnhqx\" (UniqueName: \"kubernetes.io/projected/b4f78308-a546-426b-8863-898b414f6c29-kube-api-access-nnhqx\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.742737 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742739    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 14:14:13.743586 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742747    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-host-kubelet\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.743586 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742779    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-run-systemd\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.743586 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742792    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 14:14:13.743586 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742808    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5a76ffd0-362f-4576-8836-483741c783d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-6xf9t\" (UID: \"5a76ffd0-362f-4576-8836-483741c783d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t"
Sep 16 14:14:13.743586 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742824    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-host-slash\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.743586 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742839    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-log-socket\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.743586 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742854    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-host-cni-netd\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.743586 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742878    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 14:14:13.743586 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742883    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkx2f\" (UniqueName: \"kubernetes.io/projected/5a76ffd0-362f-4576-8836-483741c783d0-kube-api-access-pkx2f\") pod \"aws-ebs-csi-driver-node-6xf9t\" (UID: \"5a76ffd0-362f-4576-8836-483741c783d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t"
Sep 16 14:14:13.743586 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742899    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-etc-sysctl-d\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.743586 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742915    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-var-lib-openvswitch\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.743586 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742934    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5a76ffd0-362f-4576-8836-483741c783d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6xf9t\" (UID: \"5a76ffd0-362f-4576-8836-483741c783d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t"
Sep 16 14:14:13.743586 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.742951    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5a76ffd0-362f-4576-8836-483741c783d0-registration-dir\") pod \"aws-ebs-csi-driver-node-6xf9t\" (UID: \"5a76ffd0-362f-4576-8836-483741c783d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t"
Sep 16 14:14:13.743586 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.743034    2587 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 14:14:13.743586 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.743071    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 14:14:13.743586 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.743230    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2dl6w"
Sep 16 14:14:13.743586 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.743287    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tv9ch\""
Sep 16 14:14:13.744597 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.744578    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vk8ld"
Sep 16 14:14:13.746727 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.746705    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 14:14:13.746879 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.746714    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 14:14:13.746879 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.746820    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 14:14:13.747129 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.747108    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-78lrp\""
Sep 16 14:14:13.747216 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.747205    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 14:14:13.747277 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.747211    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-crhd9\""
Sep 16 14:14:13.747344 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.747327    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 14:14:13.747663 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.747643    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.749956 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.749934    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-drp7m"
Sep 16 14:14:13.750559 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.750540    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9w52m\""
Sep 16 14:14:13.750658 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.750594    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 14:14:13.752472 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.752449    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 14:14:13.752633 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.752482    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 14:14:13.752633 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.752553    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rcp94\""
Sep 16 14:14:13.752633 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.752571    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 14:14:13.767197 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.767171    2587 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-m58kx"
Sep 16 14:14:13.776333 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.776308    2587 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m58kx"
Sep 16 14:14:13.828864 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.828825    2587 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 14:14:13.843216 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843185    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-host-cni-bin\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.843393 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843227    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/60bd4ded-155c-4351-af05-3e2be2bfc03c-ovnkube-script-lib\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.843393 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843256    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxch6\" (UniqueName: \"kubernetes.io/projected/60bd4ded-155c-4351-af05-3e2be2bfc03c-kube-api-access-kxch6\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.843393 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843323    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-host-cni-bin\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.843393 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843378    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5a76ffd0-362f-4576-8836-483741c783d0-device-dir\") pod \"aws-ebs-csi-driver-node-6xf9t\" (UID: \"5a76ffd0-362f-4576-8836-483741c783d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t"
Sep 16 14:14:13.843638 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843432    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/489a66c3-b01a-45d8-8747-4ee83b84958e-agent-certs\") pod \"konnectivity-agent-vk8ld\" (UID: \"489a66c3-b01a-45d8-8747-4ee83b84958e\") " pod="kube-system/konnectivity-agent-vk8ld"
Sep 16 14:14:13.843638 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843457    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-host-var-lib-cni-multus\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.843638 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843477    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5a76ffd0-362f-4576-8836-483741c783d0-socket-dir\") pod \"aws-ebs-csi-driver-node-6xf9t\" (UID: \"5a76ffd0-362f-4576-8836-483741c783d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t"
Sep 16 14:14:13.843638 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843496    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmmgg\" (UniqueName: \"kubernetes.io/projected/e71666fa-da06-4d9b-bb6f-383354341f68-kube-api-access-tmmgg\") pod \"multus-additional-cni-plugins-vtckf\" (UID: \"e71666fa-da06-4d9b-bb6f-383354341f68\") " pod="openshift-multus/multus-additional-cni-plugins-vtckf"
Sep 16 14:14:13.843638 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843520    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-etc-openvswitch\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.843638 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843537    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-lib-modules\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.843638 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843553    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b4f78308-a546-426b-8863-898b414f6c29-etc-tuned\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.843638 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843568    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-host-run-k8s-cni-cncf-io\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.843638 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843583    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5130c0de-1444-4258-8e1d-7b00b0b042a3-multus-daemon-config\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.843638 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843605    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d657d718-fd8c-4034-9cd8-232f90fc573e-serviceca\") pod \"node-ca-drp7m\" (UID: \"d657d718-fd8c-4034-9cd8-232f90fc573e\") " pod="openshift-image-registry/node-ca-drp7m"
Sep 16 14:14:13.843638 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843636    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/60bd4ded-155c-4351-af05-3e2be2bfc03c-ovn-node-metrics-cert\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.844194 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843653    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-etc-systemd\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.844194 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843668    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50d9276f-9ed0-42b9-8264-7c2baf54f7c1-host-slash\") pod \"iptables-alerter-2dl6w\" (UID: \"50d9276f-9ed0-42b9-8264-7c2baf54f7c1\") " pod="openshift-network-operator/iptables-alerter-2dl6w"
Sep 16 14:14:13.844194 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843691    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-etc-sysctl-conf\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.844194 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843708    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-var-lib-kubelet\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.844194 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843723    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnhqx\" (UniqueName: \"kubernetes.io/projected/b4f78308-a546-426b-8863-898b414f6c29-kube-api-access-nnhqx\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.844194 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843739    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e71666fa-da06-4d9b-bb6f-383354341f68-cni-binary-copy\") pod \"multus-additional-cni-plugins-vtckf\" (UID: \"e71666fa-da06-4d9b-bb6f-383354341f68\") " pod="openshift-multus/multus-additional-cni-plugins-vtckf"
Sep 16 14:14:13.844194 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843757    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f12cf92-2a09-4fe6-a73e-5692728769b3-kubelet-config\") pod \"global-pull-secret-syncer-x59fz\" (UID: \"6f12cf92-2a09-4fe6-a73e-5692728769b3\") " pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:13.844194 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843780    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-host-kubelet\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.844194 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843797    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5a76ffd0-362f-4576-8836-483741c783d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-6xf9t\" (UID: \"5a76ffd0-362f-4576-8836-483741c783d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t"
Sep 16 14:14:13.844194 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843814    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-multus-cni-dir\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.844194 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843830    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-cnibin\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.844194 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843843    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/60bd4ded-155c-4351-af05-3e2be2bfc03c-ovnkube-script-lib\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.844194 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843847    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e71666fa-da06-4d9b-bb6f-383354341f68-cnibin\") pod \"multus-additional-cni-plugins-vtckf\" (UID: \"e71666fa-da06-4d9b-bb6f-383354341f68\") " pod="openshift-multus/multus-additional-cni-plugins-vtckf"
Sep 16 14:14:13.844194 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843901    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5a76ffd0-362f-4576-8836-483741c783d0-device-dir\") pod \"aws-ebs-csi-driver-node-6xf9t\" (UID: \"5a76ffd0-362f-4576-8836-483741c783d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t"
Sep 16 14:14:13.844194 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843908    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x88p5\" (UniqueName: \"kubernetes.io/projected/d657d718-fd8c-4034-9cd8-232f90fc573e-kube-api-access-x88p5\") pod \"node-ca-drp7m\" (UID: \"d657d718-fd8c-4034-9cd8-232f90fc573e\") " pod="openshift-image-registry/node-ca-drp7m"
Sep 16 14:14:13.844194 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843930    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4193914e-79e1-4c6a-ba3b-a4d9f5731637-tmp-dir\") pod \"node-resolver-trl7p\" (UID: \"4193914e-79e1-4c6a-ba3b-a4d9f5731637\") " pod="openshift-dns/node-resolver-trl7p"
Sep 16 14:14:13.844194 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843946    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e10f563c-16f8-4699-a7d4-52e6bfba29a5-metrics-certs\") pod \"network-metrics-daemon-6qpvt\" (UID: \"e10f563c-16f8-4699-a7d4-52e6bfba29a5\") " pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:13.845036 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843966    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-host-slash\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.845036 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.843982    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-log-socket\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.845036 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844000    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkx2f\" (UniqueName: \"kubernetes.io/projected/5a76ffd0-362f-4576-8836-483741c783d0-kube-api-access-pkx2f\") pod \"aws-ebs-csi-driver-node-6xf9t\" (UID: \"5a76ffd0-362f-4576-8836-483741c783d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t"
Sep 16 14:14:13.845036 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844016    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-etc-sysctl-d\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.845036 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844020    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5a76ffd0-362f-4576-8836-483741c783d0-socket-dir\") pod \"aws-ebs-csi-driver-node-6xf9t\" (UID: \"5a76ffd0-362f-4576-8836-483741c783d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t"
Sep 16 14:14:13.845036 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844031    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-host-var-lib-cni-bin\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.845036 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844049    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-multus-conf-dir\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.845036 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844064    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-etc-openvswitch\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.845036 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844065    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2mpd\" (UniqueName: \"kubernetes.io/projected/5130c0de-1444-4258-8e1d-7b00b0b042a3-kube-api-access-c2mpd\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.845036 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844087    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-var-lib-openvswitch\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.845036 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844116    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5a76ffd0-362f-4576-8836-483741c783d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6xf9t\" (UID: \"5a76ffd0-362f-4576-8836-483741c783d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t"
Sep 16 14:14:13.845036 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844134    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5a76ffd0-362f-4576-8836-483741c783d0-registration-dir\") pod \"aws-ebs-csi-driver-node-6xf9t\" (UID: \"5a76ffd0-362f-4576-8836-483741c783d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t"
Sep 16 14:14:13.845036 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844137    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-log-socket\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.845036 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844152    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e71666fa-da06-4d9b-bb6f-383354341f68-system-cni-dir\") pod \"multus-additional-cni-plugins-vtckf\" (UID: \"e71666fa-da06-4d9b-bb6f-383354341f68\") " pod="openshift-multus/multus-additional-cni-plugins-vtckf"
Sep 16 14:14:13.845036 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844170    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f12cf92-2a09-4fe6-a73e-5692728769b3-original-pull-secret\") pod \"global-pull-secret-syncer-x59fz\" (UID: \"6f12cf92-2a09-4fe6-a73e-5692728769b3\") " pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:13.845036 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844193    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-run-ovn\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.845036 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844212    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-multus-socket-dir-parent\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.845843 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844229    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-host-run-multus-certs\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.845843 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844260    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4193914e-79e1-4c6a-ba3b-a4d9f5731637-hosts-file\") pod \"node-resolver-trl7p\" (UID: \"4193914e-79e1-4c6a-ba3b-a4d9f5731637\") " pod="openshift-dns/node-resolver-trl7p"
Sep 16 14:14:13.845843 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844285    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-host-run-ovn-kubernetes\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.845843 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844305    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-etc-sysconfig\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.845843 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844321    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-etc-kubernetes\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.845843 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844367    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-sys\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.845843 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844386    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/489a66c3-b01a-45d8-8747-4ee83b84958e-konnectivity-ca\") pod \"konnectivity-agent-vk8ld\" (UID: \"489a66c3-b01a-45d8-8747-4ee83b84958e\") " pod="kube-system/konnectivity-agent-vk8ld"
Sep 16 14:14:13.845843 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844402    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-etc-sysctl-d\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.845843 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844410    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e71666fa-da06-4d9b-bb6f-383354341f68-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vtckf\" (UID: \"e71666fa-da06-4d9b-bb6f-383354341f68\") " pod="openshift-multus/multus-additional-cni-plugins-vtckf"
Sep 16 14:14:13.845843 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844443    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/60bd4ded-155c-4351-af05-3e2be2bfc03c-ovnkube-config\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.845843 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844488    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfwbq\" (UniqueName: \"kubernetes.io/projected/4e774454-f78b-40ba-ae21-ed6a11caaea5-kube-api-access-hfwbq\") pod \"network-check-target-h7q7r\" (UID: \"4e774454-f78b-40ba-ae21-ed6a11caaea5\") " pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:14:13.845843 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844515    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-systemd-units\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.845843 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844531    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-host-run-netns\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.845843 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844548    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-run-openvswitch\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.845843 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844577    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-host-var-lib-kubelet\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.845843 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844599    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2f65\" (UniqueName: \"kubernetes.io/projected/e10f563c-16f8-4699-a7d4-52e6bfba29a5-kube-api-access-d2f65\") pod \"network-metrics-daemon-6qpvt\" (UID: \"e10f563c-16f8-4699-a7d4-52e6bfba29a5\") " pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:13.845843 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844620    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-node-log\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.846524 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844641    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.846524 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844661    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5a76ffd0-362f-4576-8836-483741c783d0-sys-fs\") pod \"aws-ebs-csi-driver-node-6xf9t\" (UID: \"5a76ffd0-362f-4576-8836-483741c783d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t"
Sep 16 14:14:13.846524 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844689    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-etc-modprobe-d\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.846524 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844796    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-etc-modprobe-d\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.846524 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844851    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-lib-modules\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.846524 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844907    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-host-kubelet\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.846524 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844967    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5a76ffd0-362f-4576-8836-483741c783d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-6xf9t\" (UID: \"5a76ffd0-362f-4576-8836-483741c783d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t"
Sep 16 14:14:13.846524 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845004    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-run-ovn\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.846524 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845016    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-var-lib-openvswitch\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.846524 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.844117    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-host-slash\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.846524 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845066    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5a76ffd0-362f-4576-8836-483741c783d0-registration-dir\") pod \"aws-ebs-csi-driver-node-6xf9t\" (UID: \"5a76ffd0-362f-4576-8836-483741c783d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t"
Sep 16 14:14:13.846524 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845068    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-host-run-ovn-kubernetes\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.846524 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845096    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5a76ffd0-362f-4576-8836-483741c783d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6xf9t\" (UID: \"5a76ffd0-362f-4576-8836-483741c783d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t"
Sep 16 14:14:13.846524 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845110    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-etc-sysconfig\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.846524 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845140    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-host-run-netns\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.846524 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845152    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-run-openvswitch\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.846524 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845197    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-etc-kubernetes\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.847241 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845200    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-node-log\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.847241 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845235    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4f78308-a546-426b-8863-898b414f6c29-tmp\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.847241 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845282    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.847241 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845287    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-sys\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.847241 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845312    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5a76ffd0-362f-4576-8836-483741c783d0-sys-fs\") pod \"aws-ebs-csi-driver-node-6xf9t\" (UID: \"5a76ffd0-362f-4576-8836-483741c783d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t"
Sep 16 14:14:13.847241 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845322    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5130c0de-1444-4258-8e1d-7b00b0b042a3-cni-binary-copy\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.847241 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845344    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-hostroot\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.847241 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845363    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/60bd4ded-155c-4351-af05-3e2be2bfc03c-env-overrides\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.847241 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845380    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-run\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.847241 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845390    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-etc-sysctl-conf\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.847241 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845402    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-host\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.847241 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845418    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-etc-systemd\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.847241 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845437    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/50d9276f-9ed0-42b9-8264-7c2baf54f7c1-iptables-alerter-script\") pod \"iptables-alerter-2dl6w\" (UID: \"50d9276f-9ed0-42b9-8264-7c2baf54f7c1\") " pod="openshift-network-operator/iptables-alerter-2dl6w"
Sep 16 14:14:13.847241 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845414    2587 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 14:14:13.847241 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845455    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-var-lib-kubelet\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.847241 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845457    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d657d718-fd8c-4034-9cd8-232f90fc573e-host\") pod \"node-ca-drp7m\" (UID: \"d657d718-fd8c-4034-9cd8-232f90fc573e\") " pod="openshift-image-registry/node-ca-drp7m"
Sep 16 14:14:13.847241 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845563    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-run\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.847241 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845601    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b4f78308-a546-426b-8863-898b414f6c29-host\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.848170 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845656    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-run-systemd\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.848170 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845689    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-system-cni-dir\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.848170 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845713    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-host-run-netns\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.848170 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845732    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpsq8\" (UniqueName: \"kubernetes.io/projected/50d9276f-9ed0-42b9-8264-7c2baf54f7c1-kube-api-access-tpsq8\") pod \"iptables-alerter-2dl6w\" (UID: \"50d9276f-9ed0-42b9-8264-7c2baf54f7c1\") " pod="openshift-network-operator/iptables-alerter-2dl6w"
Sep 16 14:14:13.848170 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845752    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f12cf92-2a09-4fe6-a73e-5692728769b3-dbus\") pod \"global-pull-secret-syncer-x59fz\" (UID: \"6f12cf92-2a09-4fe6-a73e-5692728769b3\") " pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:13.848170 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845770    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-host-cni-netd\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.848170 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845836    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e71666fa-da06-4d9b-bb6f-383354341f68-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vtckf\" (UID: \"e71666fa-da06-4d9b-bb6f-383354341f68\") " pod="openshift-multus/multus-additional-cni-plugins-vtckf"
Sep 16 14:14:13.848170 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845849    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-host-cni-netd\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.848170 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845877    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-systemd-units\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.848170 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845881    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-etc-kubernetes\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.848170 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.845896    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/60bd4ded-155c-4351-af05-3e2be2bfc03c-run-systemd\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.848170 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.846002    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e71666fa-da06-4d9b-bb6f-383354341f68-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vtckf\" (UID: \"e71666fa-da06-4d9b-bb6f-383354341f68\") " pod="openshift-multus/multus-additional-cni-plugins-vtckf"
Sep 16 14:14:13.848170 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.846047    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4dc7\" (UniqueName: \"kubernetes.io/projected/4193914e-79e1-4c6a-ba3b-a4d9f5731637-kube-api-access-s4dc7\") pod \"node-resolver-trl7p\" (UID: \"4193914e-79e1-4c6a-ba3b-a4d9f5731637\") " pod="openshift-dns/node-resolver-trl7p"
Sep 16 14:14:13.848170 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.846300    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/60bd4ded-155c-4351-af05-3e2be2bfc03c-env-overrides\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.848170 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.846519    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/60bd4ded-155c-4351-af05-3e2be2bfc03c-ovnkube-config\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.850154 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.850129    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4f78308-a546-426b-8863-898b414f6c29-tmp\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.850294 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.850272    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/60bd4ded-155c-4351-af05-3e2be2bfc03c-ovn-node-metrics-cert\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.852058 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.851783    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b4f78308-a546-426b-8863-898b414f6c29-etc-tuned\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.855989 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:13.855962    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 14:14:13.856115 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:13.855995    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 14:14:13.856115 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:13.856013    2587 projected.go:196] Error preparing data for projected volume kube-api-access-hfwbq for pod openshift-network-diagnostics/network-check-target-h7q7r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:14:13.856115 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:13.856097    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e774454-f78b-40ba-ae21-ed6a11caaea5-kube-api-access-hfwbq podName:4e774454-f78b-40ba-ae21-ed6a11caaea5 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:14.356074348 +0000 UTC m=+2.959699385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hfwbq" (UniqueName: "kubernetes.io/projected/4e774454-f78b-40ba-ae21-ed6a11caaea5-kube-api-access-hfwbq") pod "network-check-target-h7q7r" (UID: "4e774454-f78b-40ba-ae21-ed6a11caaea5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:14:13.858089 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.858049    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkx2f\" (UniqueName: \"kubernetes.io/projected/5a76ffd0-362f-4576-8836-483741c783d0-kube-api-access-pkx2f\") pod \"aws-ebs-csi-driver-node-6xf9t\" (UID: \"5a76ffd0-362f-4576-8836-483741c783d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t"
Sep 16 14:14:13.858533 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.858508    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxch6\" (UniqueName: \"kubernetes.io/projected/60bd4ded-155c-4351-af05-3e2be2bfc03c-kube-api-access-kxch6\") pod \"ovnkube-node-zbsc4\" (UID: \"60bd4ded-155c-4351-af05-3e2be2bfc03c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:13.859160 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.859132    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnhqx\" (UniqueName: \"kubernetes.io/projected/b4f78308-a546-426b-8863-898b414f6c29-kube-api-access-nnhqx\") pod \"tuned-mjxbc\" (UID: \"b4f78308-a546-426b-8863-898b414f6c29\") " pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:13.901827 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.901766    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-217.ec2.internal" event={"ID":"e37d493554911b356e79c3add3a4b130","Type":"ContainerStarted","Data":"baf8049edad7e85f938fa8f582d01e72427fa39ff9226bab98a9ef6eaeb9a116"}
Sep 16 14:14:13.903092 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.903057    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-217.ec2.internal" event={"ID":"a33e9dba3903cb84af35554e9153b83a","Type":"ContainerStarted","Data":"7d09826054c5919f26aa2414dc6057fcd2390cd4fa808363017048135f40e92a"}
Sep 16 14:14:13.946703 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.946664    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/489a66c3-b01a-45d8-8747-4ee83b84958e-agent-certs\") pod \"konnectivity-agent-vk8ld\" (UID: \"489a66c3-b01a-45d8-8747-4ee83b84958e\") " pod="kube-system/konnectivity-agent-vk8ld"
Sep 16 14:14:13.946703 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.946707    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-host-var-lib-cni-multus\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.946969 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.946727    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmmgg\" (UniqueName: \"kubernetes.io/projected/e71666fa-da06-4d9b-bb6f-383354341f68-kube-api-access-tmmgg\") pod \"multus-additional-cni-plugins-vtckf\" (UID: \"e71666fa-da06-4d9b-bb6f-383354341f68\") " pod="openshift-multus/multus-additional-cni-plugins-vtckf"
Sep 16 14:14:13.946969 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.946764    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-host-run-k8s-cni-cncf-io\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.946969 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.946813    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-host-run-k8s-cni-cncf-io\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.946969 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.946814    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-host-var-lib-cni-multus\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.946969 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.946859    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5130c0de-1444-4258-8e1d-7b00b0b042a3-multus-daemon-config\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.946969 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.946892    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d657d718-fd8c-4034-9cd8-232f90fc573e-serviceca\") pod \"node-ca-drp7m\" (UID: \"d657d718-fd8c-4034-9cd8-232f90fc573e\") " pod="openshift-image-registry/node-ca-drp7m"
Sep 16 14:14:13.946969 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.946920    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50d9276f-9ed0-42b9-8264-7c2baf54f7c1-host-slash\") pod \"iptables-alerter-2dl6w\" (UID: \"50d9276f-9ed0-42b9-8264-7c2baf54f7c1\") " pod="openshift-network-operator/iptables-alerter-2dl6w"
Sep 16 14:14:13.946969 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.946947    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e71666fa-da06-4d9b-bb6f-383354341f68-cni-binary-copy\") pod \"multus-additional-cni-plugins-vtckf\" (UID: \"e71666fa-da06-4d9b-bb6f-383354341f68\") " pod="openshift-multus/multus-additional-cni-plugins-vtckf"
Sep 16 14:14:13.947273 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.946976    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f12cf92-2a09-4fe6-a73e-5692728769b3-kubelet-config\") pod \"global-pull-secret-syncer-x59fz\" (UID: \"6f12cf92-2a09-4fe6-a73e-5692728769b3\") " pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:13.947273 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.947001    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-multus-cni-dir\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.947273 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.947023    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-cnibin\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.947273 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.947045    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e71666fa-da06-4d9b-bb6f-383354341f68-cnibin\") pod \"multus-additional-cni-plugins-vtckf\" (UID: \"e71666fa-da06-4d9b-bb6f-383354341f68\") " pod="openshift-multus/multus-additional-cni-plugins-vtckf"
Sep 16 14:14:13.947738 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.947706    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x88p5\" (UniqueName: \"kubernetes.io/projected/d657d718-fd8c-4034-9cd8-232f90fc573e-kube-api-access-x88p5\") pod \"node-ca-drp7m\" (UID: \"d657d718-fd8c-4034-9cd8-232f90fc573e\") " pod="openshift-image-registry/node-ca-drp7m"
Sep 16 14:14:13.947855 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.947766    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4193914e-79e1-4c6a-ba3b-a4d9f5731637-tmp-dir\") pod \"node-resolver-trl7p\" (UID: \"4193914e-79e1-4c6a-ba3b-a4d9f5731637\") " pod="openshift-dns/node-resolver-trl7p"
Sep 16 14:14:13.947855 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.947807    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e10f563c-16f8-4699-a7d4-52e6bfba29a5-metrics-certs\") pod \"network-metrics-daemon-6qpvt\" (UID: \"e10f563c-16f8-4699-a7d4-52e6bfba29a5\") " pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:13.947855 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.947846    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-host-var-lib-cni-bin\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.948007 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.947881    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-multus-conf-dir\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.948007 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.947910    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2mpd\" (UniqueName: \"kubernetes.io/projected/5130c0de-1444-4258-8e1d-7b00b0b042a3-kube-api-access-c2mpd\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.948007 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.947970    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50d9276f-9ed0-42b9-8264-7c2baf54f7c1-host-slash\") pod \"iptables-alerter-2dl6w\" (UID: \"50d9276f-9ed0-42b9-8264-7c2baf54f7c1\") " pod="openshift-network-operator/iptables-alerter-2dl6w"
Sep 16 14:14:13.948149 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.947958    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e71666fa-da06-4d9b-bb6f-383354341f68-system-cni-dir\") pod \"multus-additional-cni-plugins-vtckf\" (UID: \"e71666fa-da06-4d9b-bb6f-383354341f68\") " pod="openshift-multus/multus-additional-cni-plugins-vtckf"
Sep 16 14:14:13.948149 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.948056    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f12cf92-2a09-4fe6-a73e-5692728769b3-original-pull-secret\") pod \"global-pull-secret-syncer-x59fz\" (UID: \"6f12cf92-2a09-4fe6-a73e-5692728769b3\") " pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:13.948149 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.948115    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-multus-socket-dir-parent\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.948295 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.948149    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-host-run-multus-certs\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.948295 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.948185    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4193914e-79e1-4c6a-ba3b-a4d9f5731637-hosts-file\") pod \"node-resolver-trl7p\" (UID: \"4193914e-79e1-4c6a-ba3b-a4d9f5731637\") " pod="openshift-dns/node-resolver-trl7p"
Sep 16 14:14:13.948295 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.948224    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/489a66c3-b01a-45d8-8747-4ee83b84958e-konnectivity-ca\") pod \"konnectivity-agent-vk8ld\" (UID: \"489a66c3-b01a-45d8-8747-4ee83b84958e\") " pod="kube-system/konnectivity-agent-vk8ld"
Sep 16 14:14:13.948295 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.948264    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e71666fa-da06-4d9b-bb6f-383354341f68-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vtckf\" (UID: \"e71666fa-da06-4d9b-bb6f-383354341f68\") " pod="openshift-multus/multus-additional-cni-plugins-vtckf"
Sep 16 14:14:13.948295 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.948281    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5130c0de-1444-4258-8e1d-7b00b0b042a3-multus-daemon-config\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.948295 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.948281    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d657d718-fd8c-4034-9cd8-232f90fc573e-serviceca\") pod \"node-ca-drp7m\" (UID: \"d657d718-fd8c-4034-9cd8-232f90fc573e\") " pod="openshift-image-registry/node-ca-drp7m"
Sep 16 14:14:13.948605 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.948327    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-host-var-lib-kubelet\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.948605 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.948333    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-multus-conf-dir\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.948605 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.948383    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4193914e-79e1-4c6a-ba3b-a4d9f5731637-tmp-dir\") pod \"node-resolver-trl7p\" (UID: \"4193914e-79e1-4c6a-ba3b-a4d9f5731637\") " pod="openshift-dns/node-resolver-trl7p"
Sep 16 14:14:13.948605 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.948402    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2f65\" (UniqueName: \"kubernetes.io/projected/e10f563c-16f8-4699-a7d4-52e6bfba29a5-kube-api-access-d2f65\") pod \"network-metrics-daemon-6qpvt\" (UID: \"e10f563c-16f8-4699-a7d4-52e6bfba29a5\") " pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:13.948605 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:13.948429    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 14:14:13.948605 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.948482    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5130c0de-1444-4258-8e1d-7b00b0b042a3-cni-binary-copy\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.948605 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.948524    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-hostroot\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.948605 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.948555    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-host-run-multus-certs\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.948605 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.948580    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/50d9276f-9ed0-42b9-8264-7c2baf54f7c1-iptables-alerter-script\") pod \"iptables-alerter-2dl6w\" (UID: \"50d9276f-9ed0-42b9-8264-7c2baf54f7c1\") " pod="openshift-network-operator/iptables-alerter-2dl6w"
Sep 16 14:14:13.948605 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:13.948595    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f12cf92-2a09-4fe6-a73e-5692728769b3-original-pull-secret podName:6f12cf92-2a09-4fe6-a73e-5692728769b3 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:14.448574047 +0000 UTC m=+3.052199060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f12cf92-2a09-4fe6-a73e-5692728769b3-original-pull-secret") pod "global-pull-secret-syncer-x59fz" (UID: "6f12cf92-2a09-4fe6-a73e-5692728769b3") : object "kube-system"/"original-pull-secret" not registered
Sep 16 14:14:13.949066 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.948485    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e71666fa-da06-4d9b-bb6f-383354341f68-system-cni-dir\") pod \"multus-additional-cni-plugins-vtckf\" (UID: \"e71666fa-da06-4d9b-bb6f-383354341f68\") " pod="openshift-multus/multus-additional-cni-plugins-vtckf"
Sep 16 14:14:13.949066 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.948631    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4193914e-79e1-4c6a-ba3b-a4d9f5731637-hosts-file\") pod \"node-resolver-trl7p\" (UID: \"4193914e-79e1-4c6a-ba3b-a4d9f5731637\") " pod="openshift-dns/node-resolver-trl7p"
Sep 16 14:14:13.949066 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.948670    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-host-var-lib-kubelet\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.949066 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.948736    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-host-var-lib-cni-bin\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.949259 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.949151    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5130c0de-1444-4258-8e1d-7b00b0b042a3-cni-binary-copy\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.949259 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.949226    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-multus-socket-dir-parent\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.949259 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.949255    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-hostroot\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.949414 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.949290    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-multus-cni-dir\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.949414 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.949282    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/50d9276f-9ed0-42b9-8264-7c2baf54f7c1-iptables-alerter-script\") pod \"iptables-alerter-2dl6w\" (UID: \"50d9276f-9ed0-42b9-8264-7c2baf54f7c1\") " pod="openshift-network-operator/iptables-alerter-2dl6w"
Sep 16 14:14:13.949414 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.949331    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f12cf92-2a09-4fe6-a73e-5692728769b3-kubelet-config\") pod \"global-pull-secret-syncer-x59fz\" (UID: \"6f12cf92-2a09-4fe6-a73e-5692728769b3\") " pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:13.949414 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:13.949388    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:14:13.949622 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.949412    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-cnibin\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.949622 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.949552    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d657d718-fd8c-4034-9cd8-232f90fc573e-host\") pod \"node-ca-drp7m\" (UID: \"d657d718-fd8c-4034-9cd8-232f90fc573e\") " pod="openshift-image-registry/node-ca-drp7m"
Sep 16 14:14:13.949622 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.949593    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-system-cni-dir\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.949807 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.949652    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d657d718-fd8c-4034-9cd8-232f90fc573e-host\") pod \"node-ca-drp7m\" (UID: \"d657d718-fd8c-4034-9cd8-232f90fc573e\") " pod="openshift-image-registry/node-ca-drp7m"
Sep 16 14:14:13.949807 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.949680    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-host-run-netns\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.949807 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.949706    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e71666fa-da06-4d9b-bb6f-383354341f68-cnibin\") pod \"multus-additional-cni-plugins-vtckf\" (UID: \"e71666fa-da06-4d9b-bb6f-383354341f68\") " pod="openshift-multus/multus-additional-cni-plugins-vtckf"
Sep 16 14:14:13.949807 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.949709    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/489a66c3-b01a-45d8-8747-4ee83b84958e-konnectivity-ca\") pod \"konnectivity-agent-vk8ld\" (UID: \"489a66c3-b01a-45d8-8747-4ee83b84958e\") " pod="kube-system/konnectivity-agent-vk8ld"
Sep 16 14:14:13.949807 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.949721    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpsq8\" (UniqueName: \"kubernetes.io/projected/50d9276f-9ed0-42b9-8264-7c2baf54f7c1-kube-api-access-tpsq8\") pod \"iptables-alerter-2dl6w\" (UID: \"50d9276f-9ed0-42b9-8264-7c2baf54f7c1\") " pod="openshift-network-operator/iptables-alerter-2dl6w"
Sep 16 14:14:13.949807 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.949763    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f12cf92-2a09-4fe6-a73e-5692728769b3-dbus\") pod \"global-pull-secret-syncer-x59fz\" (UID: \"6f12cf92-2a09-4fe6-a73e-5692728769b3\") " pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:13.949807 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.949767    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-system-cni-dir\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.950127 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.949982    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f12cf92-2a09-4fe6-a73e-5692728769b3-dbus\") pod \"global-pull-secret-syncer-x59fz\" (UID: \"6f12cf92-2a09-4fe6-a73e-5692728769b3\") " pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:13.950127 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.949995    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e71666fa-da06-4d9b-bb6f-383354341f68-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vtckf\" (UID: \"e71666fa-da06-4d9b-bb6f-383354341f68\") " pod="openshift-multus/multus-additional-cni-plugins-vtckf"
Sep 16 14:14:13.950127 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.950040    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-host-run-netns\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.950127 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.950066    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e71666fa-da06-4d9b-bb6f-383354341f68-cni-binary-copy\") pod \"multus-additional-cni-plugins-vtckf\" (UID: \"e71666fa-da06-4d9b-bb6f-383354341f68\") " pod="openshift-multus/multus-additional-cni-plugins-vtckf"
Sep 16 14:14:13.950127 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.950100    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e71666fa-da06-4d9b-bb6f-383354341f68-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vtckf\" (UID: \"e71666fa-da06-4d9b-bb6f-383354341f68\") " pod="openshift-multus/multus-additional-cni-plugins-vtckf"
Sep 16 14:14:13.950356 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:13.950205    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e10f563c-16f8-4699-a7d4-52e6bfba29a5-metrics-certs podName:e10f563c-16f8-4699-a7d4-52e6bfba29a5 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:14.450159197 +0000 UTC m=+3.053784224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e10f563c-16f8-4699-a7d4-52e6bfba29a5-metrics-certs") pod "network-metrics-daemon-6qpvt" (UID: "e10f563c-16f8-4699-a7d4-52e6bfba29a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:14:13.950356 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.950333    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e71666fa-da06-4d9b-bb6f-383354341f68-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vtckf\" (UID: \"e71666fa-da06-4d9b-bb6f-383354341f68\") " pod="openshift-multus/multus-additional-cni-plugins-vtckf"
Sep 16 14:14:13.950535 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.950296    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-etc-kubernetes\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.950535 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.950456    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e71666fa-da06-4d9b-bb6f-383354341f68-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vtckf\" (UID: \"e71666fa-da06-4d9b-bb6f-383354341f68\") " pod="openshift-multus/multus-additional-cni-plugins-vtckf"
Sep 16 14:14:13.950535 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.950510    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4dc7\" (UniqueName: \"kubernetes.io/projected/4193914e-79e1-4c6a-ba3b-a4d9f5731637-kube-api-access-s4dc7\") pod \"node-resolver-trl7p\" (UID: \"4193914e-79e1-4c6a-ba3b-a4d9f5731637\") " pod="openshift-dns/node-resolver-trl7p"
Sep 16 14:14:13.950871 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.950367    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5130c0de-1444-4258-8e1d-7b00b0b042a3-etc-kubernetes\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.951461 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.951427    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e71666fa-da06-4d9b-bb6f-383354341f68-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vtckf\" (UID: \"e71666fa-da06-4d9b-bb6f-383354341f68\") " pod="openshift-multus/multus-additional-cni-plugins-vtckf"
Sep 16 14:14:13.953219 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.953195    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/489a66c3-b01a-45d8-8747-4ee83b84958e-agent-certs\") pod \"konnectivity-agent-vk8ld\" (UID: \"489a66c3-b01a-45d8-8747-4ee83b84958e\") " pod="kube-system/konnectivity-agent-vk8ld"
Sep 16 14:14:13.958600 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.958569    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmmgg\" (UniqueName: \"kubernetes.io/projected/e71666fa-da06-4d9b-bb6f-383354341f68-kube-api-access-tmmgg\") pod \"multus-additional-cni-plugins-vtckf\" (UID: \"e71666fa-da06-4d9b-bb6f-383354341f68\") " pod="openshift-multus/multus-additional-cni-plugins-vtckf"
Sep 16 14:14:13.959886 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.959821    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x88p5\" (UniqueName: \"kubernetes.io/projected/d657d718-fd8c-4034-9cd8-232f90fc573e-kube-api-access-x88p5\") pod \"node-ca-drp7m\" (UID: \"d657d718-fd8c-4034-9cd8-232f90fc573e\") " pod="openshift-image-registry/node-ca-drp7m"
Sep 16 14:14:13.961531 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.961499    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpsq8\" (UniqueName: \"kubernetes.io/projected/50d9276f-9ed0-42b9-8264-7c2baf54f7c1-kube-api-access-tpsq8\") pod \"iptables-alerter-2dl6w\" (UID: \"50d9276f-9ed0-42b9-8264-7c2baf54f7c1\") " pod="openshift-network-operator/iptables-alerter-2dl6w"
Sep 16 14:14:13.961694 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.961670    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4dc7\" (UniqueName: \"kubernetes.io/projected/4193914e-79e1-4c6a-ba3b-a4d9f5731637-kube-api-access-s4dc7\") pod \"node-resolver-trl7p\" (UID: \"4193914e-79e1-4c6a-ba3b-a4d9f5731637\") " pod="openshift-dns/node-resolver-trl7p"
Sep 16 14:14:13.962290 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.962267    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2mpd\" (UniqueName: \"kubernetes.io/projected/5130c0de-1444-4258-8e1d-7b00b0b042a3-kube-api-access-c2mpd\") pod \"multus-4d7hc\" (UID: \"5130c0de-1444-4258-8e1d-7b00b0b042a3\") " pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:13.962494 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:13.962452    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2f65\" (UniqueName: \"kubernetes.io/projected/e10f563c-16f8-4699-a7d4-52e6bfba29a5-kube-api-access-d2f65\") pod \"network-metrics-daemon-6qpvt\" (UID: \"e10f563c-16f8-4699-a7d4-52e6bfba29a5\") " pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:14.039608 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:14.039569    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:14.045623 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:14.045598    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t"
Sep 16 14:14:14.054077 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:14.054044    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60bd4ded_155c_4351_af05_3e2be2bfc03c.slice/crio-21721e27aae8cc310ea13031eb0f1f6122fac028724c5c25c83b62377f9cfe48 WatchSource:0}: Error finding container 21721e27aae8cc310ea13031eb0f1f6122fac028724c5c25c83b62377f9cfe48: Status 404 returned error can't find the container with id 21721e27aae8cc310ea13031eb0f1f6122fac028724c5c25c83b62377f9cfe48
Sep 16 14:14:14.054332 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:14.054292    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mjxbc"
Sep 16 14:14:14.057794 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:14.057770    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a76ffd0_362f_4576_8836_483741c783d0.slice/crio-48577ec5931531dd81574c66f3b0138f34ca4b07c9d8e21e5e78d0f378ff2053 WatchSource:0}: Error finding container 48577ec5931531dd81574c66f3b0138f34ca4b07c9d8e21e5e78d0f378ff2053: Status 404 returned error can't find the container with id 48577ec5931531dd81574c66f3b0138f34ca4b07c9d8e21e5e78d0f378ff2053
Sep 16 14:14:14.061787 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:14.061352    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-trl7p"
Sep 16 14:14:14.065821 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:14.065796    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f78308_a546_426b_8863_898b414f6c29.slice/crio-ee77e3717ec8c7deb28f2ffc37e685034e38a23d9d8ddb0a8c2d27a0f410e2e6 WatchSource:0}: Error finding container ee77e3717ec8c7deb28f2ffc37e685034e38a23d9d8ddb0a8c2d27a0f410e2e6: Status 404 returned error can't find the container with id ee77e3717ec8c7deb28f2ffc37e685034e38a23d9d8ddb0a8c2d27a0f410e2e6
Sep 16 14:14:14.070113 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:14.070091    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vtckf"
Sep 16 14:14:14.074700 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:14.074671    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4193914e_79e1_4c6a_ba3b_a4d9f5731637.slice/crio-c10c088e2733acc03d4ed0a50db3d1d22fd5e7d1daf1fdc4c37438d06c0024de WatchSource:0}: Error finding container c10c088e2733acc03d4ed0a50db3d1d22fd5e7d1daf1fdc4c37438d06c0024de: Status 404 returned error can't find the container with id c10c088e2733acc03d4ed0a50db3d1d22fd5e7d1daf1fdc4c37438d06c0024de
Sep 16 14:14:14.077082 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:14.077059    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2dl6w"
Sep 16 14:14:14.080942 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:14.080903    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71666fa_da06_4d9b_bb6f_383354341f68.slice/crio-f5f72beff801f9bf1a980882cd40ea8f4c2aa06ac964ae0d6ffdf7223baa7261 WatchSource:0}: Error finding container f5f72beff801f9bf1a980882cd40ea8f4c2aa06ac964ae0d6ffdf7223baa7261: Status 404 returned error can't find the container with id f5f72beff801f9bf1a980882cd40ea8f4c2aa06ac964ae0d6ffdf7223baa7261
Sep 16 14:14:14.086023 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:14.085998    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vk8ld"
Sep 16 14:14:14.088255 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:14.088230    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d9276f_9ed0_42b9_8264_7c2baf54f7c1.slice/crio-40e48e19129f2c4a5912ba06108e6d7d7daa107c56296077dfdb4930ab5d7692 WatchSource:0}: Error finding container 40e48e19129f2c4a5912ba06108e6d7d7daa107c56296077dfdb4930ab5d7692: Status 404 returned error can't find the container with id 40e48e19129f2c4a5912ba06108e6d7d7daa107c56296077dfdb4930ab5d7692
Sep 16 14:14:14.095174 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:14.095149    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4d7hc"
Sep 16 14:14:14.097209 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:14.097175    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod489a66c3_b01a_45d8_8747_4ee83b84958e.slice/crio-adcdda695b5d4162804d6a1f43654e542374a91666371fd930d159fff8d218cd WatchSource:0}: Error finding container adcdda695b5d4162804d6a1f43654e542374a91666371fd930d159fff8d218cd: Status 404 returned error can't find the container with id adcdda695b5d4162804d6a1f43654e542374a91666371fd930d159fff8d218cd
Sep 16 14:14:14.100994 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:14.100968    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-drp7m"
Sep 16 14:14:14.106882 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:14.106851    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5130c0de_1444_4258_8e1d_7b00b0b042a3.slice/crio-3bc41a32ade73e43596a1c6a635b7602da1b39e63e3c2688cc3162d249cc01a3 WatchSource:0}: Error finding container 3bc41a32ade73e43596a1c6a635b7602da1b39e63e3c2688cc3162d249cc01a3: Status 404 returned error can't find the container with id 3bc41a32ade73e43596a1c6a635b7602da1b39e63e3c2688cc3162d249cc01a3
Sep 16 14:14:14.114645 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:14.114614    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd657d718_fd8c_4034_9cd8_232f90fc573e.slice/crio-2cabecef1ff29b10eba283c4fd482f110e4f822406f58e4871fcc5e1efaf45ad WatchSource:0}: Error finding container 2cabecef1ff29b10eba283c4fd482f110e4f822406f58e4871fcc5e1efaf45ad: Status 404 returned error can't find the container with id 2cabecef1ff29b10eba283c4fd482f110e4f822406f58e4871fcc5e1efaf45ad
Sep 16 14:14:14.222039 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:14.221955    2587 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 14:14:14.455409 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:14.455368    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e10f563c-16f8-4699-a7d4-52e6bfba29a5-metrics-certs\") pod \"network-metrics-daemon-6qpvt\" (UID: \"e10f563c-16f8-4699-a7d4-52e6bfba29a5\") " pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:14.455638 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:14.455453    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f12cf92-2a09-4fe6-a73e-5692728769b3-original-pull-secret\") pod \"global-pull-secret-syncer-x59fz\" (UID: \"6f12cf92-2a09-4fe6-a73e-5692728769b3\") " pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:14.455638 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:14.455495    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfwbq\" (UniqueName: \"kubernetes.io/projected/4e774454-f78b-40ba-ae21-ed6a11caaea5-kube-api-access-hfwbq\") pod \"network-check-target-h7q7r\" (UID: \"4e774454-f78b-40ba-ae21-ed6a11caaea5\") " pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:14:14.455638 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:14.455549    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:14:14.455638 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:14.455632    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 14:14:14.455864 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:14.455643    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e10f563c-16f8-4699-a7d4-52e6bfba29a5-metrics-certs podName:e10f563c-16f8-4699-a7d4-52e6bfba29a5 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:15.455626915 +0000 UTC m=+4.059251930 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e10f563c-16f8-4699-a7d4-52e6bfba29a5-metrics-certs") pod "network-metrics-daemon-6qpvt" (UID: "e10f563c-16f8-4699-a7d4-52e6bfba29a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:14:14.455864 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:14.455649    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 14:14:14.455864 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:14.455664    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 14:14:14.455864 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:14.455676    2587 projected.go:196] Error preparing data for projected volume kube-api-access-hfwbq for pod openshift-network-diagnostics/network-check-target-h7q7r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:14:14.455864 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:14.455690    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f12cf92-2a09-4fe6-a73e-5692728769b3-original-pull-secret podName:6f12cf92-2a09-4fe6-a73e-5692728769b3 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:15.455672073 +0000 UTC m=+4.059297102 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f12cf92-2a09-4fe6-a73e-5692728769b3-original-pull-secret") pod "global-pull-secret-syncer-x59fz" (UID: "6f12cf92-2a09-4fe6-a73e-5692728769b3") : object "kube-system"/"original-pull-secret" not registered
Sep 16 14:14:14.455864 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:14.455722    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e774454-f78b-40ba-ae21-ed6a11caaea5-kube-api-access-hfwbq podName:4e774454-f78b-40ba-ae21-ed6a11caaea5 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:15.455708183 +0000 UTC m=+4.059333212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hfwbq" (UniqueName: "kubernetes.io/projected/4e774454-f78b-40ba-ae21-ed6a11caaea5-kube-api-access-hfwbq") pod "network-check-target-h7q7r" (UID: "4e774454-f78b-40ba-ae21-ed6a11caaea5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:14:14.780075 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:14.780002    2587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 14:09:13 +0000 UTC" deadline="2028-05-10 00:20:31.439287073 +0000 UTC"
Sep 16 14:14:14.780075 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:14.780073    2587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14434h6m16.659218536s"
Sep 16 14:14:14.909658 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:14.909410    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4d7hc" event={"ID":"5130c0de-1444-4258-8e1d-7b00b0b042a3","Type":"ContainerStarted","Data":"3bc41a32ade73e43596a1c6a635b7602da1b39e63e3c2688cc3162d249cc01a3"}
Sep 16 14:14:14.911523 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:14.911490    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vk8ld" event={"ID":"489a66c3-b01a-45d8-8747-4ee83b84958e","Type":"ContainerStarted","Data":"adcdda695b5d4162804d6a1f43654e542374a91666371fd930d159fff8d218cd"}
Sep 16 14:14:14.918238 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:14.918195    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2dl6w" event={"ID":"50d9276f-9ed0-42b9-8264-7c2baf54f7c1","Type":"ContainerStarted","Data":"40e48e19129f2c4a5912ba06108e6d7d7daa107c56296077dfdb4930ab5d7692"}
Sep 16 14:14:14.921548 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:14.921414    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4" event={"ID":"60bd4ded-155c-4351-af05-3e2be2bfc03c","Type":"ContainerStarted","Data":"21721e27aae8cc310ea13031eb0f1f6122fac028724c5c25c83b62377f9cfe48"}
Sep 16 14:14:14.923629 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:14.923540    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-drp7m" event={"ID":"d657d718-fd8c-4034-9cd8-232f90fc573e","Type":"ContainerStarted","Data":"2cabecef1ff29b10eba283c4fd482f110e4f822406f58e4871fcc5e1efaf45ad"}
Sep 16 14:14:14.929722 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:14.929680    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vtckf" event={"ID":"e71666fa-da06-4d9b-bb6f-383354341f68","Type":"ContainerStarted","Data":"f5f72beff801f9bf1a980882cd40ea8f4c2aa06ac964ae0d6ffdf7223baa7261"}
Sep 16 14:14:14.935563 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:14.935524    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-trl7p" event={"ID":"4193914e-79e1-4c6a-ba3b-a4d9f5731637","Type":"ContainerStarted","Data":"c10c088e2733acc03d4ed0a50db3d1d22fd5e7d1daf1fdc4c37438d06c0024de"}
Sep 16 14:14:14.939997 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:14.939960    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mjxbc" event={"ID":"b4f78308-a546-426b-8863-898b414f6c29","Type":"ContainerStarted","Data":"ee77e3717ec8c7deb28f2ffc37e685034e38a23d9d8ddb0a8c2d27a0f410e2e6"}
Sep 16 14:14:14.944794 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:14.944756    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t" event={"ID":"5a76ffd0-362f-4576-8836-483741c783d0","Type":"ContainerStarted","Data":"48577ec5931531dd81574c66f3b0138f34ca4b07c9d8e21e5e78d0f378ff2053"}
Sep 16 14:14:15.465454 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:15.465267    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f12cf92-2a09-4fe6-a73e-5692728769b3-original-pull-secret\") pod \"global-pull-secret-syncer-x59fz\" (UID: \"6f12cf92-2a09-4fe6-a73e-5692728769b3\") " pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:15.465454 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:15.465325    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfwbq\" (UniqueName: \"kubernetes.io/projected/4e774454-f78b-40ba-ae21-ed6a11caaea5-kube-api-access-hfwbq\") pod \"network-check-target-h7q7r\" (UID: \"4e774454-f78b-40ba-ae21-ed6a11caaea5\") " pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:14:15.465454 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:15.465385    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e10f563c-16f8-4699-a7d4-52e6bfba29a5-metrics-certs\") pod \"network-metrics-daemon-6qpvt\" (UID: \"e10f563c-16f8-4699-a7d4-52e6bfba29a5\") " pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:15.465680 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:15.465544    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:14:15.465680 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:15.465609    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e10f563c-16f8-4699-a7d4-52e6bfba29a5-metrics-certs podName:e10f563c-16f8-4699-a7d4-52e6bfba29a5 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:17.465589281 +0000 UTC m=+6.069214305 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e10f563c-16f8-4699-a7d4-52e6bfba29a5-metrics-certs") pod "network-metrics-daemon-6qpvt" (UID: "e10f563c-16f8-4699-a7d4-52e6bfba29a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:14:15.466409 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:15.466014    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 14:14:15.466409 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:15.466068    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f12cf92-2a09-4fe6-a73e-5692728769b3-original-pull-secret podName:6f12cf92-2a09-4fe6-a73e-5692728769b3 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:17.46605214 +0000 UTC m=+6.069677154 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f12cf92-2a09-4fe6-a73e-5692728769b3-original-pull-secret") pod "global-pull-secret-syncer-x59fz" (UID: "6f12cf92-2a09-4fe6-a73e-5692728769b3") : object "kube-system"/"original-pull-secret" not registered
Sep 16 14:14:15.466409 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:15.466145    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 14:14:15.466409 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:15.466158    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 14:14:15.466409 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:15.466172    2587 projected.go:196] Error preparing data for projected volume kube-api-access-hfwbq for pod openshift-network-diagnostics/network-check-target-h7q7r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:14:15.466409 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:15.466206    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e774454-f78b-40ba-ae21-ed6a11caaea5-kube-api-access-hfwbq podName:4e774454-f78b-40ba-ae21-ed6a11caaea5 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:17.466195055 +0000 UTC m=+6.069820079 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hfwbq" (UniqueName: "kubernetes.io/projected/4e774454-f78b-40ba-ae21-ed6a11caaea5-kube-api-access-hfwbq") pod "network-check-target-h7q7r" (UID: "4e774454-f78b-40ba-ae21-ed6a11caaea5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:14:15.780482 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:15.780358    2587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 14:09:13 +0000 UTC" deadline="2028-02-26 18:20:29.850914435 +0000 UTC"
Sep 16 14:14:15.780482 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:15.780438    2587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12676h6m14.070480657s"
Sep 16 14:14:15.899516 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:15.899285    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:15.900609 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:15.899770    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qpvt" podUID="e10f563c-16f8-4699-a7d4-52e6bfba29a5"
Sep 16 14:14:15.900609 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:15.900311    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:14:15.900609 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:15.900430    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7q7r" podUID="4e774454-f78b-40ba-ae21-ed6a11caaea5"
Sep 16 14:14:15.900609 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:15.900491    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:15.900609 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:15.900560    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x59fz" podUID="6f12cf92-2a09-4fe6-a73e-5692728769b3"
Sep 16 14:14:15.968438 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:15.967859    2587 generic.go:356] "Generic (PLEG): container finished" podID="a33e9dba3903cb84af35554e9153b83a" containerID="89673a08c000620fb542b34436d1353540c76d3cad0b369a39550677a35782d8" exitCode=0
Sep 16 14:14:15.968438 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:15.967925    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-217.ec2.internal" event={"ID":"a33e9dba3903cb84af35554e9153b83a","Type":"ContainerDied","Data":"89673a08c000620fb542b34436d1353540c76d3cad0b369a39550677a35782d8"}
Sep 16 14:14:16.971983 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:16.971928    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-217.ec2.internal" event={"ID":"a33e9dba3903cb84af35554e9153b83a","Type":"ContainerStarted","Data":"83c69a512ad1f8b663116da6f935a704528a0cdc226f0a5700bea489e07b62e9"}
Sep 16 14:14:16.992302 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:16.992237    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-217.ec2.internal" podStartSLOduration=3.992212883 podStartE2EDuration="3.992212883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:16.991793962 +0000 UTC m=+5.595418997" watchObservedRunningTime="2026-09-16 14:14:16.992212883 +0000 UTC m=+5.595837916"
Sep 16 14:14:17.483245 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:17.483198    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e10f563c-16f8-4699-a7d4-52e6bfba29a5-metrics-certs\") pod \"network-metrics-daemon-6qpvt\" (UID: \"e10f563c-16f8-4699-a7d4-52e6bfba29a5\") " pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:17.483472 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:17.483256    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f12cf92-2a09-4fe6-a73e-5692728769b3-original-pull-secret\") pod \"global-pull-secret-syncer-x59fz\" (UID: \"6f12cf92-2a09-4fe6-a73e-5692728769b3\") " pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:17.483472 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:17.483289    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfwbq\" (UniqueName: \"kubernetes.io/projected/4e774454-f78b-40ba-ae21-ed6a11caaea5-kube-api-access-hfwbq\") pod \"network-check-target-h7q7r\" (UID: \"4e774454-f78b-40ba-ae21-ed6a11caaea5\") " pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:14:17.483472 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:17.483436    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 14:14:17.483472 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:17.483460    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 14:14:17.483472 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:17.483473    2587 projected.go:196] Error preparing data for projected volume kube-api-access-hfwbq for pod openshift-network-diagnostics/network-check-target-h7q7r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:14:17.483715 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:17.483533    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e774454-f78b-40ba-ae21-ed6a11caaea5-kube-api-access-hfwbq podName:4e774454-f78b-40ba-ae21-ed6a11caaea5 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:21.483514398 +0000 UTC m=+10.087139422 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hfwbq" (UniqueName: "kubernetes.io/projected/4e774454-f78b-40ba-ae21-ed6a11caaea5-kube-api-access-hfwbq") pod "network-check-target-h7q7r" (UID: "4e774454-f78b-40ba-ae21-ed6a11caaea5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:14:17.483915 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:17.483896    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:14:17.483974 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:17.483944    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e10f563c-16f8-4699-a7d4-52e6bfba29a5-metrics-certs podName:e10f563c-16f8-4699-a7d4-52e6bfba29a5 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:21.483933507 +0000 UTC m=+10.087558516 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e10f563c-16f8-4699-a7d4-52e6bfba29a5-metrics-certs") pod "network-metrics-daemon-6qpvt" (UID: "e10f563c-16f8-4699-a7d4-52e6bfba29a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:14:17.484027 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:17.483986    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 14:14:17.484027 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:17.484021    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f12cf92-2a09-4fe6-a73e-5692728769b3-original-pull-secret podName:6f12cf92-2a09-4fe6-a73e-5692728769b3 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:21.48400962 +0000 UTC m=+10.087634632 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f12cf92-2a09-4fe6-a73e-5692728769b3-original-pull-secret") pod "global-pull-secret-syncer-x59fz" (UID: "6f12cf92-2a09-4fe6-a73e-5692728769b3") : object "kube-system"/"original-pull-secret" not registered
Sep 16 14:14:17.897287 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:17.896725    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:14:17.897287 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:17.896759    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:17.897287 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:17.896900    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7q7r" podUID="4e774454-f78b-40ba-ae21-ed6a11caaea5"
Sep 16 14:14:17.897287 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:17.897046    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:17.897287 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:17.897172    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x59fz" podUID="6f12cf92-2a09-4fe6-a73e-5692728769b3"
Sep 16 14:14:17.897287 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:17.897256    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qpvt" podUID="e10f563c-16f8-4699-a7d4-52e6bfba29a5"
Sep 16 14:14:19.896939 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:19.896912    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:19.897417 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:19.897018    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qpvt" podUID="e10f563c-16f8-4699-a7d4-52e6bfba29a5"
Sep 16 14:14:19.897417 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:19.897127    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:19.897417 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:19.897215    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x59fz" podUID="6f12cf92-2a09-4fe6-a73e-5692728769b3"
Sep 16 14:14:19.897580 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:19.897454    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:14:19.897580 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:19.897534    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7q7r" podUID="4e774454-f78b-40ba-ae21-ed6a11caaea5"
Sep 16 14:14:21.520682 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:21.520486    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e10f563c-16f8-4699-a7d4-52e6bfba29a5-metrics-certs\") pod \"network-metrics-daemon-6qpvt\" (UID: \"e10f563c-16f8-4699-a7d4-52e6bfba29a5\") " pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:21.521129 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:21.520705    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f12cf92-2a09-4fe6-a73e-5692728769b3-original-pull-secret\") pod \"global-pull-secret-syncer-x59fz\" (UID: \"6f12cf92-2a09-4fe6-a73e-5692728769b3\") " pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:21.521129 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:21.520644    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:14:21.521129 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:21.520741    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfwbq\" (UniqueName: \"kubernetes.io/projected/4e774454-f78b-40ba-ae21-ed6a11caaea5-kube-api-access-hfwbq\") pod \"network-check-target-h7q7r\" (UID: \"4e774454-f78b-40ba-ae21-ed6a11caaea5\") " pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:14:21.521129 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:21.520807    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e10f563c-16f8-4699-a7d4-52e6bfba29a5-metrics-certs podName:e10f563c-16f8-4699-a7d4-52e6bfba29a5 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:29.520779792 +0000 UTC m=+18.124404824 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e10f563c-16f8-4699-a7d4-52e6bfba29a5-metrics-certs") pod "network-metrics-daemon-6qpvt" (UID: "e10f563c-16f8-4699-a7d4-52e6bfba29a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:14:21.521129 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:21.520870    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 14:14:21.521129 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:21.520886    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 14:14:21.521129 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:21.520902    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 14:14:21.521129 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:21.520914    2587 projected.go:196] Error preparing data for projected volume kube-api-access-hfwbq for pod openshift-network-diagnostics/network-check-target-h7q7r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:14:21.521129 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:21.520925    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f12cf92-2a09-4fe6-a73e-5692728769b3-original-pull-secret podName:6f12cf92-2a09-4fe6-a73e-5692728769b3 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:29.520908906 +0000 UTC m=+18.124533932 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f12cf92-2a09-4fe6-a73e-5692728769b3-original-pull-secret") pod "global-pull-secret-syncer-x59fz" (UID: "6f12cf92-2a09-4fe6-a73e-5692728769b3") : object "kube-system"/"original-pull-secret" not registered
Sep 16 14:14:21.521129 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:21.520947    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e774454-f78b-40ba-ae21-ed6a11caaea5-kube-api-access-hfwbq podName:4e774454-f78b-40ba-ae21-ed6a11caaea5 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:29.520936149 +0000 UTC m=+18.124561160 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hfwbq" (UniqueName: "kubernetes.io/projected/4e774454-f78b-40ba-ae21-ed6a11caaea5-kube-api-access-hfwbq") pod "network-check-target-h7q7r" (UID: "4e774454-f78b-40ba-ae21-ed6a11caaea5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:14:21.897689 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:21.897608    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:21.897852 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:21.897741    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qpvt" podUID="e10f563c-16f8-4699-a7d4-52e6bfba29a5"
Sep 16 14:14:21.898124 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:21.898107    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:21.898239 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:21.898208    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x59fz" podUID="6f12cf92-2a09-4fe6-a73e-5692728769b3"
Sep 16 14:14:21.898337 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:21.898263    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:14:21.898337 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:21.898329    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7q7r" podUID="4e774454-f78b-40ba-ae21-ed6a11caaea5"
Sep 16 14:14:23.897045 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:23.897022    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:23.897461 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:23.897119    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qpvt" podUID="e10f563c-16f8-4699-a7d4-52e6bfba29a5"
Sep 16 14:14:23.897461 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:23.897166    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:23.897461 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:23.897210    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x59fz" podUID="6f12cf92-2a09-4fe6-a73e-5692728769b3"
Sep 16 14:14:23.897937 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:23.897920    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:14:23.898037 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:23.898017    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7q7r" podUID="4e774454-f78b-40ba-ae21-ed6a11caaea5"
Sep 16 14:14:23.988071 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:23.988030    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t" event={"ID":"5a76ffd0-362f-4576-8836-483741c783d0","Type":"ContainerStarted","Data":"2e06f728c8ff5f85376ff6fb5c6ed756239a39496dd3fe427cbf770f0888ac37"}
Sep 16 14:14:23.989997 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:23.989965    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-217.ec2.internal" event={"ID":"e37d493554911b356e79c3add3a4b130","Type":"ContainerStarted","Data":"03b684597163cc1916e7bebbc5e3aadbbfcba50e09d3225a9abd8eddd93672fb"}
Sep 16 14:14:23.992284 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:23.992251    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vk8ld" event={"ID":"489a66c3-b01a-45d8-8747-4ee83b84958e","Type":"ContainerStarted","Data":"cabfd414cf78f95cde802996ec14e4bcd8c3c2e75fe3249c162d8651e324d752"}
Sep 16 14:14:23.994360 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:23.994335    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-drp7m" event={"ID":"d657d718-fd8c-4034-9cd8-232f90fc573e","Type":"ContainerStarted","Data":"a1d752a5a0b946713daa3afa3fd7a3f7a61e23eac55c1479b9c4ae52bce3a0f1"}
Sep 16 14:14:23.996336 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:23.996310    2587 generic.go:356] "Generic (PLEG): container finished" podID="e71666fa-da06-4d9b-bb6f-383354341f68" containerID="a1c740e61cdac2b100b8082aad588214a0de57b806aefa5c32c0a053e0ec1271" exitCode=0
Sep 16 14:14:23.996452 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:23.996382    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vtckf" event={"ID":"e71666fa-da06-4d9b-bb6f-383354341f68","Type":"ContainerDied","Data":"a1c740e61cdac2b100b8082aad588214a0de57b806aefa5c32c0a053e0ec1271"}
Sep 16 14:14:23.998983 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:23.998959    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-trl7p" event={"ID":"4193914e-79e1-4c6a-ba3b-a4d9f5731637","Type":"ContainerStarted","Data":"28bf91eface302884cd4fb7f1f3725717b1656306744f72d470a8c4539b12e06"}
Sep 16 14:14:24.002406 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:24.002361    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mjxbc" event={"ID":"b4f78308-a546-426b-8863-898b414f6c29","Type":"ContainerStarted","Data":"17715ae372cf777383ee4dfaa343bac4c8ca71c0f6ecd7895cd7dc9bd92b394b"}
Sep 16 14:14:24.030188 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:24.030120    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mjxbc" podStartSLOduration=2.557861653 podStartE2EDuration="12.030098561s" totalImagesPullingTime="9.472236908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:24.02979976 +0000 UTC m=+12.633424793" watchObservedRunningTime="2026-09-16 14:14:24.030098561 +0000 UTC m=+12.633723595"
Sep 16 14:14:24.030763 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:24.030722    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-217.ec2.internal" podStartSLOduration=11.030711122 podStartE2EDuration="11.030711122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:24.011020132 +0000 UTC m=+12.614645166" watchObservedRunningTime="2026-09-16 14:14:24.030711122 +0000 UTC m=+12.634336158"
Sep 16 14:14:24.068732 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:24.068617    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vk8ld" podStartSLOduration=2.627683146 podStartE2EDuration="12.068592984s" totalImagesPullingTime="9.440909838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:24.049776701 +0000 UTC m=+12.653401746" watchObservedRunningTime="2026-09-16 14:14:24.068592984 +0000 UTC m=+12.672218017"
Sep 16 14:14:24.068893 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:24.068789    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-drp7m" podStartSLOduration=2.5643961170000003 podStartE2EDuration="12.068779337s" totalImagesPullingTime="9.50438322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:24.068259242 +0000 UTC m=+12.671884274" watchObservedRunningTime="2026-09-16 14:14:24.068779337 +0000 UTC m=+12.672404369"
Sep 16 14:14:24.121305 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:24.121247    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-trl7p" podStartSLOduration=2.657342731 podStartE2EDuration="12.121226094s" totalImagesPullingTime="9.463883363s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:24.095842386 +0000 UTC m=+12.699467419" watchObservedRunningTime="2026-09-16 14:14:24.121226094 +0000 UTC m=+12.724851126"
Sep 16 14:14:25.898405 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:25.898352    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:25.898851 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:25.898507    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:25.898851 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:25.898517    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:14:25.898851 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:25.898496    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x59fz" podUID="6f12cf92-2a09-4fe6-a73e-5692728769b3"
Sep 16 14:14:25.898851 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:25.898593    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qpvt" podUID="e10f563c-16f8-4699-a7d4-52e6bfba29a5"
Sep 16 14:14:25.898851 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:25.898720    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7q7r" podUID="4e774454-f78b-40ba-ae21-ed6a11caaea5"
Sep 16 14:14:26.009208 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:26.009135    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2dl6w" event={"ID":"50d9276f-9ed0-42b9-8264-7c2baf54f7c1","Type":"ContainerStarted","Data":"7692f6845bfa4c7dd7dfadf6457799c26f1ecd4ca3a1594ffcefac54c6fe34e7"}
Sep 16 14:14:26.025566 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:26.025501    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2dl6w" podStartSLOduration=4.574539923 podStartE2EDuration="14.025478753s" totalImagesPullingTime="9.45093883s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:26.025165083 +0000 UTC m=+14.628790116" watchObservedRunningTime="2026-09-16 14:14:26.025478753 +0000 UTC m=+14.629103786"
Sep 16 14:14:27.896799 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:27.896761    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:27.896799 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:27.896796    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:14:27.897319 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:27.896762    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:27.897319 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:27.896920    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qpvt" podUID="e10f563c-16f8-4699-a7d4-52e6bfba29a5"
Sep 16 14:14:27.897319 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:27.897020    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x59fz" podUID="6f12cf92-2a09-4fe6-a73e-5692728769b3"
Sep 16 14:14:27.897319 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:27.897087    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7q7r" podUID="4e774454-f78b-40ba-ae21-ed6a11caaea5"
Sep 16 14:14:28.988282 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:28.988244    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vk8ld"
Sep 16 14:14:28.989067 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:28.989051    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vk8ld"
Sep 16 14:14:29.015639 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:29.015608    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vk8ld"
Sep 16 14:14:29.016188 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:29.016165    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vk8ld"
Sep 16 14:14:29.586855 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:29.586650    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e10f563c-16f8-4699-a7d4-52e6bfba29a5-metrics-certs\") pod \"network-metrics-daemon-6qpvt\" (UID: \"e10f563c-16f8-4699-a7d4-52e6bfba29a5\") " pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:29.586855 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:29.586856    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:14:29.587105 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:29.586879    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f12cf92-2a09-4fe6-a73e-5692728769b3-original-pull-secret\") pod \"global-pull-secret-syncer-x59fz\" (UID: \"6f12cf92-2a09-4fe6-a73e-5692728769b3\") " pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:29.587105 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:29.586915    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfwbq\" (UniqueName: \"kubernetes.io/projected/4e774454-f78b-40ba-ae21-ed6a11caaea5-kube-api-access-hfwbq\") pod \"network-check-target-h7q7r\" (UID: \"4e774454-f78b-40ba-ae21-ed6a11caaea5\") " pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:14:29.587105 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:29.586989    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e10f563c-16f8-4699-a7d4-52e6bfba29a5-metrics-certs podName:e10f563c-16f8-4699-a7d4-52e6bfba29a5 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:45.586969288 +0000 UTC m=+34.190594312 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e10f563c-16f8-4699-a7d4-52e6bfba29a5-metrics-certs") pod "network-metrics-daemon-6qpvt" (UID: "e10f563c-16f8-4699-a7d4-52e6bfba29a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:14:29.587105 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:29.587049    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 14:14:29.587105 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:29.587082    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 14:14:29.587105 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:29.587097    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 14:14:29.587105 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:29.587107    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f12cf92-2a09-4fe6-a73e-5692728769b3-original-pull-secret podName:6f12cf92-2a09-4fe6-a73e-5692728769b3 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:45.58709015 +0000 UTC m=+34.190715178 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f12cf92-2a09-4fe6-a73e-5692728769b3-original-pull-secret") pod "global-pull-secret-syncer-x59fz" (UID: "6f12cf92-2a09-4fe6-a73e-5692728769b3") : object "kube-system"/"original-pull-secret" not registered
Sep 16 14:14:29.587105 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:29.587110    2587 projected.go:196] Error preparing data for projected volume kube-api-access-hfwbq for pod openshift-network-diagnostics/network-check-target-h7q7r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:14:29.587477 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:29.587162    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e774454-f78b-40ba-ae21-ed6a11caaea5-kube-api-access-hfwbq podName:4e774454-f78b-40ba-ae21-ed6a11caaea5 nodeName:}" failed. No retries permitted until 2026-09-16 14:14:45.58714872 +0000 UTC m=+34.190773744 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hfwbq" (UniqueName: "kubernetes.io/projected/4e774454-f78b-40ba-ae21-ed6a11caaea5-kube-api-access-hfwbq") pod "network-check-target-h7q7r" (UID: "4e774454-f78b-40ba-ae21-ed6a11caaea5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:14:29.899072 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:29.898999    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:29.899225 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:29.899130    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x59fz" podUID="6f12cf92-2a09-4fe6-a73e-5692728769b3"
Sep 16 14:14:29.899225 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:29.899197    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:14:29.899359 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:29.899261    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7q7r" podUID="4e774454-f78b-40ba-ae21-ed6a11caaea5"
Sep 16 14:14:29.899359 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:29.899313    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:29.899484 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:29.899384    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qpvt" podUID="e10f563c-16f8-4699-a7d4-52e6bfba29a5"
Sep 16 14:14:31.897806 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:31.897768    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:31.898270 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:31.897861    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x59fz" podUID="6f12cf92-2a09-4fe6-a73e-5692728769b3"
Sep 16 14:14:31.898270 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:31.897938    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:14:31.898270 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:31.898030    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7q7r" podUID="4e774454-f78b-40ba-ae21-ed6a11caaea5"
Sep 16 14:14:31.898270 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:31.898079    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:31.898270 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:31.898148    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qpvt" podUID="e10f563c-16f8-4699-a7d4-52e6bfba29a5"
Sep 16 14:14:33.896988 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:33.896960    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:33.897493 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:33.896960    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:33.897493 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:33.897096    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:14:33.897493 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:33.897091    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qpvt" podUID="e10f563c-16f8-4699-a7d4-52e6bfba29a5"
Sep 16 14:14:33.897493 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:33.897179    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x59fz" podUID="6f12cf92-2a09-4fe6-a73e-5692728769b3"
Sep 16 14:14:33.897493 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:33.897258    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7q7r" podUID="4e774454-f78b-40ba-ae21-ed6a11caaea5"
Sep 16 14:14:35.176995 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:35.176967    2587 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 14:14:35.805587 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:35.805322    2587 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T14:14:35.177531736Z","UUID":"c54b8022-d3aa-4b19-89eb-877f0b6bd47a","Handler":null,"Name":"","Endpoint":""}
Sep 16 14:14:35.806871 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:35.806850    2587 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 14:14:35.806871 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:35.806880    2587 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 14:14:35.896615 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:35.896576    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:35.896806 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:35.896586    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:35.896806 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:35.896694    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qpvt" podUID="e10f563c-16f8-4699-a7d4-52e6bfba29a5"
Sep 16 14:14:35.896806 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:35.896786    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x59fz" podUID="6f12cf92-2a09-4fe6-a73e-5692728769b3"
Sep 16 14:14:35.896962 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:35.896589    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:14:35.896962 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:35.896882    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7q7r" podUID="4e774454-f78b-40ba-ae21-ed6a11caaea5"
Sep 16 14:14:36.032122 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:36.032086    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t" event={"ID":"5a76ffd0-362f-4576-8836-483741c783d0","Type":"ContainerStarted","Data":"095f8203c0a100838b75bed791bd9d9213fd349e5064c1041cf118ecc2c0e87e"}
Sep 16 14:14:36.033532 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:36.033511    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4d7hc" event={"ID":"5130c0de-1444-4258-8e1d-7b00b0b042a3","Type":"ContainerStarted","Data":"fdd7f4f7849c9bf85d56ce754f31113f904cc9aa2a0b18d0cbed65e95acc981b"}
Sep 16 14:14:36.036300 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:36.036276    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4" event={"ID":"60bd4ded-155c-4351-af05-3e2be2bfc03c","Type":"ContainerStarted","Data":"27b12d1de4d36f7a6f56b1961e8331391f9076e50d03c0159c3f7bbef31678c1"}
Sep 16 14:14:36.036409 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:36.036339    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4" event={"ID":"60bd4ded-155c-4351-af05-3e2be2bfc03c","Type":"ContainerStarted","Data":"ed0f775c20f50b5f0ad99a862b63845780fc7e57398311db51d6ea2ed3e88580"}
Sep 16 14:14:36.036409 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:36.036371    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4" event={"ID":"60bd4ded-155c-4351-af05-3e2be2bfc03c","Type":"ContainerStarted","Data":"cd85971d7ee20a478667e2dfd7ce1974a4d6f99e6189a1ecfb5811e34d6a35ea"}
Sep 16 14:14:36.036492 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:36.036433    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4" event={"ID":"60bd4ded-155c-4351-af05-3e2be2bfc03c","Type":"ContainerStarted","Data":"158d0667be42d5a8eab0e75871907c32f3dd613d97ee31a30163724f6a202f7c"}
Sep 16 14:14:36.036492 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:36.036470    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4" event={"ID":"60bd4ded-155c-4351-af05-3e2be2bfc03c","Type":"ContainerStarted","Data":"dbf6129694b71237df8dcc3831b2dcdc197f276436f5b644c3b47b70caab8b16"}
Sep 16 14:14:36.036576 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:36.036504    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4" event={"ID":"60bd4ded-155c-4351-af05-3e2be2bfc03c","Type":"ContainerStarted","Data":"752766f76a883b9f111cf58cfa38019b4c091938f024696b9867aa36bb4806e5"}
Sep 16 14:14:36.037992 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:36.037973    2587 generic.go:356] "Generic (PLEG): container finished" podID="e71666fa-da06-4d9b-bb6f-383354341f68" containerID="8b5b0f5036f4801c70d5bce2feafe3d6472550b2ac6b63bc2262333c42551591" exitCode=0
Sep 16 14:14:36.038057 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:36.038003    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vtckf" event={"ID":"e71666fa-da06-4d9b-bb6f-383354341f68","Type":"ContainerDied","Data":"8b5b0f5036f4801c70d5bce2feafe3d6472550b2ac6b63bc2262333c42551591"}
Sep 16 14:14:36.056285 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:36.056200    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4d7hc" podStartSLOduration=3.10318129 podStartE2EDuration="24.056185688s" totalImagesPullingTime="20.953004398s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:36.056011896 +0000 UTC m=+24.659636928" watchObservedRunningTime="2026-09-16 14:14:36.056185688 +0000 UTC m=+24.659810719"
Sep 16 14:14:37.043363 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:37.043327    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t" event={"ID":"5a76ffd0-362f-4576-8836-483741c783d0","Type":"ContainerStarted","Data":"e44e9571d3f951207cd5b0c034a558138cc1786120005f9c10f0dce12b5f1fa8"}
Sep 16 14:14:37.896385 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:37.896310    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:14:37.896572 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:37.896305    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:37.896572 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:37.896449    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7q7r" podUID="4e774454-f78b-40ba-ae21-ed6a11caaea5"
Sep 16 14:14:37.896572 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:37.896500    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:37.896730 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:37.896585    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qpvt" podUID="e10f563c-16f8-4699-a7d4-52e6bfba29a5"
Sep 16 14:14:37.896730 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:37.896661    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x59fz" podUID="6f12cf92-2a09-4fe6-a73e-5692728769b3"
Sep 16 14:14:38.048554 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:38.048517    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4" event={"ID":"60bd4ded-155c-4351-af05-3e2be2bfc03c","Type":"ContainerStarted","Data":"a9e3923c6a32b1cda11b2ecea56561027fbb63de320d28baf0c33a630586830b"}
Sep 16 14:14:38.050638 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:38.050607    2587 generic.go:356] "Generic (PLEG): container finished" podID="e71666fa-da06-4d9b-bb6f-383354341f68" containerID="7c16b184edbe9ec0905f6c9816a3426b3ce1506f84835d48b64529a55885c10e" exitCode=0
Sep 16 14:14:38.050736 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:38.050689    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vtckf" event={"ID":"e71666fa-da06-4d9b-bb6f-383354341f68","Type":"ContainerDied","Data":"7c16b184edbe9ec0905f6c9816a3426b3ce1506f84835d48b64529a55885c10e"}
Sep 16 14:14:38.074838 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:38.074793    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xf9t" podStartSLOduration=3.740090349 podStartE2EDuration="26.0747786s" totalImagesPullingTime="22.334688251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:14:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:37.073665269 +0000 UTC m=+25.677290303" watchObservedRunningTime="2026-09-16 14:14:38.0747786 +0000 UTC m=+26.678403661"
Sep 16 14:14:39.896911 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:39.896722    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:39.897858 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:39.897004    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x59fz" podUID="6f12cf92-2a09-4fe6-a73e-5692728769b3"
Sep 16 14:14:39.897858 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:39.896757    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:14:39.897858 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:39.897071    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7q7r" podUID="4e774454-f78b-40ba-ae21-ed6a11caaea5"
Sep 16 14:14:39.897858 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:39.896722    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:39.897858 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:39.897128    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qpvt" podUID="e10f563c-16f8-4699-a7d4-52e6bfba29a5"
Sep 16 14:14:40.058807 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:40.058775    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4" event={"ID":"60bd4ded-155c-4351-af05-3e2be2bfc03c","Type":"ContainerStarted","Data":"cdab558d99e1a9e76fad9a73c12a926bf71cf3af7e5a5c5ef0392c7358aa8721"}
Sep 16 14:14:40.059062 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:40.059042    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:40.074744 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:40.074719    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:40.094736 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:40.092654    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4" podStartSLOduration=7.086948802 podStartE2EDuration="28.092634353s" totalImagesPullingTime="21.005685551s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:40.092316398 +0000 UTC m=+28.695941431" watchObservedRunningTime="2026-09-16 14:14:40.092634353 +0000 UTC m=+28.696259388"
Sep 16 14:14:41.064307 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:41.064274    2587 generic.go:356] "Generic (PLEG): container finished" podID="e71666fa-da06-4d9b-bb6f-383354341f68" containerID="012c99096476f80b5349a62377c270bf909bb4db0f2607cc67300d5617269f0e" exitCode=0
Sep 16 14:14:41.064885 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:41.064351    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vtckf" event={"ID":"e71666fa-da06-4d9b-bb6f-383354341f68","Type":"ContainerDied","Data":"012c99096476f80b5349a62377c270bf909bb4db0f2607cc67300d5617269f0e"}
Sep 16 14:14:41.064885 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:41.064826    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:41.064885 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:41.064885    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:41.091831 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:41.091792    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:14:41.844785 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:41.844613    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x59fz"]
Sep 16 14:14:41.845068 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:41.844918    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:41.845068 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:41.845012    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x59fz" podUID="6f12cf92-2a09-4fe6-a73e-5692728769b3"
Sep 16 14:14:41.847163 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:41.847128    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h7q7r"]
Sep 16 14:14:41.847279 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:41.847250    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:14:41.847350 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:41.847330    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7q7r" podUID="4e774454-f78b-40ba-ae21-ed6a11caaea5"
Sep 16 14:14:41.848103 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:41.848078    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6qpvt"]
Sep 16 14:14:41.848213 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:41.848200    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:41.848325 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:41.848304    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qpvt" podUID="e10f563c-16f8-4699-a7d4-52e6bfba29a5"
Sep 16 14:14:42.896204 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:42.896159    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:42.896739 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:42.896277    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x59fz" podUID="6f12cf92-2a09-4fe6-a73e-5692728769b3"
Sep 16 14:14:43.899545 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:43.899514    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:14:43.900744 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:43.899630    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7q7r" podUID="4e774454-f78b-40ba-ae21-ed6a11caaea5"
Sep 16 14:14:43.900744 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:43.899710    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:43.900744 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:43.899828    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qpvt" podUID="e10f563c-16f8-4699-a7d4-52e6bfba29a5"
Sep 16 14:14:44.896870 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:44.896840    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:44.897034 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:44.896959    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x59fz" podUID="6f12cf92-2a09-4fe6-a73e-5692728769b3"
Sep 16 14:14:45.605076 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:45.605037    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e10f563c-16f8-4699-a7d4-52e6bfba29a5-metrics-certs\") pod \"network-metrics-daemon-6qpvt\" (UID: \"e10f563c-16f8-4699-a7d4-52e6bfba29a5\") " pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:45.605076 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:45.605082    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f12cf92-2a09-4fe6-a73e-5692728769b3-original-pull-secret\") pod \"global-pull-secret-syncer-x59fz\" (UID: \"6f12cf92-2a09-4fe6-a73e-5692728769b3\") " pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:45.605564 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:45.605116    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfwbq\" (UniqueName: \"kubernetes.io/projected/4e774454-f78b-40ba-ae21-ed6a11caaea5-kube-api-access-hfwbq\") pod \"network-check-target-h7q7r\" (UID: \"4e774454-f78b-40ba-ae21-ed6a11caaea5\") " pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:14:45.605564 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:45.605215    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 14:14:45.605564 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:45.605214    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 14:14:45.605564 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:45.605234    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 14:14:45.605564 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:45.605249    2587 projected.go:196] Error preparing data for projected volume kube-api-access-hfwbq for pod openshift-network-diagnostics/network-check-target-h7q7r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:14:45.605564 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:45.605214    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:14:45.605564 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:45.605297    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f12cf92-2a09-4fe6-a73e-5692728769b3-original-pull-secret podName:6f12cf92-2a09-4fe6-a73e-5692728769b3 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:17.605274091 +0000 UTC m=+66.208899120 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f12cf92-2a09-4fe6-a73e-5692728769b3-original-pull-secret") pod "global-pull-secret-syncer-x59fz" (UID: "6f12cf92-2a09-4fe6-a73e-5692728769b3") : object "kube-system"/"original-pull-secret" not registered
Sep 16 14:14:45.605564 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:45.605318    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e10f563c-16f8-4699-a7d4-52e6bfba29a5-metrics-certs podName:e10f563c-16f8-4699-a7d4-52e6bfba29a5 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:17.605304615 +0000 UTC m=+66.208929625 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e10f563c-16f8-4699-a7d4-52e6bfba29a5-metrics-certs") pod "network-metrics-daemon-6qpvt" (UID: "e10f563c-16f8-4699-a7d4-52e6bfba29a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:14:45.605564 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:45.605336    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e774454-f78b-40ba-ae21-ed6a11caaea5-kube-api-access-hfwbq podName:4e774454-f78b-40ba-ae21-ed6a11caaea5 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:17.605325323 +0000 UTC m=+66.208950333 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hfwbq" (UniqueName: "kubernetes.io/projected/4e774454-f78b-40ba-ae21-ed6a11caaea5-kube-api-access-hfwbq") pod "network-check-target-h7q7r" (UID: "4e774454-f78b-40ba-ae21-ed6a11caaea5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:14:45.896173 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:45.896101    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:14:45.896310 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:45.896199    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7q7r" podUID="4e774454-f78b-40ba-ae21-ed6a11caaea5"
Sep 16 14:14:45.896369 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:45.896306    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:45.896468 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:45.896446    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qpvt" podUID="e10f563c-16f8-4699-a7d4-52e6bfba29a5"
Sep 16 14:14:46.896520 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:46.896474    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:46.896932 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:46.896594    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x59fz" podUID="6f12cf92-2a09-4fe6-a73e-5692728769b3"
Sep 16 14:14:47.084719 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:47.084646    2587 generic.go:356] "Generic (PLEG): container finished" podID="e71666fa-da06-4d9b-bb6f-383354341f68" containerID="8a73756d4ea20dd5781f1a2adf092a3b7c819bb11104aedfaa9f4eed128711f9" exitCode=0
Sep 16 14:14:47.084719 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:47.084708    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vtckf" event={"ID":"e71666fa-da06-4d9b-bb6f-383354341f68","Type":"ContainerDied","Data":"8a73756d4ea20dd5781f1a2adf092a3b7c819bb11104aedfaa9f4eed128711f9"}
Sep 16 14:14:47.896288 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:47.896249    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:14:47.896530 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:47.896359    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h7q7r" podUID="4e774454-f78b-40ba-ae21-ed6a11caaea5"
Sep 16 14:14:47.896530 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:47.896441    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:47.896815 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:47.896540    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6qpvt" podUID="e10f563c-16f8-4699-a7d4-52e6bfba29a5"
Sep 16 14:14:48.092471 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.092350    2587 generic.go:356] "Generic (PLEG): container finished" podID="e71666fa-da06-4d9b-bb6f-383354341f68" containerID="055feb9d6b7b3237a1b945a13d842bac5e132557a1fa8ca7d0da96c1adfa1d7a" exitCode=0
Sep 16 14:14:48.092471 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.092435    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vtckf" event={"ID":"e71666fa-da06-4d9b-bb6f-383354341f68","Type":"ContainerDied","Data":"055feb9d6b7b3237a1b945a13d842bac5e132557a1fa8ca7d0da96c1adfa1d7a"}
Sep 16 14:14:48.635909 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.635886    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-217.ec2.internal" event="NodeReady"
Sep 16 14:14:48.636074 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.636004    2587 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 14:14:48.688437 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.688235    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4ntk2"]
Sep 16 14:14:48.706844 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.706803    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-v2zl4"]
Sep 16 14:14:48.707005 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.706962    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4ntk2"
Sep 16 14:14:48.709446 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.709413    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 14:14:48.709446 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.709440    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9f4mh\""
Sep 16 14:14:48.709660 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.709546    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 14:14:48.720851 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.720824    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v2zl4"]
Sep 16 14:14:48.720851 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.720859    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4ntk2"]
Sep 16 14:14:48.721011 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.720951    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v2zl4"
Sep 16 14:14:48.724016 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.723796    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 14:14:48.724016 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.723814    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 14:14:48.724016 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.723824    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 14:14:48.724231 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.724023    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xnd4l\""
Sep 16 14:14:48.827694 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.827608    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-config-volume\") pod \"dns-default-4ntk2\" (UID: \"38e73538-4f3c-4d56-bc47-e9bfb8525e0a\") " pod="openshift-dns/dns-default-4ntk2"
Sep 16 14:14:48.827694 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.827647    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-metrics-tls\") pod \"dns-default-4ntk2\" (UID: \"38e73538-4f3c-4d56-bc47-e9bfb8525e0a\") " pod="openshift-dns/dns-default-4ntk2"
Sep 16 14:14:48.827694 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.827673    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6skr\" (UniqueName: \"kubernetes.io/projected/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-kube-api-access-z6skr\") pod \"dns-default-4ntk2\" (UID: \"38e73538-4f3c-4d56-bc47-e9bfb8525e0a\") " pod="openshift-dns/dns-default-4ntk2"
Sep 16 14:14:48.827900 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.827721    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw45m\" (UniqueName: \"kubernetes.io/projected/bd778876-4a84-403c-b6a7-bfa346cfe0d5-kube-api-access-nw45m\") pod \"ingress-canary-v2zl4\" (UID: \"bd778876-4a84-403c-b6a7-bfa346cfe0d5\") " pod="openshift-ingress-canary/ingress-canary-v2zl4"
Sep 16 14:14:48.827900 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.827756    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-tmp-dir\") pod \"dns-default-4ntk2\" (UID: \"38e73538-4f3c-4d56-bc47-e9bfb8525e0a\") " pod="openshift-dns/dns-default-4ntk2"
Sep 16 14:14:48.827900 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.827772    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd778876-4a84-403c-b6a7-bfa346cfe0d5-cert\") pod \"ingress-canary-v2zl4\" (UID: \"bd778876-4a84-403c-b6a7-bfa346cfe0d5\") " pod="openshift-ingress-canary/ingress-canary-v2zl4"
Sep 16 14:14:48.896863 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.896827    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:14:48.900128 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.900108    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 14:14:48.928625 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.928589    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-tmp-dir\") pod \"dns-default-4ntk2\" (UID: \"38e73538-4f3c-4d56-bc47-e9bfb8525e0a\") " pod="openshift-dns/dns-default-4ntk2"
Sep 16 14:14:48.928625 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.928625    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd778876-4a84-403c-b6a7-bfa346cfe0d5-cert\") pod \"ingress-canary-v2zl4\" (UID: \"bd778876-4a84-403c-b6a7-bfa346cfe0d5\") " pod="openshift-ingress-canary/ingress-canary-v2zl4"
Sep 16 14:14:48.928853 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.928690    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-config-volume\") pod \"dns-default-4ntk2\" (UID: \"38e73538-4f3c-4d56-bc47-e9bfb8525e0a\") " pod="openshift-dns/dns-default-4ntk2"
Sep 16 14:14:48.928853 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.928709    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-metrics-tls\") pod \"dns-default-4ntk2\" (UID: \"38e73538-4f3c-4d56-bc47-e9bfb8525e0a\") " pod="openshift-dns/dns-default-4ntk2"
Sep 16 14:14:48.928853 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.928740    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6skr\" (UniqueName: \"kubernetes.io/projected/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-kube-api-access-z6skr\") pod \"dns-default-4ntk2\" (UID: \"38e73538-4f3c-4d56-bc47-e9bfb8525e0a\") " pod="openshift-dns/dns-default-4ntk2"
Sep 16 14:14:48.928853 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.928764    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw45m\" (UniqueName: \"kubernetes.io/projected/bd778876-4a84-403c-b6a7-bfa346cfe0d5-kube-api-access-nw45m\") pod \"ingress-canary-v2zl4\" (UID: \"bd778876-4a84-403c-b6a7-bfa346cfe0d5\") " pod="openshift-ingress-canary/ingress-canary-v2zl4"
Sep 16 14:14:48.929044 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:48.928855    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 14:14:48.929044 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:48.928929    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-metrics-tls podName:38e73538-4f3c-4d56-bc47-e9bfb8525e0a nodeName:}" failed. No retries permitted until 2026-09-16 14:14:49.428911324 +0000 UTC m=+38.032536356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-metrics-tls") pod "dns-default-4ntk2" (UID: "38e73538-4f3c-4d56-bc47-e9bfb8525e0a") : secret "dns-default-metrics-tls" not found
Sep 16 14:14:48.929190 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.929170    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-tmp-dir\") pod \"dns-default-4ntk2\" (UID: \"38e73538-4f3c-4d56-bc47-e9bfb8525e0a\") " pod="openshift-dns/dns-default-4ntk2"
Sep 16 14:14:48.929464 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.929446    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-config-volume\") pod \"dns-default-4ntk2\" (UID: \"38e73538-4f3c-4d56-bc47-e9bfb8525e0a\") " pod="openshift-dns/dns-default-4ntk2"
Sep 16 14:14:48.933099 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.933081    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd778876-4a84-403c-b6a7-bfa346cfe0d5-cert\") pod \"ingress-canary-v2zl4\" (UID: \"bd778876-4a84-403c-b6a7-bfa346cfe0d5\") " pod="openshift-ingress-canary/ingress-canary-v2zl4"
Sep 16 14:14:48.938517 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.938489    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw45m\" (UniqueName: \"kubernetes.io/projected/bd778876-4a84-403c-b6a7-bfa346cfe0d5-kube-api-access-nw45m\") pod \"ingress-canary-v2zl4\" (UID: \"bd778876-4a84-403c-b6a7-bfa346cfe0d5\") " pod="openshift-ingress-canary/ingress-canary-v2zl4"
Sep 16 14:14:48.938517 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:48.938505    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6skr\" (UniqueName: \"kubernetes.io/projected/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-kube-api-access-z6skr\") pod \"dns-default-4ntk2\" (UID: \"38e73538-4f3c-4d56-bc47-e9bfb8525e0a\") " pod="openshift-dns/dns-default-4ntk2"
Sep 16 14:14:49.029418 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:49.029354    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v2zl4"
Sep 16 14:14:49.098780 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:49.098689    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vtckf" event={"ID":"e71666fa-da06-4d9b-bb6f-383354341f68","Type":"ContainerStarted","Data":"78e16036b18dc5367fd827c672b34aadeaad3ae2b0f90c0a962e614818756088"}
Sep 16 14:14:49.122998 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:49.122924    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vtckf" podStartSLOduration=7.763587093 podStartE2EDuration="37.122901244s" totalImagesPullingTime="29.359314151s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:14:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:49.12175228 +0000 UTC m=+37.725377312" watchObservedRunningTime="2026-09-16 14:14:49.122901244 +0000 UTC m=+37.726526276"
Sep 16 14:14:49.364122 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:49.364048    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v2zl4"]
Sep 16 14:14:49.367615 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:14:49.367579    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd778876_4a84_403c_b6a7_bfa346cfe0d5.slice/crio-ec69316cf8efbf6ba474c8e924b8410978ef194a327b9b06c657506026ebeab2 WatchSource:0}: Error finding container ec69316cf8efbf6ba474c8e924b8410978ef194a327b9b06c657506026ebeab2: Status 404 returned error can't find the container with id ec69316cf8efbf6ba474c8e924b8410978ef194a327b9b06c657506026ebeab2
Sep 16 14:14:49.433902 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:49.433859    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-metrics-tls\") pod \"dns-default-4ntk2\" (UID: \"38e73538-4f3c-4d56-bc47-e9bfb8525e0a\") " pod="openshift-dns/dns-default-4ntk2"
Sep 16 14:14:49.434039 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:49.433972    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 14:14:49.434039 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:49.434027    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-metrics-tls podName:38e73538-4f3c-4d56-bc47-e9bfb8525e0a nodeName:}" failed. No retries permitted until 2026-09-16 14:14:50.434012004 +0000 UTC m=+39.037637027 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-metrics-tls") pod "dns-default-4ntk2" (UID: "38e73538-4f3c-4d56-bc47-e9bfb8525e0a") : secret "dns-default-metrics-tls" not found
Sep 16 14:14:49.896826 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:49.896791    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:14:49.897020 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:49.896790    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:14:49.900113 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:49.900090    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 14:14:49.901162 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:49.901144    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-999gw\""
Sep 16 14:14:49.901269 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:49.901178    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 14:14:49.901540 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:49.901521    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8dcj9\""
Sep 16 14:14:49.901590 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:49.901556    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 14:14:50.102274 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:50.102240    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v2zl4" event={"ID":"bd778876-4a84-403c-b6a7-bfa346cfe0d5","Type":"ContainerStarted","Data":"ec69316cf8efbf6ba474c8e924b8410978ef194a327b9b06c657506026ebeab2"}
Sep 16 14:14:50.441697 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:50.441663    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-metrics-tls\") pod \"dns-default-4ntk2\" (UID: \"38e73538-4f3c-4d56-bc47-e9bfb8525e0a\") " pod="openshift-dns/dns-default-4ntk2"
Sep 16 14:14:50.441868 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:50.441773    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 14:14:50.441868 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:50.441839    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-metrics-tls podName:38e73538-4f3c-4d56-bc47-e9bfb8525e0a nodeName:}" failed. No retries permitted until 2026-09-16 14:14:52.441825446 +0000 UTC m=+41.045450455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-metrics-tls") pod "dns-default-4ntk2" (UID: "38e73538-4f3c-4d56-bc47-e9bfb8525e0a") : secret "dns-default-metrics-tls" not found
Sep 16 14:14:52.454502 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:52.454462    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-metrics-tls\") pod \"dns-default-4ntk2\" (UID: \"38e73538-4f3c-4d56-bc47-e9bfb8525e0a\") " pod="openshift-dns/dns-default-4ntk2"
Sep 16 14:14:52.454916 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:52.454659    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 14:14:52.454916 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:52.454742    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-metrics-tls podName:38e73538-4f3c-4d56-bc47-e9bfb8525e0a nodeName:}" failed. No retries permitted until 2026-09-16 14:14:56.454719438 +0000 UTC m=+45.058344464 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-metrics-tls") pod "dns-default-4ntk2" (UID: "38e73538-4f3c-4d56-bc47-e9bfb8525e0a") : secret "dns-default-metrics-tls" not found
Sep 16 14:14:54.112439 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:54.112217    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v2zl4" event={"ID":"bd778876-4a84-403c-b6a7-bfa346cfe0d5","Type":"ContainerStarted","Data":"f0b01531e5621fb20229c6673ca57d2945b1dee14ab720113e2aa06d2cad9377"}
Sep 16 14:14:54.131887 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:54.131830    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-v2zl4" podStartSLOduration=2.302048383 podStartE2EDuration="6.131812017s" totalImagesPullingTime="3.829763634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:14:54.131204536 +0000 UTC m=+42.734829568" watchObservedRunningTime="2026-09-16 14:14:54.131812017 +0000 UTC m=+42.735437049"
Sep 16 14:14:56.480564 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:14:56.480500    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-metrics-tls\") pod \"dns-default-4ntk2\" (UID: \"38e73538-4f3c-4d56-bc47-e9bfb8525e0a\") " pod="openshift-dns/dns-default-4ntk2"
Sep 16 14:14:56.480981 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:56.480667    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 14:14:56.480981 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:14:56.480735    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-metrics-tls podName:38e73538-4f3c-4d56-bc47-e9bfb8525e0a nodeName:}" failed. No retries permitted until 2026-09-16 14:15:04.480719408 +0000 UTC m=+53.084344419 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-metrics-tls") pod "dns-default-4ntk2" (UID: "38e73538-4f3c-4d56-bc47-e9bfb8525e0a") : secret "dns-default-metrics-tls" not found
Sep 16 14:15:04.537854 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:04.537809    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-metrics-tls\") pod \"dns-default-4ntk2\" (UID: \"38e73538-4f3c-4d56-bc47-e9bfb8525e0a\") " pod="openshift-dns/dns-default-4ntk2"
Sep 16 14:15:04.538330 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:04.537994    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 14:15:04.538330 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:04.538082    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-metrics-tls podName:38e73538-4f3c-4d56-bc47-e9bfb8525e0a nodeName:}" failed. No retries permitted until 2026-09-16 14:15:20.538060105 +0000 UTC m=+69.141685124 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-metrics-tls") pod "dns-default-4ntk2" (UID: "38e73538-4f3c-4d56-bc47-e9bfb8525e0a") : secret "dns-default-metrics-tls" not found
Sep 16 14:15:07.791037 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.791006    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-j674w"]
Sep 16 14:15:07.808525 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.808494    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6f487956db-zccck"]
Sep 16 14:15:07.808679 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.808599    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-j674w"
Sep 16 14:15:07.811322 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.811298    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 16 14:15:07.811481 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.811319    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 16 14:15:07.811481 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.811367    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 14:15:07.811633 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.811614    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 14:15:07.811691 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.811679    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-ts82b\""
Sep 16 14:15:07.817035 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.817010    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 16 14:15:07.830565 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.830540    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-j674w"]
Sep 16 14:15:07.830672 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.830593    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6f487956db-zccck"]
Sep 16 14:15:07.830672 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.830620    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:07.833151 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.833124    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 14:15:07.833262 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.833126    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 14:15:07.833262 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.833237    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2gx92\""
Sep 16 14:15:07.833374 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.833305    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 14:15:07.833450 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.833412    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 14:15:07.833683 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.833666    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 14:15:07.833738 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.833723    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 14:15:07.862914 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.862880    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3e8e2284-b089-4525-80c5-93353f07252f-snapshots\") pod \"insights-operator-5844b856db-j674w\" (UID: \"3e8e2284-b089-4525-80c5-93353f07252f\") " pod="openshift-insights/insights-operator-5844b856db-j674w"
Sep 16 14:15:07.862914 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.862919    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3e8e2284-b089-4525-80c5-93353f07252f-serving-cert\") pod \"insights-operator-5844b856db-j674w\" (UID: \"3e8e2284-b089-4525-80c5-93353f07252f\") " pod="openshift-insights/insights-operator-5844b856db-j674w"
Sep 16 14:15:07.863087 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.862935    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e8e2284-b089-4525-80c5-93353f07252f-tmp\") pod \"insights-operator-5844b856db-j674w\" (UID: \"3e8e2284-b089-4525-80c5-93353f07252f\") " pod="openshift-insights/insights-operator-5844b856db-j674w"
Sep 16 14:15:07.863087 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.863004    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q7n9\" (UniqueName: \"kubernetes.io/projected/3e8e2284-b089-4525-80c5-93353f07252f-kube-api-access-8q7n9\") pod \"insights-operator-5844b856db-j674w\" (UID: \"3e8e2284-b089-4525-80c5-93353f07252f\") " pod="openshift-insights/insights-operator-5844b856db-j674w"
Sep 16 14:15:07.863087 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.863036    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/75e1ae3b-e19a-47f1-9395-c18e8e393062-default-certificate\") pod \"router-default-6f487956db-zccck\" (UID: \"75e1ae3b-e19a-47f1-9395-c18e8e393062\") " pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:07.863087 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.863060    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e8e2284-b089-4525-80c5-93353f07252f-service-ca-bundle\") pod \"insights-operator-5844b856db-j674w\" (UID: \"3e8e2284-b089-4525-80c5-93353f07252f\") " pod="openshift-insights/insights-operator-5844b856db-j674w"
Sep 16 14:15:07.863087 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.863075    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/75e1ae3b-e19a-47f1-9395-c18e8e393062-stats-auth\") pod \"router-default-6f487956db-zccck\" (UID: \"75e1ae3b-e19a-47f1-9395-c18e8e393062\") " pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:07.863236 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.863099    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e1ae3b-e19a-47f1-9395-c18e8e393062-metrics-certs\") pod \"router-default-6f487956db-zccck\" (UID: \"75e1ae3b-e19a-47f1-9395-c18e8e393062\") " pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:07.863236 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.863119    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjddr\" (UniqueName: \"kubernetes.io/projected/75e1ae3b-e19a-47f1-9395-c18e8e393062-kube-api-access-hjddr\") pod \"router-default-6f487956db-zccck\" (UID: \"75e1ae3b-e19a-47f1-9395-c18e8e393062\") " pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:07.863236 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.863145    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75e1ae3b-e19a-47f1-9395-c18e8e393062-service-ca-bundle\") pod \"router-default-6f487956db-zccck\" (UID: \"75e1ae3b-e19a-47f1-9395-c18e8e393062\") " pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:07.863236 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.863211    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e8e2284-b089-4525-80c5-93353f07252f-trusted-ca-bundle\") pod \"insights-operator-5844b856db-j674w\" (UID: \"3e8e2284-b089-4525-80c5-93353f07252f\") " pod="openshift-insights/insights-operator-5844b856db-j674w"
Sep 16 14:15:07.895197 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.895163    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f74cc64bb-gpqp4"]
Sep 16 14:15:07.946894 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.946856    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f74cc64bb-gpqp4"]
Sep 16 14:15:07.947052 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.946961    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:07.949559 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.949502    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 14:15:07.949559 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.949553    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 14:15:07.949776 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.949627    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 14:15:07.950357 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.950338    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-566s2\""
Sep 16 14:15:07.963682 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.963652    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q7n9\" (UniqueName: \"kubernetes.io/projected/3e8e2284-b089-4525-80c5-93353f07252f-kube-api-access-8q7n9\") pod \"insights-operator-5844b856db-j674w\" (UID: \"3e8e2284-b089-4525-80c5-93353f07252f\") " pod="openshift-insights/insights-operator-5844b856db-j674w"
Sep 16 14:15:07.963846 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.963758    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/75e1ae3b-e19a-47f1-9395-c18e8e393062-default-certificate\") pod \"router-default-6f487956db-zccck\" (UID: \"75e1ae3b-e19a-47f1-9395-c18e8e393062\") " pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:07.963846 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.963684    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 14:15:07.964353 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.964325    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-ca-trust-extracted\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:07.964459 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.964377    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e8e2284-b089-4525-80c5-93353f07252f-service-ca-bundle\") pod \"insights-operator-5844b856db-j674w\" (UID: \"3e8e2284-b089-4525-80c5-93353f07252f\") " pod="openshift-insights/insights-operator-5844b856db-j674w"
Sep 16 14:15:07.964459 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.964419    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/75e1ae3b-e19a-47f1-9395-c18e8e393062-stats-auth\") pod \"router-default-6f487956db-zccck\" (UID: \"75e1ae3b-e19a-47f1-9395-c18e8e393062\") " pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:07.964459 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.964441    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e1ae3b-e19a-47f1-9395-c18e8e393062-metrics-certs\") pod \"router-default-6f487956db-zccck\" (UID: \"75e1ae3b-e19a-47f1-9395-c18e8e393062\") " pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:07.964610 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.964463    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjddr\" (UniqueName: \"kubernetes.io/projected/75e1ae3b-e19a-47f1-9395-c18e8e393062-kube-api-access-hjddr\") pod \"router-default-6f487956db-zccck\" (UID: \"75e1ae3b-e19a-47f1-9395-c18e8e393062\") " pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:07.964610 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.964499    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75e1ae3b-e19a-47f1-9395-c18e8e393062-service-ca-bundle\") pod \"router-default-6f487956db-zccck\" (UID: \"75e1ae3b-e19a-47f1-9395-c18e8e393062\") " pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:07.964610 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.964529    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-trusted-ca\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:07.964610 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.964550    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-installation-pull-secrets\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:07.964610 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:07.964577    2587 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 14:15:07.964610 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.964585    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e8e2284-b089-4525-80c5-93353f07252f-trusted-ca-bundle\") pod \"insights-operator-5844b856db-j674w\" (UID: \"3e8e2284-b089-4525-80c5-93353f07252f\") " pod="openshift-insights/insights-operator-5844b856db-j674w"
Sep 16 14:15:07.964885 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.964612    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-registry-certificates\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:07.964885 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:07.964655    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75e1ae3b-e19a-47f1-9395-c18e8e393062-metrics-certs podName:75e1ae3b-e19a-47f1-9395-c18e8e393062 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:08.464630617 +0000 UTC m=+57.068255640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75e1ae3b-e19a-47f1-9395-c18e8e393062-metrics-certs") pod "router-default-6f487956db-zccck" (UID: "75e1ae3b-e19a-47f1-9395-c18e8e393062") : secret "router-metrics-certs-default" not found
Sep 16 14:15:07.964885 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.964702    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3e8e2284-b089-4525-80c5-93353f07252f-snapshots\") pod \"insights-operator-5844b856db-j674w\" (UID: \"3e8e2284-b089-4525-80c5-93353f07252f\") " pod="openshift-insights/insights-operator-5844b856db-j674w"
Sep 16 14:15:07.964885 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:07.964746    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75e1ae3b-e19a-47f1-9395-c18e8e393062-service-ca-bundle podName:75e1ae3b-e19a-47f1-9395-c18e8e393062 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:08.464731102 +0000 UTC m=+57.068356126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/75e1ae3b-e19a-47f1-9395-c18e8e393062-service-ca-bundle") pod "router-default-6f487956db-zccck" (UID: "75e1ae3b-e19a-47f1-9395-c18e8e393062") : configmap references non-existent config key: service-ca.crt
Sep 16 14:15:07.964885 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.964774    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8xc7\" (UniqueName: \"kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-kube-api-access-s8xc7\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:07.964885 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.964817    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3e8e2284-b089-4525-80c5-93353f07252f-serving-cert\") pod \"insights-operator-5844b856db-j674w\" (UID: \"3e8e2284-b089-4525-80c5-93353f07252f\") " pod="openshift-insights/insights-operator-5844b856db-j674w"
Sep 16 14:15:07.964885 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.964841    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-registry-tls\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:07.964885 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.964868    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-bound-sa-token\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:07.965242 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.964903    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e8e2284-b089-4525-80c5-93353f07252f-tmp\") pod \"insights-operator-5844b856db-j674w\" (UID: \"3e8e2284-b089-4525-80c5-93353f07252f\") " pod="openshift-insights/insights-operator-5844b856db-j674w"
Sep 16 14:15:07.965242 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.964928    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-image-registry-private-configuration\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:07.965242 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.965065    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e8e2284-b089-4525-80c5-93353f07252f-service-ca-bundle\") pod \"insights-operator-5844b856db-j674w\" (UID: \"3e8e2284-b089-4525-80c5-93353f07252f\") " pod="openshift-insights/insights-operator-5844b856db-j674w"
Sep 16 14:15:07.965413 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.965272    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3e8e2284-b089-4525-80c5-93353f07252f-snapshots\") pod \"insights-operator-5844b856db-j674w\" (UID: \"3e8e2284-b089-4525-80c5-93353f07252f\") " pod="openshift-insights/insights-operator-5844b856db-j674w"
Sep 16 14:15:07.965413 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.965338    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e8e2284-b089-4525-80c5-93353f07252f-tmp\") pod \"insights-operator-5844b856db-j674w\" (UID: \"3e8e2284-b089-4525-80c5-93353f07252f\") " pod="openshift-insights/insights-operator-5844b856db-j674w"
Sep 16 14:15:07.965568 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.965551    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e8e2284-b089-4525-80c5-93353f07252f-trusted-ca-bundle\") pod \"insights-operator-5844b856db-j674w\" (UID: \"3e8e2284-b089-4525-80c5-93353f07252f\") " pod="openshift-insights/insights-operator-5844b856db-j674w"
Sep 16 14:15:07.967946 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.967924    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3e8e2284-b089-4525-80c5-93353f07252f-serving-cert\") pod \"insights-operator-5844b856db-j674w\" (UID: \"3e8e2284-b089-4525-80c5-93353f07252f\") " pod="openshift-insights/insights-operator-5844b856db-j674w"
Sep 16 14:15:07.968025 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.967987    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/75e1ae3b-e19a-47f1-9395-c18e8e393062-default-certificate\") pod \"router-default-6f487956db-zccck\" (UID: \"75e1ae3b-e19a-47f1-9395-c18e8e393062\") " pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:07.968025 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.967986    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/75e1ae3b-e19a-47f1-9395-c18e8e393062-stats-auth\") pod \"router-default-6f487956db-zccck\" (UID: \"75e1ae3b-e19a-47f1-9395-c18e8e393062\") " pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:07.976216 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.976183    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjddr\" (UniqueName: \"kubernetes.io/projected/75e1ae3b-e19a-47f1-9395-c18e8e393062-kube-api-access-hjddr\") pod \"router-default-6f487956db-zccck\" (UID: \"75e1ae3b-e19a-47f1-9395-c18e8e393062\") " pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:07.976446 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.976429    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q7n9\" (UniqueName: \"kubernetes.io/projected/3e8e2284-b089-4525-80c5-93353f07252f-kube-api-access-8q7n9\") pod \"insights-operator-5844b856db-j674w\" (UID: \"3e8e2284-b089-4525-80c5-93353f07252f\") " pod="openshift-insights/insights-operator-5844b856db-j674w"
Sep 16 14:15:07.996525 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:07.996492    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-zssjp"]
Sep 16 14:15:08.016143 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.016109    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-w82jn"]
Sep 16 14:15:08.016296 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.016239    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-zssjp"
Sep 16 14:15:08.019265 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.019246    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 16 14:15:08.019265 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.019249    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 16 14:15:08.019491 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.019321    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 16 14:15:08.019491 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.019349    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-rpkck\""
Sep 16 14:15:08.025974 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.025956    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-zssjp"]
Sep 16 14:15:08.026097 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.025987    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-w82jn"]
Sep 16 14:15:08.026097 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.026048    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w82jn"
Sep 16 14:15:08.029712 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.029684    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 16 14:15:08.029815 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.029734    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-w6vcf\""
Sep 16 14:15:08.029815 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.029802    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 16 14:15:08.030176 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.030159    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 16 14:15:08.030671 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.030583    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 16 14:15:08.065472 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.065354    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8xc7\" (UniqueName: \"kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-kube-api-access-s8xc7\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:08.065472 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.065449    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68975d6e-a86b-480f-94ba-b657b5b09fd1-config\") pod \"service-ca-operator-cf7c776b7-w82jn\" (UID: \"68975d6e-a86b-480f-94ba-b657b5b09fd1\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w82jn"
Sep 16 14:15:08.065679 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.065478    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/077ee101-a706-4d61-8c6d-69d002680eba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-zssjp\" (UID: \"077ee101-a706-4d61-8c6d-69d002680eba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-zssjp"
Sep 16 14:15:08.065679 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.065504    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-ca-trust-extracted\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:08.065679 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.065537    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-489fc\" (UniqueName: \"kubernetes.io/projected/68975d6e-a86b-480f-94ba-b657b5b09fd1-kube-api-access-489fc\") pod \"service-ca-operator-cf7c776b7-w82jn\" (UID: \"68975d6e-a86b-480f-94ba-b657b5b09fd1\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w82jn"
Sep 16 14:15:08.065679 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.065660    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-registry-tls\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:08.065826 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.065682    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-bound-sa-token\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:08.065826 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.065722    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-image-registry-private-configuration\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:08.065826 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.065753    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts886\" (UniqueName: \"kubernetes.io/projected/077ee101-a706-4d61-8c6d-69d002680eba-kube-api-access-ts886\") pod \"cluster-samples-operator-86499c89c4-zssjp\" (UID: \"077ee101-a706-4d61-8c6d-69d002680eba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-zssjp"
Sep 16 14:15:08.065826 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.065777    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68975d6e-a86b-480f-94ba-b657b5b09fd1-serving-cert\") pod \"service-ca-operator-cf7c776b7-w82jn\" (UID: \"68975d6e-a86b-480f-94ba-b657b5b09fd1\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w82jn"
Sep 16 14:15:08.065826 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:08.065787    2587 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 14:15:08.065826 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.065795    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-trusted-ca\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:08.065826 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:08.065802    2587 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f74cc64bb-gpqp4: secret "image-registry-tls" not found
Sep 16 14:15:08.065826 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.065825    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-installation-pull-secrets\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:08.066120 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:08.065868    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-registry-tls podName:8c16289f-192f-4eda-a7e5-3c8aef6f14e6 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:08.565848458 +0000 UTC m=+57.169473472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-registry-tls") pod "image-registry-7f74cc64bb-gpqp4" (UID: "8c16289f-192f-4eda-a7e5-3c8aef6f14e6") : secret "image-registry-tls" not found
Sep 16 14:15:08.066120 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.065933    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-ca-trust-extracted\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:08.066120 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.066045    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-registry-certificates\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:08.066514 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.066490    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-registry-certificates\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:08.066810 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.066790    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-trusted-ca\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:08.068427 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.068384    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-image-registry-private-configuration\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:08.068511 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.068405    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-installation-pull-secrets\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:08.077632 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.077614    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-bound-sa-token\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:08.077718 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.077632    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8xc7\" (UniqueName: \"kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-kube-api-access-s8xc7\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:08.119764 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.119730    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-j674w"
Sep 16 14:15:08.166918 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.166893    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts886\" (UniqueName: \"kubernetes.io/projected/077ee101-a706-4d61-8c6d-69d002680eba-kube-api-access-ts886\") pod \"cluster-samples-operator-86499c89c4-zssjp\" (UID: \"077ee101-a706-4d61-8c6d-69d002680eba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-zssjp"
Sep 16 14:15:08.167060 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.166929    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68975d6e-a86b-480f-94ba-b657b5b09fd1-serving-cert\") pod \"service-ca-operator-cf7c776b7-w82jn\" (UID: \"68975d6e-a86b-480f-94ba-b657b5b09fd1\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w82jn"
Sep 16 14:15:08.167060 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.166973    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68975d6e-a86b-480f-94ba-b657b5b09fd1-config\") pod \"service-ca-operator-cf7c776b7-w82jn\" (UID: \"68975d6e-a86b-480f-94ba-b657b5b09fd1\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w82jn"
Sep 16 14:15:08.167060 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.166999    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/077ee101-a706-4d61-8c6d-69d002680eba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-zssjp\" (UID: \"077ee101-a706-4d61-8c6d-69d002680eba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-zssjp"
Sep 16 14:15:08.167060 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.167044    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-489fc\" (UniqueName: \"kubernetes.io/projected/68975d6e-a86b-480f-94ba-b657b5b09fd1-kube-api-access-489fc\") pod \"service-ca-operator-cf7c776b7-w82jn\" (UID: \"68975d6e-a86b-480f-94ba-b657b5b09fd1\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w82jn"
Sep 16 14:15:08.167310 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:08.167170    2587 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 14:15:08.167310 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:08.167229    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/077ee101-a706-4d61-8c6d-69d002680eba-samples-operator-tls podName:077ee101-a706-4d61-8c6d-69d002680eba nodeName:}" failed. No retries permitted until 2026-09-16 14:15:08.667213857 +0000 UTC m=+57.270838889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/077ee101-a706-4d61-8c6d-69d002680eba-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-zssjp" (UID: "077ee101-a706-4d61-8c6d-69d002680eba") : secret "samples-operator-tls" not found
Sep 16 14:15:08.167606 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.167583    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68975d6e-a86b-480f-94ba-b657b5b09fd1-config\") pod \"service-ca-operator-cf7c776b7-w82jn\" (UID: \"68975d6e-a86b-480f-94ba-b657b5b09fd1\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w82jn"
Sep 16 14:15:08.169513 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.169493    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68975d6e-a86b-480f-94ba-b657b5b09fd1-serving-cert\") pod \"service-ca-operator-cf7c776b7-w82jn\" (UID: \"68975d6e-a86b-480f-94ba-b657b5b09fd1\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w82jn"
Sep 16 14:15:08.179513 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.179484    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts886\" (UniqueName: \"kubernetes.io/projected/077ee101-a706-4d61-8c6d-69d002680eba-kube-api-access-ts886\") pod \"cluster-samples-operator-86499c89c4-zssjp\" (UID: \"077ee101-a706-4d61-8c6d-69d002680eba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-zssjp"
Sep 16 14:15:08.179513 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.179509    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-489fc\" (UniqueName: \"kubernetes.io/projected/68975d6e-a86b-480f-94ba-b657b5b09fd1-kube-api-access-489fc\") pod \"service-ca-operator-cf7c776b7-w82jn\" (UID: \"68975d6e-a86b-480f-94ba-b657b5b09fd1\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w82jn"
Sep 16 14:15:08.334264 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.334177    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w82jn"
Sep 16 14:15:08.418813 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.418778    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-j674w"]
Sep 16 14:15:08.470202 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.470164    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75e1ae3b-e19a-47f1-9395-c18e8e393062-service-ca-bundle\") pod \"router-default-6f487956db-zccck\" (UID: \"75e1ae3b-e19a-47f1-9395-c18e8e393062\") " pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:08.470381 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.470229    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e1ae3b-e19a-47f1-9395-c18e8e393062-metrics-certs\") pod \"router-default-6f487956db-zccck\" (UID: \"75e1ae3b-e19a-47f1-9395-c18e8e393062\") " pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:08.470381 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:08.470314    2587 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 14:15:08.470486 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:08.470381    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75e1ae3b-e19a-47f1-9395-c18e8e393062-service-ca-bundle podName:75e1ae3b-e19a-47f1-9395-c18e8e393062 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:09.470357621 +0000 UTC m=+58.073982648 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/75e1ae3b-e19a-47f1-9395-c18e8e393062-service-ca-bundle") pod "router-default-6f487956db-zccck" (UID: "75e1ae3b-e19a-47f1-9395-c18e8e393062") : configmap references non-existent config key: service-ca.crt
Sep 16 14:15:08.470486 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:08.470429    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75e1ae3b-e19a-47f1-9395-c18e8e393062-metrics-certs podName:75e1ae3b-e19a-47f1-9395-c18e8e393062 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:09.470417537 +0000 UTC m=+58.074042552 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75e1ae3b-e19a-47f1-9395-c18e8e393062-metrics-certs") pod "router-default-6f487956db-zccck" (UID: "75e1ae3b-e19a-47f1-9395-c18e8e393062") : secret "router-metrics-certs-default" not found
Sep 16 14:15:08.571033 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.571001    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-registry-tls\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:08.571171 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:08.571122    2587 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 14:15:08.571171 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:08.571135    2587 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f74cc64bb-gpqp4: secret "image-registry-tls" not found
Sep 16 14:15:08.571244 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:08.571185    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-registry-tls podName:8c16289f-192f-4eda-a7e5-3c8aef6f14e6 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:09.571171729 +0000 UTC m=+58.174796739 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-registry-tls") pod "image-registry-7f74cc64bb-gpqp4" (UID: "8c16289f-192f-4eda-a7e5-3c8aef6f14e6") : secret "image-registry-tls" not found
Sep 16 14:15:08.639346 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.639314    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-w82jn"]
Sep 16 14:15:08.641644 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:15:08.641616    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68975d6e_a86b_480f_94ba_b657b5b09fd1.slice/crio-6a4a2cc44e60d9ad0baac5f1223711154d12e88d325c16c34cfc84810d233444 WatchSource:0}: Error finding container 6a4a2cc44e60d9ad0baac5f1223711154d12e88d325c16c34cfc84810d233444: Status 404 returned error can't find the container with id 6a4a2cc44e60d9ad0baac5f1223711154d12e88d325c16c34cfc84810d233444
Sep 16 14:15:08.671837 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:08.671806    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/077ee101-a706-4d61-8c6d-69d002680eba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-zssjp\" (UID: \"077ee101-a706-4d61-8c6d-69d002680eba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-zssjp"
Sep 16 14:15:08.671984 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:08.671920    2587 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 14:15:08.671984 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:08.671968    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/077ee101-a706-4d61-8c6d-69d002680eba-samples-operator-tls podName:077ee101-a706-4d61-8c6d-69d002680eba nodeName:}" failed. No retries permitted until 2026-09-16 14:15:09.671956486 +0000 UTC m=+58.275581496 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/077ee101-a706-4d61-8c6d-69d002680eba-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-zssjp" (UID: "077ee101-a706-4d61-8c6d-69d002680eba") : secret "samples-operator-tls" not found
Sep 16 14:15:09.149012 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:09.148979    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w82jn" event={"ID":"68975d6e-a86b-480f-94ba-b657b5b09fd1","Type":"ContainerStarted","Data":"6a4a2cc44e60d9ad0baac5f1223711154d12e88d325c16c34cfc84810d233444"}
Sep 16 14:15:09.150104 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:09.150083    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-j674w" event={"ID":"3e8e2284-b089-4525-80c5-93353f07252f","Type":"ContainerStarted","Data":"d8138c336c8fd75affccd474dc3a693ab411b839642db6374c4a8623a460a3e8"}
Sep 16 14:15:09.478372 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:09.478288    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75e1ae3b-e19a-47f1-9395-c18e8e393062-service-ca-bundle\") pod \"router-default-6f487956db-zccck\" (UID: \"75e1ae3b-e19a-47f1-9395-c18e8e393062\") " pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:09.478372 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:09.478358    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e1ae3b-e19a-47f1-9395-c18e8e393062-metrics-certs\") pod \"router-default-6f487956db-zccck\" (UID: \"75e1ae3b-e19a-47f1-9395-c18e8e393062\") " pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:09.478603 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:09.478488    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75e1ae3b-e19a-47f1-9395-c18e8e393062-service-ca-bundle podName:75e1ae3b-e19a-47f1-9395-c18e8e393062 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:11.478470141 +0000 UTC m=+60.082095150 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/75e1ae3b-e19a-47f1-9395-c18e8e393062-service-ca-bundle") pod "router-default-6f487956db-zccck" (UID: "75e1ae3b-e19a-47f1-9395-c18e8e393062") : configmap references non-existent config key: service-ca.crt
Sep 16 14:15:09.478603 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:09.478504    2587 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 14:15:09.478603 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:09.478555    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75e1ae3b-e19a-47f1-9395-c18e8e393062-metrics-certs podName:75e1ae3b-e19a-47f1-9395-c18e8e393062 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:11.478539397 +0000 UTC m=+60.082164430 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75e1ae3b-e19a-47f1-9395-c18e8e393062-metrics-certs") pod "router-default-6f487956db-zccck" (UID: "75e1ae3b-e19a-47f1-9395-c18e8e393062") : secret "router-metrics-certs-default" not found
Sep 16 14:15:09.579056 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:09.579021    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-registry-tls\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:09.579203 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:09.579179    2587 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 14:15:09.579243 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:09.579205    2587 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f74cc64bb-gpqp4: secret "image-registry-tls" not found
Sep 16 14:15:09.579322 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:09.579313    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-registry-tls podName:8c16289f-192f-4eda-a7e5-3c8aef6f14e6 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:11.579298689 +0000 UTC m=+60.182923713 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-registry-tls") pod "image-registry-7f74cc64bb-gpqp4" (UID: "8c16289f-192f-4eda-a7e5-3c8aef6f14e6") : secret "image-registry-tls" not found
Sep 16 14:15:09.679917 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:09.679876    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/077ee101-a706-4d61-8c6d-69d002680eba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-zssjp\" (UID: \"077ee101-a706-4d61-8c6d-69d002680eba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-zssjp"
Sep 16 14:15:09.680091 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:09.679999    2587 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 14:15:09.680091 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:09.680056    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/077ee101-a706-4d61-8c6d-69d002680eba-samples-operator-tls podName:077ee101-a706-4d61-8c6d-69d002680eba nodeName:}" failed. No retries permitted until 2026-09-16 14:15:11.68004155 +0000 UTC m=+60.283666560 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/077ee101-a706-4d61-8c6d-69d002680eba-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-zssjp" (UID: "077ee101-a706-4d61-8c6d-69d002680eba") : secret "samples-operator-tls" not found
Sep 16 14:15:11.496312 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:11.496269    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75e1ae3b-e19a-47f1-9395-c18e8e393062-service-ca-bundle\") pod \"router-default-6f487956db-zccck\" (UID: \"75e1ae3b-e19a-47f1-9395-c18e8e393062\") " pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:11.496829 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:11.496379    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e1ae3b-e19a-47f1-9395-c18e8e393062-metrics-certs\") pod \"router-default-6f487956db-zccck\" (UID: \"75e1ae3b-e19a-47f1-9395-c18e8e393062\") " pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:11.496829 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:11.496519    2587 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 14:15:11.496829 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:11.496528    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75e1ae3b-e19a-47f1-9395-c18e8e393062-service-ca-bundle podName:75e1ae3b-e19a-47f1-9395-c18e8e393062 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:15.496503024 +0000 UTC m=+64.100128040 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/75e1ae3b-e19a-47f1-9395-c18e8e393062-service-ca-bundle") pod "router-default-6f487956db-zccck" (UID: "75e1ae3b-e19a-47f1-9395-c18e8e393062") : configmap references non-existent config key: service-ca.crt
Sep 16 14:15:11.496829 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:11.496577    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75e1ae3b-e19a-47f1-9395-c18e8e393062-metrics-certs podName:75e1ae3b-e19a-47f1-9395-c18e8e393062 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:15.496559275 +0000 UTC m=+64.100184289 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75e1ae3b-e19a-47f1-9395-c18e8e393062-metrics-certs") pod "router-default-6f487956db-zccck" (UID: "75e1ae3b-e19a-47f1-9395-c18e8e393062") : secret "router-metrics-certs-default" not found
Sep 16 14:15:11.597692 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:11.597647    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-registry-tls\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:11.597876 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:11.597858    2587 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 14:15:11.597930 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:11.597884    2587 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f74cc64bb-gpqp4: secret "image-registry-tls" not found
Sep 16 14:15:11.597976 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:11.597959    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-registry-tls podName:8c16289f-192f-4eda-a7e5-3c8aef6f14e6 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:15.597937767 +0000 UTC m=+64.201562797 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-registry-tls") pod "image-registry-7f74cc64bb-gpqp4" (UID: "8c16289f-192f-4eda-a7e5-3c8aef6f14e6") : secret "image-registry-tls" not found
Sep 16 14:15:11.698469 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:11.698432    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/077ee101-a706-4d61-8c6d-69d002680eba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-zssjp\" (UID: \"077ee101-a706-4d61-8c6d-69d002680eba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-zssjp"
Sep 16 14:15:11.698637 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:11.698569    2587 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 14:15:11.698754 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:11.698640    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/077ee101-a706-4d61-8c6d-69d002680eba-samples-operator-tls podName:077ee101-a706-4d61-8c6d-69d002680eba nodeName:}" failed. No retries permitted until 2026-09-16 14:15:15.698620315 +0000 UTC m=+64.302245339 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/077ee101-a706-4d61-8c6d-69d002680eba-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-zssjp" (UID: "077ee101-a706-4d61-8c6d-69d002680eba") : secret "samples-operator-tls" not found
Sep 16 14:15:12.167785 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:12.167749    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w82jn" event={"ID":"68975d6e-a86b-480f-94ba-b657b5b09fd1","Type":"ContainerStarted","Data":"a59b2311394f3abc12ac2393cd265e19213f0e062228642d0b4a10ff20c1aa63"}
Sep 16 14:15:12.169376 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:12.169351    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-j674w" event={"ID":"3e8e2284-b089-4525-80c5-93353f07252f","Type":"ContainerStarted","Data":"1cdedc16d60b66d3fd1fa8856370de0849d5a4fc9b2f1a2da0d3923ea7e7b2ff"}
Sep 16 14:15:12.187186 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:12.187132    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w82jn" podStartSLOduration=2.088053313 podStartE2EDuration="5.187117158s" totalImagesPullingTime="3.099063845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:12.186575685 +0000 UTC m=+60.790200718" watchObservedRunningTime="2026-09-16 14:15:12.187117158 +0000 UTC m=+60.790742206"
Sep 16 14:15:12.209480 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:12.208864    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-j674w" podStartSLOduration=1.8898160339999999 podStartE2EDuration="5.208841885s" totalImagesPullingTime="3.319025851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:12.204008985 +0000 UTC m=+60.807634015" watchObservedRunningTime="2026-09-16 14:15:12.208841885 +0000 UTC m=+60.812466920"
Sep 16 14:15:12.887304 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:12.887274    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lqtmz"]
Sep 16 14:15:12.890728 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:12.890703    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lqtmz"
Sep 16 14:15:12.893456 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:12.893424    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qst7x\""
Sep 16 14:15:12.893456 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:12.893438    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 14:15:12.893456 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:12.893451    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 14:15:12.907241 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:12.907208    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lqtmz"]
Sep 16 14:15:12.908220 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:12.908196    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ae7e37a-60a5-4d09-8e0f-905cad155314-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lqtmz\" (UID: \"1ae7e37a-60a5-4d09-8e0f-905cad155314\") " pod="openshift-insights/insights-runtime-extractor-lqtmz"
Sep 16 14:15:12.908354 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:12.908255    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1ae7e37a-60a5-4d09-8e0f-905cad155314-crio-socket\") pod \"insights-runtime-extractor-lqtmz\" (UID: \"1ae7e37a-60a5-4d09-8e0f-905cad155314\") " pod="openshift-insights/insights-runtime-extractor-lqtmz"
Sep 16 14:15:12.908354 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:12.908290    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ae7e37a-60a5-4d09-8e0f-905cad155314-data-volume\") pod \"insights-runtime-extractor-lqtmz\" (UID: \"1ae7e37a-60a5-4d09-8e0f-905cad155314\") " pod="openshift-insights/insights-runtime-extractor-lqtmz"
Sep 16 14:15:12.908354 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:12.908307    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ae7e37a-60a5-4d09-8e0f-905cad155314-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lqtmz\" (UID: \"1ae7e37a-60a5-4d09-8e0f-905cad155314\") " pod="openshift-insights/insights-runtime-extractor-lqtmz"
Sep 16 14:15:12.908354 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:12.908324    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lzk8\" (UniqueName: \"kubernetes.io/projected/1ae7e37a-60a5-4d09-8e0f-905cad155314-kube-api-access-6lzk8\") pod \"insights-runtime-extractor-lqtmz\" (UID: \"1ae7e37a-60a5-4d09-8e0f-905cad155314\") " pod="openshift-insights/insights-runtime-extractor-lqtmz"
Sep 16 14:15:13.008714 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:13.008675    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1ae7e37a-60a5-4d09-8e0f-905cad155314-crio-socket\") pod \"insights-runtime-extractor-lqtmz\" (UID: \"1ae7e37a-60a5-4d09-8e0f-905cad155314\") " pod="openshift-insights/insights-runtime-extractor-lqtmz"
Sep 16 14:15:13.008894 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:13.008724    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ae7e37a-60a5-4d09-8e0f-905cad155314-data-volume\") pod \"insights-runtime-extractor-lqtmz\" (UID: \"1ae7e37a-60a5-4d09-8e0f-905cad155314\") " pod="openshift-insights/insights-runtime-extractor-lqtmz"
Sep 16 14:15:13.008894 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:13.008744    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ae7e37a-60a5-4d09-8e0f-905cad155314-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lqtmz\" (UID: \"1ae7e37a-60a5-4d09-8e0f-905cad155314\") " pod="openshift-insights/insights-runtime-extractor-lqtmz"
Sep 16 14:15:13.008894 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:13.008769    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lzk8\" (UniqueName: \"kubernetes.io/projected/1ae7e37a-60a5-4d09-8e0f-905cad155314-kube-api-access-6lzk8\") pod \"insights-runtime-extractor-lqtmz\" (UID: \"1ae7e37a-60a5-4d09-8e0f-905cad155314\") " pod="openshift-insights/insights-runtime-extractor-lqtmz"
Sep 16 14:15:13.008894 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:13.008809    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1ae7e37a-60a5-4d09-8e0f-905cad155314-crio-socket\") pod \"insights-runtime-extractor-lqtmz\" (UID: \"1ae7e37a-60a5-4d09-8e0f-905cad155314\") " pod="openshift-insights/insights-runtime-extractor-lqtmz"
Sep 16 14:15:13.008894 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:13.008878    2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 14:15:13.009083 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:13.008921    2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 14:15:13.009083 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:13.008884    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ae7e37a-60a5-4d09-8e0f-905cad155314-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lqtmz\" (UID: \"1ae7e37a-60a5-4d09-8e0f-905cad155314\") " pod="openshift-insights/insights-runtime-extractor-lqtmz"
Sep 16 14:15:13.009083 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:13.008960    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ae7e37a-60a5-4d09-8e0f-905cad155314-insights-runtime-extractor-tls podName:1ae7e37a-60a5-4d09-8e0f-905cad155314 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:13.508938989 +0000 UTC m=+62.112564004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1ae7e37a-60a5-4d09-8e0f-905cad155314-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lqtmz" (UID: "1ae7e37a-60a5-4d09-8e0f-905cad155314") : secret "insights-runtime-extractor-tls" not found
Sep 16 14:15:13.009083 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:13.008995    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1ae7e37a-60a5-4d09-8e0f-905cad155314-kube-rbac-proxy-cm podName:1ae7e37a-60a5-4d09-8e0f-905cad155314 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:13.508979779 +0000 UTC m=+62.112604788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1ae7e37a-60a5-4d09-8e0f-905cad155314-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lqtmz" (UID: "1ae7e37a-60a5-4d09-8e0f-905cad155314") : configmap "kube-rbac-proxy" not found
Sep 16 14:15:13.009221 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:13.009089    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ae7e37a-60a5-4d09-8e0f-905cad155314-data-volume\") pod \"insights-runtime-extractor-lqtmz\" (UID: \"1ae7e37a-60a5-4d09-8e0f-905cad155314\") " pod="openshift-insights/insights-runtime-extractor-lqtmz"
Sep 16 14:15:13.017825 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:13.017795    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lzk8\" (UniqueName: \"kubernetes.io/projected/1ae7e37a-60a5-4d09-8e0f-905cad155314-kube-api-access-6lzk8\") pod \"insights-runtime-extractor-lqtmz\" (UID: \"1ae7e37a-60a5-4d09-8e0f-905cad155314\") " pod="openshift-insights/insights-runtime-extractor-lqtmz"
Sep 16 14:15:13.081732 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:13.081703    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbsc4"
Sep 16 14:15:13.513803 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:13.513764    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ae7e37a-60a5-4d09-8e0f-905cad155314-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lqtmz\" (UID: \"1ae7e37a-60a5-4d09-8e0f-905cad155314\") " pod="openshift-insights/insights-runtime-extractor-lqtmz"
Sep 16 14:15:13.513996 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:13.513877    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ae7e37a-60a5-4d09-8e0f-905cad155314-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lqtmz\" (UID: \"1ae7e37a-60a5-4d09-8e0f-905cad155314\") " pod="openshift-insights/insights-runtime-extractor-lqtmz"
Sep 16 14:15:13.513996 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:13.513937    2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 14:15:13.513996 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:13.513965    2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 14:15:13.514099 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:13.514009    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ae7e37a-60a5-4d09-8e0f-905cad155314-insights-runtime-extractor-tls podName:1ae7e37a-60a5-4d09-8e0f-905cad155314 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:14.513993186 +0000 UTC m=+63.117618216 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1ae7e37a-60a5-4d09-8e0f-905cad155314-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lqtmz" (UID: "1ae7e37a-60a5-4d09-8e0f-905cad155314") : secret "insights-runtime-extractor-tls" not found
Sep 16 14:15:13.514099 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:13.514024    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1ae7e37a-60a5-4d09-8e0f-905cad155314-kube-rbac-proxy-cm podName:1ae7e37a-60a5-4d09-8e0f-905cad155314 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:14.5140181 +0000 UTC m=+63.117643109 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1ae7e37a-60a5-4d09-8e0f-905cad155314-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lqtmz" (UID: "1ae7e37a-60a5-4d09-8e0f-905cad155314") : configmap "kube-rbac-proxy" not found
Sep 16 14:15:14.522534 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:14.522490    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ae7e37a-60a5-4d09-8e0f-905cad155314-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lqtmz\" (UID: \"1ae7e37a-60a5-4d09-8e0f-905cad155314\") " pod="openshift-insights/insights-runtime-extractor-lqtmz"
Sep 16 14:15:14.522928 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:14.522613    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ae7e37a-60a5-4d09-8e0f-905cad155314-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lqtmz\" (UID: \"1ae7e37a-60a5-4d09-8e0f-905cad155314\") " pod="openshift-insights/insights-runtime-extractor-lqtmz"
Sep 16 14:15:14.522928 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:14.522613    2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 14:15:14.522928 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:14.522683    2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 14:15:14.522928 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:14.522727    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1ae7e37a-60a5-4d09-8e0f-905cad155314-kube-rbac-proxy-cm podName:1ae7e37a-60a5-4d09-8e0f-905cad155314 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:16.522711969 +0000 UTC m=+65.126336979 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1ae7e37a-60a5-4d09-8e0f-905cad155314-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lqtmz" (UID: "1ae7e37a-60a5-4d09-8e0f-905cad155314") : configmap "kube-rbac-proxy" not found
Sep 16 14:15:14.522928 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:14.522741    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ae7e37a-60a5-4d09-8e0f-905cad155314-insights-runtime-extractor-tls podName:1ae7e37a-60a5-4d09-8e0f-905cad155314 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:16.522734813 +0000 UTC m=+65.126359822 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1ae7e37a-60a5-4d09-8e0f-905cad155314-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lqtmz" (UID: "1ae7e37a-60a5-4d09-8e0f-905cad155314") : secret "insights-runtime-extractor-tls" not found
Sep 16 14:15:14.637790 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:14.637755    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-trl7p_4193914e-79e1-4c6a-ba3b-a4d9f5731637/dns-node-resolver/0.log"
Sep 16 14:15:15.437890 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:15.437865    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-drp7m_d657d718-fd8c-4034-9cd8-232f90fc573e/node-ca/0.log"
Sep 16 14:15:15.530672 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:15.530616    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e1ae3b-e19a-47f1-9395-c18e8e393062-metrics-certs\") pod \"router-default-6f487956db-zccck\" (UID: \"75e1ae3b-e19a-47f1-9395-c18e8e393062\") " pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:15.531068 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:15.530761    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75e1ae3b-e19a-47f1-9395-c18e8e393062-service-ca-bundle\") pod \"router-default-6f487956db-zccck\" (UID: \"75e1ae3b-e19a-47f1-9395-c18e8e393062\") " pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:15.531068 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:15.530777    2587 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 14:15:15.531068 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:15.530842    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75e1ae3b-e19a-47f1-9395-c18e8e393062-metrics-certs podName:75e1ae3b-e19a-47f1-9395-c18e8e393062 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:23.53082271 +0000 UTC m=+72.134447743 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/75e1ae3b-e19a-47f1-9395-c18e8e393062-metrics-certs") pod "router-default-6f487956db-zccck" (UID: "75e1ae3b-e19a-47f1-9395-c18e8e393062") : secret "router-metrics-certs-default" not found
Sep 16 14:15:15.531068 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:15.530924    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75e1ae3b-e19a-47f1-9395-c18e8e393062-service-ca-bundle podName:75e1ae3b-e19a-47f1-9395-c18e8e393062 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:23.530905921 +0000 UTC m=+72.134530930 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/75e1ae3b-e19a-47f1-9395-c18e8e393062-service-ca-bundle") pod "router-default-6f487956db-zccck" (UID: "75e1ae3b-e19a-47f1-9395-c18e8e393062") : configmap references non-existent config key: service-ca.crt
Sep 16 14:15:15.631754 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:15.631718    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-registry-tls\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:15.631915 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:15.631870    2587 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 14:15:15.631915 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:15.631890    2587 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f74cc64bb-gpqp4: secret "image-registry-tls" not found
Sep 16 14:15:15.631983 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:15.631944    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-registry-tls podName:8c16289f-192f-4eda-a7e5-3c8aef6f14e6 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:23.631929725 +0000 UTC m=+72.235554748 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-registry-tls") pod "image-registry-7f74cc64bb-gpqp4" (UID: "8c16289f-192f-4eda-a7e5-3c8aef6f14e6") : secret "image-registry-tls" not found
Sep 16 14:15:15.733096 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:15.732990    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/077ee101-a706-4d61-8c6d-69d002680eba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-zssjp\" (UID: \"077ee101-a706-4d61-8c6d-69d002680eba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-zssjp"
Sep 16 14:15:15.733250 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:15.733128    2587 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 14:15:15.733250 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:15.733190    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/077ee101-a706-4d61-8c6d-69d002680eba-samples-operator-tls podName:077ee101-a706-4d61-8c6d-69d002680eba nodeName:}" failed. No retries permitted until 2026-09-16 14:15:23.73317346 +0000 UTC m=+72.336798475 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/077ee101-a706-4d61-8c6d-69d002680eba-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-zssjp" (UID: "077ee101-a706-4d61-8c6d-69d002680eba") : secret "samples-operator-tls" not found
Sep 16 14:15:15.907407 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:15.907353    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-rr2fz"]
Sep 16 14:15:15.911241 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:15.911224    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-rr2fz"
Sep 16 14:15:15.914055 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:15.914026    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5bxrv\""
Sep 16 14:15:15.915127 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:15.915107    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 16 14:15:15.915271 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:15.915152    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-rr2fz"]
Sep 16 14:15:15.915271 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:15.915245    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 16 14:15:15.915382 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:15.915279    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 16 14:15:15.915382 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:15.915340    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 16 14:15:15.934649 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:15.934619    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c25d2ef2-c0e5-4480-a626-76c99f186e7a-signing-cabundle\") pod \"service-ca-568869fdc5-rr2fz\" (UID: \"c25d2ef2-c0e5-4480-a626-76c99f186e7a\") " pod="openshift-service-ca/service-ca-568869fdc5-rr2fz"
Sep 16 14:15:15.934649 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:15.934650    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bskjj\" (UniqueName: \"kubernetes.io/projected/c25d2ef2-c0e5-4480-a626-76c99f186e7a-kube-api-access-bskjj\") pod \"service-ca-568869fdc5-rr2fz\" (UID: \"c25d2ef2-c0e5-4480-a626-76c99f186e7a\") " pod="openshift-service-ca/service-ca-568869fdc5-rr2fz"
Sep 16 14:15:15.934866 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:15.934798    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c25d2ef2-c0e5-4480-a626-76c99f186e7a-signing-key\") pod \"service-ca-568869fdc5-rr2fz\" (UID: \"c25d2ef2-c0e5-4480-a626-76c99f186e7a\") " pod="openshift-service-ca/service-ca-568869fdc5-rr2fz"
Sep 16 14:15:16.035583 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:16.035498    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c25d2ef2-c0e5-4480-a626-76c99f186e7a-signing-key\") pod \"service-ca-568869fdc5-rr2fz\" (UID: \"c25d2ef2-c0e5-4480-a626-76c99f186e7a\") " pod="openshift-service-ca/service-ca-568869fdc5-rr2fz"
Sep 16 14:15:16.035583 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:16.035568    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c25d2ef2-c0e5-4480-a626-76c99f186e7a-signing-cabundle\") pod \"service-ca-568869fdc5-rr2fz\" (UID: \"c25d2ef2-c0e5-4480-a626-76c99f186e7a\") " pod="openshift-service-ca/service-ca-568869fdc5-rr2fz"
Sep 16 14:15:16.035814 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:16.035588    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bskjj\" (UniqueName: \"kubernetes.io/projected/c25d2ef2-c0e5-4480-a626-76c99f186e7a-kube-api-access-bskjj\") pod \"service-ca-568869fdc5-rr2fz\" (UID: \"c25d2ef2-c0e5-4480-a626-76c99f186e7a\") " pod="openshift-service-ca/service-ca-568869fdc5-rr2fz"
Sep 16 14:15:16.036271 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:16.036247    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c25d2ef2-c0e5-4480-a626-76c99f186e7a-signing-cabundle\") pod \"service-ca-568869fdc5-rr2fz\" (UID: \"c25d2ef2-c0e5-4480-a626-76c99f186e7a\") " pod="openshift-service-ca/service-ca-568869fdc5-rr2fz"
Sep 16 14:15:16.038109 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:16.038085    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c25d2ef2-c0e5-4480-a626-76c99f186e7a-signing-key\") pod \"service-ca-568869fdc5-rr2fz\" (UID: \"c25d2ef2-c0e5-4480-a626-76c99f186e7a\") " pod="openshift-service-ca/service-ca-568869fdc5-rr2fz"
Sep 16 14:15:16.040045 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:16.040025    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v2zl4_bd778876-4a84-403c-b6a7-bfa346cfe0d5/serve-healthcheck-canary/0.log"
Sep 16 14:15:16.047324 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:16.047302    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bskjj\" (UniqueName: \"kubernetes.io/projected/c25d2ef2-c0e5-4480-a626-76c99f186e7a-kube-api-access-bskjj\") pod \"service-ca-568869fdc5-rr2fz\" (UID: \"c25d2ef2-c0e5-4480-a626-76c99f186e7a\") " pod="openshift-service-ca/service-ca-568869fdc5-rr2fz"
Sep 16 14:15:16.220873 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:16.220830    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-rr2fz"
Sep 16 14:15:16.509019 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:16.508982    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-rr2fz"]
Sep 16 14:15:16.511760 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:15:16.511733    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc25d2ef2_c0e5_4480_a626_76c99f186e7a.slice/crio-14aa57bb511f8801e4bfc5e2cca3af9500021e94640c07de99f60e19b1ee5414 WatchSource:0}: Error finding container 14aa57bb511f8801e4bfc5e2cca3af9500021e94640c07de99f60e19b1ee5414: Status 404 returned error can't find the container with id 14aa57bb511f8801e4bfc5e2cca3af9500021e94640c07de99f60e19b1ee5414
Sep 16 14:15:16.540596 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:16.540505    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ae7e37a-60a5-4d09-8e0f-905cad155314-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lqtmz\" (UID: \"1ae7e37a-60a5-4d09-8e0f-905cad155314\") " pod="openshift-insights/insights-runtime-extractor-lqtmz"
Sep 16 14:15:16.540952 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:16.540616    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ae7e37a-60a5-4d09-8e0f-905cad155314-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lqtmz\" (UID: \"1ae7e37a-60a5-4d09-8e0f-905cad155314\") " pod="openshift-insights/insights-runtime-extractor-lqtmz"
Sep 16 14:15:16.540952 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:16.540634    2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 14:15:16.540952 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:16.540697    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1ae7e37a-60a5-4d09-8e0f-905cad155314-kube-rbac-proxy-cm podName:1ae7e37a-60a5-4d09-8e0f-905cad155314 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:20.540681953 +0000 UTC m=+69.144306969 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1ae7e37a-60a5-4d09-8e0f-905cad155314-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lqtmz" (UID: "1ae7e37a-60a5-4d09-8e0f-905cad155314") : configmap "kube-rbac-proxy" not found
Sep 16 14:15:16.540952 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:16.540760    2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 14:15:16.540952 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:16.540810    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ae7e37a-60a5-4d09-8e0f-905cad155314-insights-runtime-extractor-tls podName:1ae7e37a-60a5-4d09-8e0f-905cad155314 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:20.540797685 +0000 UTC m=+69.144422701 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1ae7e37a-60a5-4d09-8e0f-905cad155314-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lqtmz" (UID: "1ae7e37a-60a5-4d09-8e0f-905cad155314") : secret "insights-runtime-extractor-tls" not found
Sep 16 14:15:17.183191 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:17.183161    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-rr2fz" event={"ID":"c25d2ef2-c0e5-4480-a626-76c99f186e7a","Type":"ContainerStarted","Data":"fe3b632406b6b636c75287ffe7304b19b455c0abc807b9587802c702fefd146c"}
Sep 16 14:15:17.183355 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:17.183207    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-rr2fz" event={"ID":"c25d2ef2-c0e5-4480-a626-76c99f186e7a","Type":"ContainerStarted","Data":"14aa57bb511f8801e4bfc5e2cca3af9500021e94640c07de99f60e19b1ee5414"}
Sep 16 14:15:17.210968 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:17.210817    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-rr2fz" podStartSLOduration=2.210798425 podStartE2EDuration="2.210798425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:17.210082444 +0000 UTC m=+65.813707479" watchObservedRunningTime="2026-09-16 14:15:17.210798425 +0000 UTC m=+65.814423494"
Sep 16 14:15:17.651644 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:17.651612    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfwbq\" (UniqueName: \"kubernetes.io/projected/4e774454-f78b-40ba-ae21-ed6a11caaea5-kube-api-access-hfwbq\") pod \"network-check-target-h7q7r\" (UID: \"4e774454-f78b-40ba-ae21-ed6a11caaea5\") " pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:15:17.652085 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:17.651735    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e10f563c-16f8-4699-a7d4-52e6bfba29a5-metrics-certs\") pod \"network-metrics-daemon-6qpvt\" (UID: \"e10f563c-16f8-4699-a7d4-52e6bfba29a5\") " pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:15:17.652085 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:17.651768    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f12cf92-2a09-4fe6-a73e-5692728769b3-original-pull-secret\") pod \"global-pull-secret-syncer-x59fz\" (UID: \"6f12cf92-2a09-4fe6-a73e-5692728769b3\") " pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:15:17.654894 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:17.654871    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 14:15:17.655034 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:17.654897    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 14:15:17.655034 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:17.654996    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 14:15:17.662412 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:17.662361    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 14:15:17.662543 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:17.662457    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e10f563c-16f8-4699-a7d4-52e6bfba29a5-metrics-certs podName:e10f563c-16f8-4699-a7d4-52e6bfba29a5 nodeName:}" failed. No retries permitted until 2026-09-16 14:16:21.662433683 +0000 UTC m=+130.266058707 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e10f563c-16f8-4699-a7d4-52e6bfba29a5-metrics-certs") pod "network-metrics-daemon-6qpvt" (UID: "e10f563c-16f8-4699-a7d4-52e6bfba29a5") : secret "metrics-daemon-secret" not found
Sep 16 14:15:17.664994 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:17.664971    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f12cf92-2a09-4fe6-a73e-5692728769b3-original-pull-secret\") pod \"global-pull-secret-syncer-x59fz\" (UID: \"6f12cf92-2a09-4fe6-a73e-5692728769b3\") " pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:15:17.665476 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:17.665457    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 14:15:17.675080 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:17.675045    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfwbq\" (UniqueName: \"kubernetes.io/projected/4e774454-f78b-40ba-ae21-ed6a11caaea5-kube-api-access-hfwbq\") pod \"network-check-target-h7q7r\" (UID: \"4e774454-f78b-40ba-ae21-ed6a11caaea5\") " pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:15:17.705565 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:17.705526    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x59fz"
Sep 16 14:15:17.814974 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:17.814935    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8dcj9\""
Sep 16 14:15:17.822560 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:17.822512    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:15:18.003186 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:18.003163    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x59fz"]
Sep 16 14:15:18.006429 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:15:18.005969    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f12cf92_2a09_4fe6_a73e_5692728769b3.slice/crio-97ffbe4e88155cb73ca2e08317176c5c056ea66ae1ed0960b593ed2ed804ad19 WatchSource:0}: Error finding container 97ffbe4e88155cb73ca2e08317176c5c056ea66ae1ed0960b593ed2ed804ad19: Status 404 returned error can't find the container with id 97ffbe4e88155cb73ca2e08317176c5c056ea66ae1ed0960b593ed2ed804ad19
Sep 16 14:15:18.141407 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:18.141353    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h7q7r"]
Sep 16 14:15:18.146051 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:15:18.146015    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e774454_f78b_40ba_ae21_ed6a11caaea5.slice/crio-9b614b6ff3d8e1ecc0693813fad6dd37d2ca49519370e42f008906d54fd812de WatchSource:0}: Error finding container 9b614b6ff3d8e1ecc0693813fad6dd37d2ca49519370e42f008906d54fd812de: Status 404 returned error can't find the container with id 9b614b6ff3d8e1ecc0693813fad6dd37d2ca49519370e42f008906d54fd812de
Sep 16 14:15:18.187004 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:18.186966    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h7q7r" event={"ID":"4e774454-f78b-40ba-ae21-ed6a11caaea5","Type":"ContainerStarted","Data":"9b614b6ff3d8e1ecc0693813fad6dd37d2ca49519370e42f008906d54fd812de"}
Sep 16 14:15:18.188288 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:18.188263    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x59fz" event={"ID":"6f12cf92-2a09-4fe6-a73e-5692728769b3","Type":"ContainerStarted","Data":"97ffbe4e88155cb73ca2e08317176c5c056ea66ae1ed0960b593ed2ed804ad19"}
Sep 16 14:15:20.580172 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:20.580137    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ae7e37a-60a5-4d09-8e0f-905cad155314-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lqtmz\" (UID: \"1ae7e37a-60a5-4d09-8e0f-905cad155314\") " pod="openshift-insights/insights-runtime-extractor-lqtmz"
Sep 16 14:15:20.580172 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:20.580175    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-metrics-tls\") pod \"dns-default-4ntk2\" (UID: \"38e73538-4f3c-4d56-bc47-e9bfb8525e0a\") " pod="openshift-dns/dns-default-4ntk2"
Sep 16 14:15:20.580800 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:20.580264    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ae7e37a-60a5-4d09-8e0f-905cad155314-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lqtmz\" (UID: \"1ae7e37a-60a5-4d09-8e0f-905cad155314\") " pod="openshift-insights/insights-runtime-extractor-lqtmz"
Sep 16 14:15:20.580800 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:20.580314    2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 14:15:20.580800 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:20.580352    2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 14:15:20.580800 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:20.580373    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 14:15:20.580800 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:20.580421    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ae7e37a-60a5-4d09-8e0f-905cad155314-insights-runtime-extractor-tls podName:1ae7e37a-60a5-4d09-8e0f-905cad155314 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:28.580375887 +0000 UTC m=+77.184000903 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1ae7e37a-60a5-4d09-8e0f-905cad155314-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lqtmz" (UID: "1ae7e37a-60a5-4d09-8e0f-905cad155314") : secret "insights-runtime-extractor-tls" not found
Sep 16 14:15:20.580800 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:20.580441    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1ae7e37a-60a5-4d09-8e0f-905cad155314-kube-rbac-proxy-cm podName:1ae7e37a-60a5-4d09-8e0f-905cad155314 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:28.580431013 +0000 UTC m=+77.184056026 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1ae7e37a-60a5-4d09-8e0f-905cad155314-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lqtmz" (UID: "1ae7e37a-60a5-4d09-8e0f-905cad155314") : configmap "kube-rbac-proxy" not found
Sep 16 14:15:20.580800 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:20.580456    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-metrics-tls podName:38e73538-4f3c-4d56-bc47-e9bfb8525e0a nodeName:}" failed. No retries permitted until 2026-09-16 14:15:52.580447699 +0000 UTC m=+101.184072709 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-metrics-tls") pod "dns-default-4ntk2" (UID: "38e73538-4f3c-4d56-bc47-e9bfb8525e0a") : secret "dns-default-metrics-tls" not found
Sep 16 14:15:23.609622 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:23.609532    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75e1ae3b-e19a-47f1-9395-c18e8e393062-service-ca-bundle\") pod \"router-default-6f487956db-zccck\" (UID: \"75e1ae3b-e19a-47f1-9395-c18e8e393062\") " pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:23.609974 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:23.609634    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e1ae3b-e19a-47f1-9395-c18e8e393062-metrics-certs\") pod \"router-default-6f487956db-zccck\" (UID: \"75e1ae3b-e19a-47f1-9395-c18e8e393062\") " pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:23.610156 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:23.610124    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75e1ae3b-e19a-47f1-9395-c18e8e393062-service-ca-bundle\") pod \"router-default-6f487956db-zccck\" (UID: \"75e1ae3b-e19a-47f1-9395-c18e8e393062\") " pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:23.612142 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:23.612113    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e1ae3b-e19a-47f1-9395-c18e8e393062-metrics-certs\") pod \"router-default-6f487956db-zccck\" (UID: \"75e1ae3b-e19a-47f1-9395-c18e8e393062\") " pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:23.710478 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:23.710428    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-registry-tls\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:23.713029 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:23.713006    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-registry-tls\") pod \"image-registry-7f74cc64bb-gpqp4\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") " pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:23.740981 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:23.740946    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2gx92\""
Sep 16 14:15:23.749276 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:23.749258    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:23.811374 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:23.811342    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/077ee101-a706-4d61-8c6d-69d002680eba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-zssjp\" (UID: \"077ee101-a706-4d61-8c6d-69d002680eba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-zssjp"
Sep 16 14:15:23.815015 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:23.814949    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/077ee101-a706-4d61-8c6d-69d002680eba-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-zssjp\" (UID: \"077ee101-a706-4d61-8c6d-69d002680eba\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-zssjp"
Sep 16 14:15:23.861981 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:23.861906    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-566s2\""
Sep 16 14:15:23.869375 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:23.869350    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:23.929189 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:23.929161    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-rpkck\""
Sep 16 14:15:23.936371 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:23.936306    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-zssjp"
Sep 16 14:15:24.025435 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:24.025411    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6f487956db-zccck"]
Sep 16 14:15:24.028209 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:15:24.028179    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e1ae3b_e19a_47f1_9395_c18e8e393062.slice/crio-ca33f6006133e69c1d5df1fc36fe33dbfd59f881fd97722298a5c1f94e19525c WatchSource:0}: Error finding container ca33f6006133e69c1d5df1fc36fe33dbfd59f881fd97722298a5c1f94e19525c: Status 404 returned error can't find the container with id ca33f6006133e69c1d5df1fc36fe33dbfd59f881fd97722298a5c1f94e19525c
Sep 16 14:15:24.148615 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:24.146370    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f74cc64bb-gpqp4"]
Sep 16 14:15:24.151328 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:15:24.151296    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c16289f_192f_4eda_a7e5_3c8aef6f14e6.slice/crio-eb1c32803edbece1d3f0961079d2075c71b434f77a2d147521b8c0184e31b733 WatchSource:0}: Error finding container eb1c32803edbece1d3f0961079d2075c71b434f77a2d147521b8c0184e31b733: Status 404 returned error can't find the container with id eb1c32803edbece1d3f0961079d2075c71b434f77a2d147521b8c0184e31b733
Sep 16 14:15:24.208630 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:24.208604    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-zssjp"]
Sep 16 14:15:24.209418 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:24.209323    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h7q7r" event={"ID":"4e774454-f78b-40ba-ae21-ed6a11caaea5","Type":"ContainerStarted","Data":"df7da01fdc0bacbb59dd192a249e6609a748fcedc1c51d310316cdf002f20334"}
Sep 16 14:15:24.209510 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:24.209442    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:15:24.211167 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:24.211144    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x59fz" event={"ID":"6f12cf92-2a09-4fe6-a73e-5692728769b3","Type":"ContainerStarted","Data":"cb7961704faf49805e38ad0f17e29dbe4ae635ee8723e2d1888eb501e1fc9e66"}
Sep 16 14:15:24.213881 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:24.213861    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4" event={"ID":"8c16289f-192f-4eda-a7e5-3c8aef6f14e6","Type":"ContainerStarted","Data":"ef393ea5dd9fb91e7ae09653537b1d78359754dc0c0b0a90641b79a29a11b5a7"}
Sep 16 14:15:24.213998 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:24.213902    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4" event={"ID":"8c16289f-192f-4eda-a7e5-3c8aef6f14e6","Type":"ContainerStarted","Data":"eb1c32803edbece1d3f0961079d2075c71b434f77a2d147521b8c0184e31b733"}
Sep 16 14:15:24.213998 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:24.213964    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:24.215817 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:24.215794    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6f487956db-zccck" event={"ID":"75e1ae3b-e19a-47f1-9395-c18e8e393062","Type":"ContainerStarted","Data":"25b1a9ac46d6e7f93095415fd3c73828ac0543a5dcfce268feb7a11abb6a0179"}
Sep 16 14:15:24.215895 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:24.215838    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6f487956db-zccck" event={"ID":"75e1ae3b-e19a-47f1-9395-c18e8e393062","Type":"ContainerStarted","Data":"ca33f6006133e69c1d5df1fc36fe33dbfd59f881fd97722298a5c1f94e19525c"}
Sep 16 14:15:24.227170 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:24.227113    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-h7q7r" podStartSLOduration=67.090006673 podStartE2EDuration="1m12.227089867s" totalImagesPullingTime="5.137083194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:24.226019308 +0000 UTC m=+72.829644341" watchObservedRunningTime="2026-09-16 14:15:24.227089867 +0000 UTC m=+72.830714899"
Sep 16 14:15:24.249349 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:24.249293    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6f487956db-zccck" podStartSLOduration=17.249273729 podStartE2EDuration="17.249273729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:24.249066393 +0000 UTC m=+72.852691426" watchObservedRunningTime="2026-09-16 14:15:24.249273729 +0000 UTC m=+72.852898762"
Sep 16 14:15:24.273686 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:24.273630    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4" podStartSLOduration=17.273608291 podStartE2EDuration="17.273608291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:24.272518519 +0000 UTC m=+72.876143551" watchObservedRunningTime="2026-09-16 14:15:24.273608291 +0000 UTC m=+72.877233322"
Sep 16 14:15:24.750466 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:24.750428    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:24.753238 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:24.753213    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:24.773201 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:24.773150    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-x59fz" podStartSLOduration=67.494907661 podStartE2EDuration="1m12.773133263s" totalImagesPullingTime="5.278225602s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:24.290555265 +0000 UTC m=+72.894180298" watchObservedRunningTime="2026-09-16 14:15:24.773133263 +0000 UTC m=+73.376758294"
Sep 16 14:15:25.220279 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:25.220244    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-zssjp" event={"ID":"077ee101-a706-4d61-8c6d-69d002680eba","Type":"ContainerStarted","Data":"8a3e6a9c74e0019bf2abe5a49169569a0af6c10c402077b322e424e862bb28ae"}
Sep 16 14:15:25.220489 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:25.220305    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:25.221832 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:25.221807    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6f487956db-zccck"
Sep 16 14:15:27.231257 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:27.231209    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-zssjp" event={"ID":"077ee101-a706-4d61-8c6d-69d002680eba","Type":"ContainerStarted","Data":"021f964bd4a78b8cd411f24f89d02dce0f08aeb8e3059ddbebd31ec49439d41b"}
Sep 16 14:15:27.231734 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:27.231306    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-zssjp" event={"ID":"077ee101-a706-4d61-8c6d-69d002680eba","Type":"ContainerStarted","Data":"e170513d338ad15fe280046d7f56dcb9d4a9782103c034d5bc8924553ce54dec"}
Sep 16 14:15:27.250990 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:27.250944    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-zssjp" podStartSLOduration=18.340138816 podStartE2EDuration="20.250930254s" totalImagesPullingTime="1.910791438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:27.249745873 +0000 UTC m=+75.853370905" watchObservedRunningTime="2026-09-16 14:15:27.250930254 +0000 UTC m=+75.854555341"
Sep 16 14:15:28.654599 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:28.654560    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ae7e37a-60a5-4d09-8e0f-905cad155314-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lqtmz\" (UID: \"1ae7e37a-60a5-4d09-8e0f-905cad155314\") " pod="openshift-insights/insights-runtime-extractor-lqtmz"
Sep 16 14:15:28.654975 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:28.654641    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ae7e37a-60a5-4d09-8e0f-905cad155314-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lqtmz\" (UID: \"1ae7e37a-60a5-4d09-8e0f-905cad155314\") " pod="openshift-insights/insights-runtime-extractor-lqtmz"
Sep 16 14:15:28.655130 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:28.655112    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ae7e37a-60a5-4d09-8e0f-905cad155314-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lqtmz\" (UID: \"1ae7e37a-60a5-4d09-8e0f-905cad155314\") " pod="openshift-insights/insights-runtime-extractor-lqtmz"
Sep 16 14:15:28.656970 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:28.656951    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ae7e37a-60a5-4d09-8e0f-905cad155314-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lqtmz\" (UID: \"1ae7e37a-60a5-4d09-8e0f-905cad155314\") " pod="openshift-insights/insights-runtime-extractor-lqtmz"
Sep 16 14:15:28.799330 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:28.799297    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lqtmz"
Sep 16 14:15:29.072436 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:29.072226    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lqtmz"]
Sep 16 14:15:29.074884 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:15:29.074858    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae7e37a_60a5_4d09_8e0f_905cad155314.slice/crio-83c3b006f1794751f9d4d98364dbcebfc4ff697716479544901b5e3a7286ae47 WatchSource:0}: Error finding container 83c3b006f1794751f9d4d98364dbcebfc4ff697716479544901b5e3a7286ae47: Status 404 returned error can't find the container with id 83c3b006f1794751f9d4d98364dbcebfc4ff697716479544901b5e3a7286ae47
Sep 16 14:15:29.238913 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:29.238877    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lqtmz" event={"ID":"1ae7e37a-60a5-4d09-8e0f-905cad155314","Type":"ContainerStarted","Data":"23b346ef37a1e1c3234dc437a779c26def898598bc5f56c129a7cf41fd736cf0"}
Sep 16 14:15:29.239063 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:29.238933    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lqtmz" event={"ID":"1ae7e37a-60a5-4d09-8e0f-905cad155314","Type":"ContainerStarted","Data":"83c3b006f1794751f9d4d98364dbcebfc4ff697716479544901b5e3a7286ae47"}
Sep 16 14:15:30.244081 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:30.244051    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lqtmz" event={"ID":"1ae7e37a-60a5-4d09-8e0f-905cad155314","Type":"ContainerStarted","Data":"dbe0dfb29893361268bcaa314b465071582e59e278b2f3ab8ad75462e3069d1c"}
Sep 16 14:15:32.255536 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:32.255499    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lqtmz" event={"ID":"1ae7e37a-60a5-4d09-8e0f-905cad155314","Type":"ContainerStarted","Data":"1f4fb9d6953746563524cb101251c77ae38abde64715e8de4ae6fc5c28b53c6d"}
Sep 16 14:15:32.273822 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:32.273770    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lqtmz" podStartSLOduration=17.876255069 podStartE2EDuration="20.273754209s" totalImagesPullingTime="2.39749914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:15:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:32.272881709 +0000 UTC m=+80.876506740" watchObservedRunningTime="2026-09-16 14:15:32.273754209 +0000 UTC m=+80.877379242"
Sep 16 14:15:35.704449 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:35.704308    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f74cc64bb-gpqp4"]
Sep 16 14:15:35.910505 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:35.910475    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f45d8ddc4-2zl52"]
Sep 16 14:15:35.914701 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:35.914684    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52"
Sep 16 14:15:35.934149 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:35.934121    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f45d8ddc4-2zl52"]
Sep 16 14:15:36.016156 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:36.016065    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39e08d27-3153-4a39-9ecd-7d162f1965aa-bound-sa-token\") pod \"image-registry-f45d8ddc4-2zl52\" (UID: \"39e08d27-3153-4a39-9ecd-7d162f1965aa\") " pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52"
Sep 16 14:15:36.016156 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:36.016103    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39e08d27-3153-4a39-9ecd-7d162f1965aa-image-registry-private-configuration\") pod \"image-registry-f45d8ddc4-2zl52\" (UID: \"39e08d27-3153-4a39-9ecd-7d162f1965aa\") " pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52"
Sep 16 14:15:36.016156 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:36.016128    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39e08d27-3153-4a39-9ecd-7d162f1965aa-installation-pull-secrets\") pod \"image-registry-f45d8ddc4-2zl52\" (UID: \"39e08d27-3153-4a39-9ecd-7d162f1965aa\") " pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52"
Sep 16 14:15:36.016466 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:36.016227    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39e08d27-3153-4a39-9ecd-7d162f1965aa-ca-trust-extracted\") pod \"image-registry-f45d8ddc4-2zl52\" (UID: \"39e08d27-3153-4a39-9ecd-7d162f1965aa\") " pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52"
Sep 16 14:15:36.016466 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:36.016297    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39e08d27-3153-4a39-9ecd-7d162f1965aa-registry-tls\") pod \"image-registry-f45d8ddc4-2zl52\" (UID: \"39e08d27-3153-4a39-9ecd-7d162f1965aa\") " pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52"
Sep 16 14:15:36.016466 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:36.016340    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39e08d27-3153-4a39-9ecd-7d162f1965aa-trusted-ca\") pod \"image-registry-f45d8ddc4-2zl52\" (UID: \"39e08d27-3153-4a39-9ecd-7d162f1965aa\") " pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52"
Sep 16 14:15:36.016466 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:36.016361    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptxh5\" (UniqueName: \"kubernetes.io/projected/39e08d27-3153-4a39-9ecd-7d162f1965aa-kube-api-access-ptxh5\") pod \"image-registry-f45d8ddc4-2zl52\" (UID: \"39e08d27-3153-4a39-9ecd-7d162f1965aa\") " pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52"
Sep 16 14:15:36.016466 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:36.016428    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39e08d27-3153-4a39-9ecd-7d162f1965aa-registry-certificates\") pod \"image-registry-f45d8ddc4-2zl52\" (UID: \"39e08d27-3153-4a39-9ecd-7d162f1965aa\") " pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52"
Sep 16 14:15:36.116776 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:36.116743    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39e08d27-3153-4a39-9ecd-7d162f1965aa-installation-pull-secrets\") pod \"image-registry-f45d8ddc4-2zl52\" (UID: \"39e08d27-3153-4a39-9ecd-7d162f1965aa\") " pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52"
Sep 16 14:15:36.116776 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:36.116775    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39e08d27-3153-4a39-9ecd-7d162f1965aa-ca-trust-extracted\") pod \"image-registry-f45d8ddc4-2zl52\" (UID: \"39e08d27-3153-4a39-9ecd-7d162f1965aa\") " pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52"
Sep 16 14:15:36.117017 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:36.116809    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39e08d27-3153-4a39-9ecd-7d162f1965aa-registry-tls\") pod \"image-registry-f45d8ddc4-2zl52\" (UID: \"39e08d27-3153-4a39-9ecd-7d162f1965aa\") " pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52"
Sep 16 14:15:36.117017 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:36.116837    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39e08d27-3153-4a39-9ecd-7d162f1965aa-trusted-ca\") pod \"image-registry-f45d8ddc4-2zl52\" (UID: \"39e08d27-3153-4a39-9ecd-7d162f1965aa\") " pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52"
Sep 16 14:15:36.117017 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:36.116851    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptxh5\" (UniqueName: \"kubernetes.io/projected/39e08d27-3153-4a39-9ecd-7d162f1965aa-kube-api-access-ptxh5\") pod \"image-registry-f45d8ddc4-2zl52\" (UID: \"39e08d27-3153-4a39-9ecd-7d162f1965aa\") " pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52"
Sep 16 14:15:36.117017 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:36.116875    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39e08d27-3153-4a39-9ecd-7d162f1965aa-registry-certificates\") pod \"image-registry-f45d8ddc4-2zl52\" (UID: \"39e08d27-3153-4a39-9ecd-7d162f1965aa\") " pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52"
Sep 16 14:15:36.117017 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:36.116905    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39e08d27-3153-4a39-9ecd-7d162f1965aa-bound-sa-token\") pod \"image-registry-f45d8ddc4-2zl52\" (UID: \"39e08d27-3153-4a39-9ecd-7d162f1965aa\") " pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52"
Sep 16 14:15:36.117017 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:36.116940    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39e08d27-3153-4a39-9ecd-7d162f1965aa-image-registry-private-configuration\") pod \"image-registry-f45d8ddc4-2zl52\" (UID: \"39e08d27-3153-4a39-9ecd-7d162f1965aa\") " pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52"
Sep 16 14:15:36.117308 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:36.117250    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39e08d27-3153-4a39-9ecd-7d162f1965aa-ca-trust-extracted\") pod \"image-registry-f45d8ddc4-2zl52\" (UID: \"39e08d27-3153-4a39-9ecd-7d162f1965aa\") " pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52"
Sep 16 14:15:36.118004 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:36.117979    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39e08d27-3153-4a39-9ecd-7d162f1965aa-registry-certificates\") pod \"image-registry-f45d8ddc4-2zl52\" (UID: \"39e08d27-3153-4a39-9ecd-7d162f1965aa\") " pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52"
Sep 16 14:15:36.118099 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:36.117986    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39e08d27-3153-4a39-9ecd-7d162f1965aa-trusted-ca\") pod \"image-registry-f45d8ddc4-2zl52\" (UID: \"39e08d27-3153-4a39-9ecd-7d162f1965aa\") " pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52"
Sep 16 14:15:36.119584 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:36.119542    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39e08d27-3153-4a39-9ecd-7d162f1965aa-installation-pull-secrets\") pod \"image-registry-f45d8ddc4-2zl52\" (UID: \"39e08d27-3153-4a39-9ecd-7d162f1965aa\") " pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52"
Sep 16 14:15:36.120083 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:36.120055    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39e08d27-3153-4a39-9ecd-7d162f1965aa-registry-tls\") pod \"image-registry-f45d8ddc4-2zl52\" (UID: \"39e08d27-3153-4a39-9ecd-7d162f1965aa\") " pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52"
Sep 16 14:15:36.120083 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:36.120071    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39e08d27-3153-4a39-9ecd-7d162f1965aa-image-registry-private-configuration\") pod \"image-registry-f45d8ddc4-2zl52\" (UID: \"39e08d27-3153-4a39-9ecd-7d162f1965aa\") " pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52"
Sep 16 14:15:36.130997 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:36.130969    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39e08d27-3153-4a39-9ecd-7d162f1965aa-bound-sa-token\") pod \"image-registry-f45d8ddc4-2zl52\" (UID: \"39e08d27-3153-4a39-9ecd-7d162f1965aa\") " pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52"
Sep 16 14:15:36.131206 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:36.131183    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptxh5\" (UniqueName: \"kubernetes.io/projected/39e08d27-3153-4a39-9ecd-7d162f1965aa-kube-api-access-ptxh5\") pod \"image-registry-f45d8ddc4-2zl52\" (UID: \"39e08d27-3153-4a39-9ecd-7d162f1965aa\") " pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52"
Sep 16 14:15:36.227857 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:36.227823    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52"
Sep 16 14:15:36.505469 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:36.505446    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f45d8ddc4-2zl52"]
Sep 16 14:15:36.507706 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:15:36.507673    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e08d27_3153_4a39_9ecd_7d162f1965aa.slice/crio-3bcebb9aa23d2b0057a3d5538ae73e332499950b64bf7c681f92d96bf7aff278 WatchSource:0}: Error finding container 3bcebb9aa23d2b0057a3d5538ae73e332499950b64bf7c681f92d96bf7aff278: Status 404 returned error can't find the container with id 3bcebb9aa23d2b0057a3d5538ae73e332499950b64bf7c681f92d96bf7aff278
Sep 16 14:15:37.275809 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:37.275774    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52" event={"ID":"39e08d27-3153-4a39-9ecd-7d162f1965aa","Type":"ContainerStarted","Data":"3e2d64ca84382eb349a28b707272ab628476ced577b6cf56239b96118f3d9cc4"}
Sep 16 14:15:37.276196 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:37.275825    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52" event={"ID":"39e08d27-3153-4a39-9ecd-7d162f1965aa","Type":"ContainerStarted","Data":"3bcebb9aa23d2b0057a3d5538ae73e332499950b64bf7c681f92d96bf7aff278"}
Sep 16 14:15:37.276196 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:37.275892    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52"
Sep 16 14:15:37.297873 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:37.297815    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52" podStartSLOduration=2.297797251 podStartE2EDuration="2.297797251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:37.297194209 +0000 UTC m=+85.900819438" watchObservedRunningTime="2026-09-16 14:15:37.297797251 +0000 UTC m=+85.901422282"
Sep 16 14:15:45.711551 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:45.711519    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:15:49.679199 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:49.679165    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-mp9z7"]
Sep 16 14:15:49.682652 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:49.682630    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-mp9z7"
Sep 16 14:15:49.685257 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:49.685231    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 14:15:49.685453 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:49.685258    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 14:15:49.685453 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:49.685325    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 14:15:49.685453 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:49.685343    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 14:15:49.686291 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:49.686275    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-jlfwc\""
Sep 16 14:15:49.686411 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:49.686275    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 14:15:49.694202 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:49.694184    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-mp9z7"]
Sep 16 14:15:49.729671 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:49.729636    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6a259dcd-592a-4579-9180-60f6dc5eac36-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-mp9z7\" (UID: \"6a259dcd-592a-4579-9180-60f6dc5eac36\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mp9z7"
Sep 16 14:15:49.729831 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:49.729675    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk8xv\" (UniqueName: \"kubernetes.io/projected/6a259dcd-592a-4579-9180-60f6dc5eac36-kube-api-access-dk8xv\") pod \"prometheus-operator-68bcd7f987-mp9z7\" (UID: \"6a259dcd-592a-4579-9180-60f6dc5eac36\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mp9z7"
Sep 16 14:15:49.729831 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:49.729697    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a259dcd-592a-4579-9180-60f6dc5eac36-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-mp9z7\" (UID: \"6a259dcd-592a-4579-9180-60f6dc5eac36\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mp9z7"
Sep 16 14:15:49.729831 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:49.729775    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a259dcd-592a-4579-9180-60f6dc5eac36-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-mp9z7\" (UID: \"6a259dcd-592a-4579-9180-60f6dc5eac36\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mp9z7"
Sep 16 14:15:49.831104 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:49.831065    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6a259dcd-592a-4579-9180-60f6dc5eac36-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-mp9z7\" (UID: \"6a259dcd-592a-4579-9180-60f6dc5eac36\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mp9z7"
Sep 16 14:15:49.831104 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:49.831103    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk8xv\" (UniqueName: \"kubernetes.io/projected/6a259dcd-592a-4579-9180-60f6dc5eac36-kube-api-access-dk8xv\") pod \"prometheus-operator-68bcd7f987-mp9z7\" (UID: \"6a259dcd-592a-4579-9180-60f6dc5eac36\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mp9z7"
Sep 16 14:15:49.831348 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:49.831120    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a259dcd-592a-4579-9180-60f6dc5eac36-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-mp9z7\" (UID: \"6a259dcd-592a-4579-9180-60f6dc5eac36\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mp9z7"
Sep 16 14:15:49.831348 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:49.831168    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a259dcd-592a-4579-9180-60f6dc5eac36-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-mp9z7\" (UID: \"6a259dcd-592a-4579-9180-60f6dc5eac36\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mp9z7"
Sep 16 14:15:49.831348 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:49.831248    2587 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 16 14:15:49.831348 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:15:49.831330    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a259dcd-592a-4579-9180-60f6dc5eac36-prometheus-operator-tls podName:6a259dcd-592a-4579-9180-60f6dc5eac36 nodeName:}" failed. No retries permitted until 2026-09-16 14:15:50.331308528 +0000 UTC m=+98.934933554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/6a259dcd-592a-4579-9180-60f6dc5eac36-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-mp9z7" (UID: "6a259dcd-592a-4579-9180-60f6dc5eac36") : secret "prometheus-operator-tls" not found
Sep 16 14:15:49.831951 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:49.831924    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a259dcd-592a-4579-9180-60f6dc5eac36-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-mp9z7\" (UID: \"6a259dcd-592a-4579-9180-60f6dc5eac36\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mp9z7"
Sep 16 14:15:49.833819 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:49.833802    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a259dcd-592a-4579-9180-60f6dc5eac36-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-mp9z7\" (UID: \"6a259dcd-592a-4579-9180-60f6dc5eac36\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mp9z7"
Sep 16 14:15:49.840638 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:49.840611    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk8xv\" (UniqueName: \"kubernetes.io/projected/6a259dcd-592a-4579-9180-60f6dc5eac36-kube-api-access-dk8xv\") pod \"prometheus-operator-68bcd7f987-mp9z7\" (UID: \"6a259dcd-592a-4579-9180-60f6dc5eac36\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mp9z7"
Sep 16 14:15:50.298003 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.297968    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-666f668987-s624v"]
Sep 16 14:15:50.300479 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.300455    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-666f668987-s624v"
Sep 16 14:15:50.303124 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.303099    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 14:15:50.304109 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.304088    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-gqfcf\""
Sep 16 14:15:50.304245 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.304131    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 14:15:50.304245 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.304226    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 14:15:50.304352 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.304231    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 14:15:50.304352 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.304230    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 14:15:50.304470 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.304454    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 14:15:50.304525 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.304513    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 14:15:50.308583 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.308564    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 14:15:50.311195 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.311175    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-666f668987-s624v"]
Sep 16 14:15:50.335251 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.335221    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0db79e86-e959-4025-a1cb-4c54d27e2d2f-console-config\") pod \"console-666f668987-s624v\" (UID: \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\") " pod="openshift-console/console-666f668987-s624v"
Sep 16 14:15:50.335251 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.335248    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0db79e86-e959-4025-a1cb-4c54d27e2d2f-service-ca\") pod \"console-666f668987-s624v\" (UID: \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\") " pod="openshift-console/console-666f668987-s624v"
Sep 16 14:15:50.335511 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.335265    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0db79e86-e959-4025-a1cb-4c54d27e2d2f-trusted-ca-bundle\") pod \"console-666f668987-s624v\" (UID: \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\") " pod="openshift-console/console-666f668987-s624v"
Sep 16 14:15:50.335511 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.335288    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltl9g\" (UniqueName: \"kubernetes.io/projected/0db79e86-e959-4025-a1cb-4c54d27e2d2f-kube-api-access-ltl9g\") pod \"console-666f668987-s624v\" (UID: \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\") " pod="openshift-console/console-666f668987-s624v"
Sep 16 14:15:50.335511 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.335360    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0db79e86-e959-4025-a1cb-4c54d27e2d2f-oauth-serving-cert\") pod \"console-666f668987-s624v\" (UID: \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\") " pod="openshift-console/console-666f668987-s624v"
Sep 16 14:15:50.335511 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.335464    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0db79e86-e959-4025-a1cb-4c54d27e2d2f-console-serving-cert\") pod \"console-666f668987-s624v\" (UID: \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\") " pod="openshift-console/console-666f668987-s624v"
Sep 16 14:15:50.335511 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.335497    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0db79e86-e959-4025-a1cb-4c54d27e2d2f-console-oauth-config\") pod \"console-666f668987-s624v\" (UID: \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\") " pod="openshift-console/console-666f668987-s624v"
Sep 16 14:15:50.335672 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.335535    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6a259dcd-592a-4579-9180-60f6dc5eac36-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-mp9z7\" (UID: \"6a259dcd-592a-4579-9180-60f6dc5eac36\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mp9z7"
Sep 16 14:15:50.338065 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.338039    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6a259dcd-592a-4579-9180-60f6dc5eac36-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-mp9z7\" (UID: \"6a259dcd-592a-4579-9180-60f6dc5eac36\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mp9z7"
Sep 16 14:15:50.436700 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.436652    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0db79e86-e959-4025-a1cb-4c54d27e2d2f-console-config\") pod \"console-666f668987-s624v\" (UID: \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\") " pod="openshift-console/console-666f668987-s624v"
Sep 16 14:15:50.436700 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.436703    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0db79e86-e959-4025-a1cb-4c54d27e2d2f-service-ca\") pod \"console-666f668987-s624v\" (UID: \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\") " pod="openshift-console/console-666f668987-s624v"
Sep 16 14:15:50.436900 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.436721    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0db79e86-e959-4025-a1cb-4c54d27e2d2f-trusted-ca-bundle\") pod \"console-666f668987-s624v\" (UID: \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\") " pod="openshift-console/console-666f668987-s624v"
Sep 16 14:15:50.436900 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.436747    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltl9g\" (UniqueName: \"kubernetes.io/projected/0db79e86-e959-4025-a1cb-4c54d27e2d2f-kube-api-access-ltl9g\") pod \"console-666f668987-s624v\" (UID: \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\") " pod="openshift-console/console-666f668987-s624v"
Sep 16 14:15:50.437194 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.437169    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0db79e86-e959-4025-a1cb-4c54d27e2d2f-oauth-serving-cert\") pod \"console-666f668987-s624v\" (UID: \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\") " pod="openshift-console/console-666f668987-s624v"
Sep 16 14:15:50.437293 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.437258    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0db79e86-e959-4025-a1cb-4c54d27e2d2f-console-serving-cert\") pod \"console-666f668987-s624v\" (UID: \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\") " pod="openshift-console/console-666f668987-s624v"
Sep 16 14:15:50.437354 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.437301    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0db79e86-e959-4025-a1cb-4c54d27e2d2f-console-oauth-config\") pod \"console-666f668987-s624v\" (UID: \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\") " pod="openshift-console/console-666f668987-s624v"
Sep 16 14:15:50.437792 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.437768    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0db79e86-e959-4025-a1cb-4c54d27e2d2f-service-ca\") pod \"console-666f668987-s624v\" (UID: \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\") " pod="openshift-console/console-666f668987-s624v"
Sep 16 14:15:50.438051 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.437844    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0db79e86-e959-4025-a1cb-4c54d27e2d2f-console-config\") pod \"console-666f668987-s624v\" (UID: \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\") " pod="openshift-console/console-666f668987-s624v"
Sep 16 14:15:50.438216 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.438188    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0db79e86-e959-4025-a1cb-4c54d27e2d2f-trusted-ca-bundle\") pod \"console-666f668987-s624v\" (UID: \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\") " pod="openshift-console/console-666f668987-s624v"
Sep 16 14:15:50.438784 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.438761    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0db79e86-e959-4025-a1cb-4c54d27e2d2f-oauth-serving-cert\") pod \"console-666f668987-s624v\" (UID: \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\") " pod="openshift-console/console-666f668987-s624v"
Sep 16 14:15:50.442008 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.440644    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0db79e86-e959-4025-a1cb-4c54d27e2d2f-console-serving-cert\") pod \"console-666f668987-s624v\" (UID: \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\") " pod="openshift-console/console-666f668987-s624v"
Sep 16 14:15:50.442008 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.440936    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0db79e86-e959-4025-a1cb-4c54d27e2d2f-console-oauth-config\") pod \"console-666f668987-s624v\" (UID: \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\") " pod="openshift-console/console-666f668987-s624v"
Sep 16 14:15:50.446146 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.446124    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltl9g\" (UniqueName: \"kubernetes.io/projected/0db79e86-e959-4025-a1cb-4c54d27e2d2f-kube-api-access-ltl9g\") pod \"console-666f668987-s624v\" (UID: \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\") " pod="openshift-console/console-666f668987-s624v"
Sep 16 14:15:50.591811 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.591721    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-mp9z7"
Sep 16 14:15:50.610608 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.610578    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-666f668987-s624v"
Sep 16 14:15:50.864488 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.864405    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-mp9z7"]
Sep 16 14:15:50.867554 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:15:50.867526    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a259dcd_592a_4579_9180_60f6dc5eac36.slice/crio-79914c654420d4996d233a0fbee43488ed903dc39e885705b3031ce7f29b38c0 WatchSource:0}: Error finding container 79914c654420d4996d233a0fbee43488ed903dc39e885705b3031ce7f29b38c0: Status 404 returned error can't find the container with id 79914c654420d4996d233a0fbee43488ed903dc39e885705b3031ce7f29b38c0
Sep 16 14:15:50.891408 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:50.891362    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-666f668987-s624v"]
Sep 16 14:15:50.893809 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:15:50.893781    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db79e86_e959_4025_a1cb_4c54d27e2d2f.slice/crio-73fa6f99604ef3303eff33c9bd497b729c8007788fcdc85f87611f5c336fc8b0 WatchSource:0}: Error finding container 73fa6f99604ef3303eff33c9bd497b729c8007788fcdc85f87611f5c336fc8b0: Status 404 returned error can't find the container with id 73fa6f99604ef3303eff33c9bd497b729c8007788fcdc85f87611f5c336fc8b0
Sep 16 14:15:51.323017 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:51.322977    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-666f668987-s624v" event={"ID":"0db79e86-e959-4025-a1cb-4c54d27e2d2f","Type":"ContainerStarted","Data":"73fa6f99604ef3303eff33c9bd497b729c8007788fcdc85f87611f5c336fc8b0"}
Sep 16 14:15:51.324114 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:51.324090    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-mp9z7" event={"ID":"6a259dcd-592a-4579-9180-60f6dc5eac36","Type":"ContainerStarted","Data":"79914c654420d4996d233a0fbee43488ed903dc39e885705b3031ce7f29b38c0"}
Sep 16 14:15:52.658987 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:52.658961    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-metrics-tls\") pod \"dns-default-4ntk2\" (UID: \"38e73538-4f3c-4d56-bc47-e9bfb8525e0a\") " pod="openshift-dns/dns-default-4ntk2"
Sep 16 14:15:52.661301 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:52.661278    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38e73538-4f3c-4d56-bc47-e9bfb8525e0a-metrics-tls\") pod \"dns-default-4ntk2\" (UID: \"38e73538-4f3c-4d56-bc47-e9bfb8525e0a\") " pod="openshift-dns/dns-default-4ntk2"
Sep 16 14:15:52.918477 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:52.918440    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9f4mh\""
Sep 16 14:15:52.926697 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:52.926665    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4ntk2"
Sep 16 14:15:53.335479 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:53.335368    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-mp9z7" event={"ID":"6a259dcd-592a-4579-9180-60f6dc5eac36","Type":"ContainerStarted","Data":"fbcd8f58008a82cba9f3f5cb825590a7820e184a083b7926f1080ae87312fb97"}
Sep 16 14:15:53.335479 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:53.335451    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-mp9z7" event={"ID":"6a259dcd-592a-4579-9180-60f6dc5eac36","Type":"ContainerStarted","Data":"8c13a714906b806a3a4648a80392e2cdfc3bcdb21b668b1bd9b0d8be12f81f18"}
Sep 16 14:15:53.359211 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:53.359150    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-mp9z7" podStartSLOduration=2.618934206 podStartE2EDuration="4.359126437s" totalImagesPullingTime="1.740192231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:53.356782357 +0000 UTC m=+101.960407390" watchObservedRunningTime="2026-09-16 14:15:53.359126437 +0000 UTC m=+101.962751468"
Sep 16 14:15:54.172325 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:54.172297    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4ntk2"]
Sep 16 14:15:54.174373 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:15:54.174343    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e73538_4f3c_4d56_bc47_e9bfb8525e0a.slice/crio-084dd0b007d258ffa60172d02b22f06e945ef80a251c0c63ffb6c4cfbfd60674 WatchSource:0}: Error finding container 084dd0b007d258ffa60172d02b22f06e945ef80a251c0c63ffb6c4cfbfd60674: Status 404 returned error can't find the container with id 084dd0b007d258ffa60172d02b22f06e945ef80a251c0c63ffb6c4cfbfd60674
Sep 16 14:15:54.339533 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:54.339501    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4ntk2" event={"ID":"38e73538-4f3c-4d56-bc47-e9bfb8525e0a","Type":"ContainerStarted","Data":"084dd0b007d258ffa60172d02b22f06e945ef80a251c0c63ffb6c4cfbfd60674"}
Sep 16 14:15:54.341073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:54.341038    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-666f668987-s624v" event={"ID":"0db79e86-e959-4025-a1cb-4c54d27e2d2f","Type":"ContainerStarted","Data":"4b79c2a8f18bf06d3e71e09b2635661816004f48fe237b3bcc63e51ead5f2cce"}
Sep 16 14:15:54.362753 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:54.362705    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-666f668987-s624v" podStartSLOduration=1.296969882 podStartE2EDuration="4.362687213s" totalImagesPullingTime="3.065717331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:54.361417731 +0000 UTC m=+102.965042762" watchObservedRunningTime="2026-09-16 14:15:54.362687213 +0000 UTC m=+102.966312245"
Sep 16 14:15:55.071439 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.071093    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-8w9hx"]
Sep 16 14:15:55.074855 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.074829    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8w9hx"
Sep 16 14:15:55.077780 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.077740    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 16 14:15:55.081068 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.080733    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-pltxx\""
Sep 16 14:15:55.081068 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.080940    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 16 14:15:55.087347 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.087320    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-8w9hx"]
Sep 16 14:15:55.139275 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.139240    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zs8bz"]
Sep 16 14:15:55.142419 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.142400    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.153063 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.153037    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-njn57\""
Sep 16 14:15:55.153291 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.153278    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 14:15:55.154046 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.154030    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 14:15:55.154130 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.154096    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 14:15:55.183678 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.183652    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdhpl\" (UniqueName: \"kubernetes.io/projected/7761e508-6372-4021-a780-1cdf293c9ea1-kube-api-access-gdhpl\") pod \"node-exporter-zs8bz\" (UID: \"7761e508-6372-4021-a780-1cdf293c9ea1\") " pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.184118 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.183695    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5257ed36-4242-4d92-96e6-9981d89366cc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-8w9hx\" (UID: \"5257ed36-4242-4d92-96e6-9981d89366cc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8w9hx"
Sep 16 14:15:55.184118 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.183720    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7761e508-6372-4021-a780-1cdf293c9ea1-node-exporter-textfile\") pod \"node-exporter-zs8bz\" (UID: \"7761e508-6372-4021-a780-1cdf293c9ea1\") " pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.184118 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.183737    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns5nx\" (UniqueName: \"kubernetes.io/projected/5257ed36-4242-4d92-96e6-9981d89366cc-kube-api-access-ns5nx\") pod \"openshift-state-metrics-67dcf8c979-8w9hx\" (UID: \"5257ed36-4242-4d92-96e6-9981d89366cc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8w9hx"
Sep 16 14:15:55.184118 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.183792    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7761e508-6372-4021-a780-1cdf293c9ea1-metrics-client-ca\") pod \"node-exporter-zs8bz\" (UID: \"7761e508-6372-4021-a780-1cdf293c9ea1\") " pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.184118 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.183893    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5257ed36-4242-4d92-96e6-9981d89366cc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-8w9hx\" (UID: \"5257ed36-4242-4d92-96e6-9981d89366cc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8w9hx"
Sep 16 14:15:55.184118 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.183915    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7761e508-6372-4021-a780-1cdf293c9ea1-node-exporter-wtmp\") pod \"node-exporter-zs8bz\" (UID: \"7761e508-6372-4021-a780-1cdf293c9ea1\") " pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.184118 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.183934    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7761e508-6372-4021-a780-1cdf293c9ea1-root\") pod \"node-exporter-zs8bz\" (UID: \"7761e508-6372-4021-a780-1cdf293c9ea1\") " pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.184118 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.183954    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7761e508-6372-4021-a780-1cdf293c9ea1-node-exporter-tls\") pod \"node-exporter-zs8bz\" (UID: \"7761e508-6372-4021-a780-1cdf293c9ea1\") " pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.184118 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.183982    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7761e508-6372-4021-a780-1cdf293c9ea1-sys\") pod \"node-exporter-zs8bz\" (UID: \"7761e508-6372-4021-a780-1cdf293c9ea1\") " pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.184118 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.184001    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5257ed36-4242-4d92-96e6-9981d89366cc-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-8w9hx\" (UID: \"5257ed36-4242-4d92-96e6-9981d89366cc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8w9hx"
Sep 16 14:15:55.184118 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.184032    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7761e508-6372-4021-a780-1cdf293c9ea1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zs8bz\" (UID: \"7761e508-6372-4021-a780-1cdf293c9ea1\") " pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.184118 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.184103    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7761e508-6372-4021-a780-1cdf293c9ea1-node-exporter-accelerators-collector-config\") pod \"node-exporter-zs8bz\" (UID: \"7761e508-6372-4021-a780-1cdf293c9ea1\") " pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.223505 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.223480    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h7q7r"
Sep 16 14:15:55.284826 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.284778    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7761e508-6372-4021-a780-1cdf293c9ea1-node-exporter-accelerators-collector-config\") pod \"node-exporter-zs8bz\" (UID: \"7761e508-6372-4021-a780-1cdf293c9ea1\") " pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.284999 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.284834    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdhpl\" (UniqueName: \"kubernetes.io/projected/7761e508-6372-4021-a780-1cdf293c9ea1-kube-api-access-gdhpl\") pod \"node-exporter-zs8bz\" (UID: \"7761e508-6372-4021-a780-1cdf293c9ea1\") " pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.284999 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.284894    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5257ed36-4242-4d92-96e6-9981d89366cc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-8w9hx\" (UID: \"5257ed36-4242-4d92-96e6-9981d89366cc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8w9hx"
Sep 16 14:15:55.285331 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.285298    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7761e508-6372-4021-a780-1cdf293c9ea1-node-exporter-textfile\") pod \"node-exporter-zs8bz\" (UID: \"7761e508-6372-4021-a780-1cdf293c9ea1\") " pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.285445 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.285353    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns5nx\" (UniqueName: \"kubernetes.io/projected/5257ed36-4242-4d92-96e6-9981d89366cc-kube-api-access-ns5nx\") pod \"openshift-state-metrics-67dcf8c979-8w9hx\" (UID: \"5257ed36-4242-4d92-96e6-9981d89366cc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8w9hx"
Sep 16 14:15:55.285445 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.285419    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7761e508-6372-4021-a780-1cdf293c9ea1-metrics-client-ca\") pod \"node-exporter-zs8bz\" (UID: \"7761e508-6372-4021-a780-1cdf293c9ea1\") " pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.285522 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.285467    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7761e508-6372-4021-a780-1cdf293c9ea1-node-exporter-accelerators-collector-config\") pod \"node-exporter-zs8bz\" (UID: \"7761e508-6372-4021-a780-1cdf293c9ea1\") " pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.285522 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.285503    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5257ed36-4242-4d92-96e6-9981d89366cc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-8w9hx\" (UID: \"5257ed36-4242-4d92-96e6-9981d89366cc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8w9hx"
Sep 16 14:15:55.285602 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.285531    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7761e508-6372-4021-a780-1cdf293c9ea1-node-exporter-wtmp\") pod \"node-exporter-zs8bz\" (UID: \"7761e508-6372-4021-a780-1cdf293c9ea1\") " pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.285602 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.285561    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7761e508-6372-4021-a780-1cdf293c9ea1-root\") pod \"node-exporter-zs8bz\" (UID: \"7761e508-6372-4021-a780-1cdf293c9ea1\") " pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.285602 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.285594    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7761e508-6372-4021-a780-1cdf293c9ea1-node-exporter-tls\") pod \"node-exporter-zs8bz\" (UID: \"7761e508-6372-4021-a780-1cdf293c9ea1\") " pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.285715 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.285631    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7761e508-6372-4021-a780-1cdf293c9ea1-sys\") pod \"node-exporter-zs8bz\" (UID: \"7761e508-6372-4021-a780-1cdf293c9ea1\") " pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.285715 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.285658    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5257ed36-4242-4d92-96e6-9981d89366cc-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-8w9hx\" (UID: \"5257ed36-4242-4d92-96e6-9981d89366cc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8w9hx"
Sep 16 14:15:55.285782 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.285749    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7761e508-6372-4021-a780-1cdf293c9ea1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zs8bz\" (UID: \"7761e508-6372-4021-a780-1cdf293c9ea1\") " pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.285959 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.285868    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7761e508-6372-4021-a780-1cdf293c9ea1-sys\") pod \"node-exporter-zs8bz\" (UID: \"7761e508-6372-4021-a780-1cdf293c9ea1\") " pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.286034 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.285981    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7761e508-6372-4021-a780-1cdf293c9ea1-node-exporter-textfile\") pod \"node-exporter-zs8bz\" (UID: \"7761e508-6372-4021-a780-1cdf293c9ea1\") " pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.286094 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.286038    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7761e508-6372-4021-a780-1cdf293c9ea1-root\") pod \"node-exporter-zs8bz\" (UID: \"7761e508-6372-4021-a780-1cdf293c9ea1\") " pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.287306 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.286241    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7761e508-6372-4021-a780-1cdf293c9ea1-metrics-client-ca\") pod \"node-exporter-zs8bz\" (UID: \"7761e508-6372-4021-a780-1cdf293c9ea1\") " pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.287306 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.287233    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5257ed36-4242-4d92-96e6-9981d89366cc-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-8w9hx\" (UID: \"5257ed36-4242-4d92-96e6-9981d89366cc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8w9hx"
Sep 16 14:15:55.287611 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.287533    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7761e508-6372-4021-a780-1cdf293c9ea1-node-exporter-wtmp\") pod \"node-exporter-zs8bz\" (UID: \"7761e508-6372-4021-a780-1cdf293c9ea1\") " pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.288035 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.288014    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5257ed36-4242-4d92-96e6-9981d89366cc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-8w9hx\" (UID: \"5257ed36-4242-4d92-96e6-9981d89366cc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8w9hx"
Sep 16 14:15:55.288821 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.288797    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7761e508-6372-4021-a780-1cdf293c9ea1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zs8bz\" (UID: \"7761e508-6372-4021-a780-1cdf293c9ea1\") " pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.289804 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.289768    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7761e508-6372-4021-a780-1cdf293c9ea1-node-exporter-tls\") pod \"node-exporter-zs8bz\" (UID: \"7761e508-6372-4021-a780-1cdf293c9ea1\") " pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.289905 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.289876    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5257ed36-4242-4d92-96e6-9981d89366cc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-8w9hx\" (UID: \"5257ed36-4242-4d92-96e6-9981d89366cc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8w9hx"
Sep 16 14:15:55.315568 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.315534    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns5nx\" (UniqueName: \"kubernetes.io/projected/5257ed36-4242-4d92-96e6-9981d89366cc-kube-api-access-ns5nx\") pod \"openshift-state-metrics-67dcf8c979-8w9hx\" (UID: \"5257ed36-4242-4d92-96e6-9981d89366cc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8w9hx"
Sep 16 14:15:55.319870 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.319762    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdhpl\" (UniqueName: \"kubernetes.io/projected/7761e508-6372-4021-a780-1cdf293c9ea1-kube-api-access-gdhpl\") pod \"node-exporter-zs8bz\" (UID: \"7761e508-6372-4021-a780-1cdf293c9ea1\") " pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.389720 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.389645    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8w9hx"
Sep 16 14:15:55.450863 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.450826    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zs8bz"
Sep 16 14:15:55.682017 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:55.681990    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-8w9hx"]
Sep 16 14:15:56.210089 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:15:56.210056    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5257ed36_4242_4d92_96e6_9981d89366cc.slice/crio-5b95a6f473cc032a45b099a4f8302d4562929e758148f13a888b55720a2090c7 WatchSource:0}: Error finding container 5b95a6f473cc032a45b099a4f8302d4562929e758148f13a888b55720a2090c7: Status 404 returned error can't find the container with id 5b95a6f473cc032a45b099a4f8302d4562929e758148f13a888b55720a2090c7
Sep 16 14:15:56.351450 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:56.351265    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8w9hx" event={"ID":"5257ed36-4242-4d92-96e6-9981d89366cc","Type":"ContainerStarted","Data":"bec1661041e01d023acc675a6698fc11efec6eea367ec8d5e882993f7775b455"}
Sep 16 14:15:56.351450 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:56.351334    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8w9hx" event={"ID":"5257ed36-4242-4d92-96e6-9981d89366cc","Type":"ContainerStarted","Data":"5b95a6f473cc032a45b099a4f8302d4562929e758148f13a888b55720a2090c7"}
Sep 16 14:15:56.352709 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:56.352681    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zs8bz" event={"ID":"7761e508-6372-4021-a780-1cdf293c9ea1","Type":"ContainerStarted","Data":"cad6d9451c73bc157639becdb8637c23ba709d49df3b1bd4e12b5d5d65bb5af6"}
Sep 16 14:15:57.357523 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:57.357488    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8w9hx" event={"ID":"5257ed36-4242-4d92-96e6-9981d89366cc","Type":"ContainerStarted","Data":"0a725f50dec9923bb8c6b01d7942512a255767bbd70038c043fe58eb8c979d2f"}
Sep 16 14:15:57.359096 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:57.359070    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4ntk2" event={"ID":"38e73538-4f3c-4d56-bc47-e9bfb8525e0a","Type":"ContainerStarted","Data":"c7020339ef9cdc82232266f90b701c90d268b07dea9535f14f97e3aabaaf79bd"}
Sep 16 14:15:57.359208 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:57.359117    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4ntk2" event={"ID":"38e73538-4f3c-4d56-bc47-e9bfb8525e0a","Type":"ContainerStarted","Data":"310fc882fa5c1c3e0f3bcf4b74bc2bc8e825aa469296cbda2a710f5a3a8b07b0"}
Sep 16 14:15:57.359247 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:57.359207    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4ntk2"
Sep 16 14:15:57.377789 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:57.377739    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4ntk2" podStartSLOduration=66.951491734 podStartE2EDuration="1m9.377722568s" totalImagesPullingTime="2.426230834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:57.377023204 +0000 UTC m=+105.980648235" watchObservedRunningTime="2026-09-16 14:15:57.377722568 +0000 UTC m=+105.981347600"
Sep 16 14:15:58.285096 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:58.285026    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-f45d8ddc4-2zl52"
Sep 16 14:15:58.365986 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:58.365948    2587 generic.go:356] "Generic (PLEG): container finished" podID="7761e508-6372-4021-a780-1cdf293c9ea1" containerID="ef02084e30d8eba6cc7e4724ff36a95f8b46ec01344d3c58942b4b496973976f" exitCode=0
Sep 16 14:15:58.366486 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:58.366038    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zs8bz" event={"ID":"7761e508-6372-4021-a780-1cdf293c9ea1","Type":"ContainerDied","Data":"ef02084e30d8eba6cc7e4724ff36a95f8b46ec01344d3c58942b4b496973976f"}
Sep 16 14:15:59.371840 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.371742    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zs8bz" event={"ID":"7761e508-6372-4021-a780-1cdf293c9ea1","Type":"ContainerStarted","Data":"6bc24c4b4777459de09bf5667bcedb3ead0f67f73485a6d4a21aac3689cfd02b"}
Sep 16 14:15:59.371840 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.371807    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zs8bz" event={"ID":"7761e508-6372-4021-a780-1cdf293c9ea1","Type":"ContainerStarted","Data":"e498762df53727209df9e9c8294aecde40b4ed2b11fd50ce223b03c2de3ae381"}
Sep 16 14:15:59.373811 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.373787    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8w9hx" event={"ID":"5257ed36-4242-4d92-96e6-9981d89366cc","Type":"ContainerStarted","Data":"cc163b2bc014185ba7de4a1664ce4aa1447827584616498c58eb88f3e0df8955"}
Sep 16 14:15:59.399193 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.399147    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zs8bz" podStartSLOduration=1.744791646 podStartE2EDuration="4.399132123s" totalImagesPullingTime="2.546363998s" totalInitContainerRuntime="107.976479ms" isStatefulPod=true podCreationTimestamp="2026-09-16 14:15:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:59.397799173 +0000 UTC m=+108.001424207" watchObservedRunningTime="2026-09-16 14:15:59.399132123 +0000 UTC m=+108.002757156"
Sep 16 14:15:59.418205 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.418157    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8w9hx" podStartSLOduration=1.971247103 podStartE2EDuration="4.418119716s" totalImagesPullingTime="2.446872613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:15:59.417285467 +0000 UTC m=+108.020910512" watchObservedRunningTime="2026-09-16 14:15:59.418119716 +0000 UTC m=+108.021744747"
Sep 16 14:15:59.532072 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.532038    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-78569c95d-l6whj"]
Sep 16 14:15:59.553945 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.553913    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-78569c95d-l6whj"]
Sep 16 14:15:59.554095 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.554010    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-78569c95d-l6whj"
Sep 16 14:15:59.556779 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.556749    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 14:15:59.556930 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.556805    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 14:15:59.556930 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.556923    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 14:15:59.557181 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.557165    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-bztj4\""
Sep 16 14:15:59.557230 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.557187    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 14:15:59.557278 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.557259    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-97ggl606ktkak\""
Sep 16 14:15:59.627423 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.627320    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/380482d8-a477-4212-9214-ce63a801e2bc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78569c95d-l6whj\" (UID: \"380482d8-a477-4212-9214-ce63a801e2bc\") " pod="openshift-monitoring/metrics-server-78569c95d-l6whj"
Sep 16 14:15:59.627423 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.627360    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/380482d8-a477-4212-9214-ce63a801e2bc-secret-metrics-server-client-certs\") pod \"metrics-server-78569c95d-l6whj\" (UID: \"380482d8-a477-4212-9214-ce63a801e2bc\") " pod="openshift-monitoring/metrics-server-78569c95d-l6whj"
Sep 16 14:15:59.627423 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.627384    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/380482d8-a477-4212-9214-ce63a801e2bc-secret-metrics-server-tls\") pod \"metrics-server-78569c95d-l6whj\" (UID: \"380482d8-a477-4212-9214-ce63a801e2bc\") " pod="openshift-monitoring/metrics-server-78569c95d-l6whj"
Sep 16 14:15:59.627656 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.627442    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/380482d8-a477-4212-9214-ce63a801e2bc-client-ca-bundle\") pod \"metrics-server-78569c95d-l6whj\" (UID: \"380482d8-a477-4212-9214-ce63a801e2bc\") " pod="openshift-monitoring/metrics-server-78569c95d-l6whj"
Sep 16 14:15:59.627656 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.627483    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz9lm\" (UniqueName: \"kubernetes.io/projected/380482d8-a477-4212-9214-ce63a801e2bc-kube-api-access-hz9lm\") pod \"metrics-server-78569c95d-l6whj\" (UID: \"380482d8-a477-4212-9214-ce63a801e2bc\") " pod="openshift-monitoring/metrics-server-78569c95d-l6whj"
Sep 16 14:15:59.627656 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.627541    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/380482d8-a477-4212-9214-ce63a801e2bc-audit-log\") pod \"metrics-server-78569c95d-l6whj\" (UID: \"380482d8-a477-4212-9214-ce63a801e2bc\") " pod="openshift-monitoring/metrics-server-78569c95d-l6whj"
Sep 16 14:15:59.627656 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.627563    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/380482d8-a477-4212-9214-ce63a801e2bc-metrics-server-audit-profiles\") pod \"metrics-server-78569c95d-l6whj\" (UID: \"380482d8-a477-4212-9214-ce63a801e2bc\") " pod="openshift-monitoring/metrics-server-78569c95d-l6whj"
Sep 16 14:15:59.728622 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.728585    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/380482d8-a477-4212-9214-ce63a801e2bc-audit-log\") pod \"metrics-server-78569c95d-l6whj\" (UID: \"380482d8-a477-4212-9214-ce63a801e2bc\") " pod="openshift-monitoring/metrics-server-78569c95d-l6whj"
Sep 16 14:15:59.728622 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.728622    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/380482d8-a477-4212-9214-ce63a801e2bc-metrics-server-audit-profiles\") pod \"metrics-server-78569c95d-l6whj\" (UID: \"380482d8-a477-4212-9214-ce63a801e2bc\") " pod="openshift-monitoring/metrics-server-78569c95d-l6whj"
Sep 16 14:15:59.728877 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.728765    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/380482d8-a477-4212-9214-ce63a801e2bc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78569c95d-l6whj\" (UID: \"380482d8-a477-4212-9214-ce63a801e2bc\") " pod="openshift-monitoring/metrics-server-78569c95d-l6whj"
Sep 16 14:15:59.728877 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.728823    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/380482d8-a477-4212-9214-ce63a801e2bc-secret-metrics-server-client-certs\") pod \"metrics-server-78569c95d-l6whj\" (UID: \"380482d8-a477-4212-9214-ce63a801e2bc\") " pod="openshift-monitoring/metrics-server-78569c95d-l6whj"
Sep 16 14:15:59.728877 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.728860    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/380482d8-a477-4212-9214-ce63a801e2bc-secret-metrics-server-tls\") pod \"metrics-server-78569c95d-l6whj\" (UID: \"380482d8-a477-4212-9214-ce63a801e2bc\") " pod="openshift-monitoring/metrics-server-78569c95d-l6whj"
Sep 16 14:15:59.729019 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.728893    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/380482d8-a477-4212-9214-ce63a801e2bc-client-ca-bundle\") pod \"metrics-server-78569c95d-l6whj\" (UID: \"380482d8-a477-4212-9214-ce63a801e2bc\") " pod="openshift-monitoring/metrics-server-78569c95d-l6whj"
Sep 16 14:15:59.729019 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.728953    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz9lm\" (UniqueName: \"kubernetes.io/projected/380482d8-a477-4212-9214-ce63a801e2bc-kube-api-access-hz9lm\") pod \"metrics-server-78569c95d-l6whj\" (UID: \"380482d8-a477-4212-9214-ce63a801e2bc\") " pod="openshift-monitoring/metrics-server-78569c95d-l6whj"
Sep 16 14:15:59.729112 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.729067    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/380482d8-a477-4212-9214-ce63a801e2bc-audit-log\") pod \"metrics-server-78569c95d-l6whj\" (UID: \"380482d8-a477-4212-9214-ce63a801e2bc\") " pod="openshift-monitoring/metrics-server-78569c95d-l6whj"
Sep 16 14:15:59.729367 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.729338    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/380482d8-a477-4212-9214-ce63a801e2bc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78569c95d-l6whj\" (UID: \"380482d8-a477-4212-9214-ce63a801e2bc\") " pod="openshift-monitoring/metrics-server-78569c95d-l6whj"
Sep 16 14:15:59.729860 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.729835    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/380482d8-a477-4212-9214-ce63a801e2bc-metrics-server-audit-profiles\") pod \"metrics-server-78569c95d-l6whj\" (UID: \"380482d8-a477-4212-9214-ce63a801e2bc\") " pod="openshift-monitoring/metrics-server-78569c95d-l6whj"
Sep 16 14:15:59.731469 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.731438    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/380482d8-a477-4212-9214-ce63a801e2bc-secret-metrics-server-tls\") pod \"metrics-server-78569c95d-l6whj\" (UID: \"380482d8-a477-4212-9214-ce63a801e2bc\") " pod="openshift-monitoring/metrics-server-78569c95d-l6whj"
Sep 16 14:15:59.731620 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.731602    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/380482d8-a477-4212-9214-ce63a801e2bc-client-ca-bundle\") pod \"metrics-server-78569c95d-l6whj\" (UID: \"380482d8-a477-4212-9214-ce63a801e2bc\") " pod="openshift-monitoring/metrics-server-78569c95d-l6whj"
Sep 16 14:15:59.731688 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.731604    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/380482d8-a477-4212-9214-ce63a801e2bc-secret-metrics-server-client-certs\") pod \"metrics-server-78569c95d-l6whj\" (UID: \"380482d8-a477-4212-9214-ce63a801e2bc\") " pod="openshift-monitoring/metrics-server-78569c95d-l6whj"
Sep 16 14:15:59.738147 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.738122    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz9lm\" (UniqueName: \"kubernetes.io/projected/380482d8-a477-4212-9214-ce63a801e2bc-kube-api-access-hz9lm\") pod \"metrics-server-78569c95d-l6whj\" (UID: \"380482d8-a477-4212-9214-ce63a801e2bc\") " pod="openshift-monitoring/metrics-server-78569c95d-l6whj"
Sep 16 14:15:59.858696 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.858658    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-jn8t7"]
Sep 16 14:15:59.864567 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.864532    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-78569c95d-l6whj"
Sep 16 14:15:59.864915 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.864874    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-jn8t7"
Sep 16 14:15:59.868955 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.868431    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 14:15:59.868955 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.868609    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-6dqgp\""
Sep 16 14:15:59.871350 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.871323    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-jn8t7"]
Sep 16 14:15:59.931123 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.931091    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9d47a49-cd8e-4a5f-88b4-a96ea09f9340-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-jn8t7\" (UID: \"a9d47a49-cd8e-4a5f-88b4-a96ea09f9340\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-jn8t7"
Sep 16 14:15:59.931250 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:15:59.931203    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvgm4\" (UniqueName: \"kubernetes.io/projected/a9d47a49-cd8e-4a5f-88b4-a96ea09f9340-kube-api-access-hvgm4\") pod \"monitoring-plugin-5d85778b9d-jn8t7\" (UID: \"a9d47a49-cd8e-4a5f-88b4-a96ea09f9340\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-jn8t7"
Sep 16 14:16:00.032374 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:00.032340    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvgm4\" (UniqueName: \"kubernetes.io/projected/a9d47a49-cd8e-4a5f-88b4-a96ea09f9340-kube-api-access-hvgm4\") pod \"monitoring-plugin-5d85778b9d-jn8t7\" (UID: \"a9d47a49-cd8e-4a5f-88b4-a96ea09f9340\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-jn8t7"
Sep 16 14:16:00.032574 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:00.032443    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9d47a49-cd8e-4a5f-88b4-a96ea09f9340-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-jn8t7\" (UID: \"a9d47a49-cd8e-4a5f-88b4-a96ea09f9340\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-jn8t7"
Sep 16 14:16:00.032574 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:16:00.032542    2587 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 16 14:16:00.032647 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:16:00.032596    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9d47a49-cd8e-4a5f-88b4-a96ea09f9340-monitoring-plugin-cert podName:a9d47a49-cd8e-4a5f-88b4-a96ea09f9340 nodeName:}" failed. No retries permitted until 2026-09-16 14:16:00.532582247 +0000 UTC m=+109.136207261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/a9d47a49-cd8e-4a5f-88b4-a96ea09f9340-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-jn8t7" (UID: "a9d47a49-cd8e-4a5f-88b4-a96ea09f9340") : secret "monitoring-plugin-cert" not found
Sep 16 14:16:00.040877 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:00.040850    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvgm4\" (UniqueName: \"kubernetes.io/projected/a9d47a49-cd8e-4a5f-88b4-a96ea09f9340-kube-api-access-hvgm4\") pod \"monitoring-plugin-5d85778b9d-jn8t7\" (UID: \"a9d47a49-cd8e-4a5f-88b4-a96ea09f9340\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-jn8t7"
Sep 16 14:16:00.145405 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:00.145359    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-78569c95d-l6whj"]
Sep 16 14:16:00.147267 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:16:00.147234    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380482d8_a477_4212_9214_ce63a801e2bc.slice/crio-7c3daf6a50c37c68bc29c6827d4a5ba01cd8a01c61b2d6cd14108b184acf455a WatchSource:0}: Error finding container 7c3daf6a50c37c68bc29c6827d4a5ba01cd8a01c61b2d6cd14108b184acf455a: Status 404 returned error can't find the container with id 7c3daf6a50c37c68bc29c6827d4a5ba01cd8a01c61b2d6cd14108b184acf455a
Sep 16 14:16:00.378836 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:00.378806    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-78569c95d-l6whj" event={"ID":"380482d8-a477-4212-9214-ce63a801e2bc","Type":"ContainerStarted","Data":"7c3daf6a50c37c68bc29c6827d4a5ba01cd8a01c61b2d6cd14108b184acf455a"}
Sep 16 14:16:00.537735 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:00.537700    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9d47a49-cd8e-4a5f-88b4-a96ea09f9340-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-jn8t7\" (UID: \"a9d47a49-cd8e-4a5f-88b4-a96ea09f9340\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-jn8t7"
Sep 16 14:16:00.540203 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:00.540184    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9d47a49-cd8e-4a5f-88b4-a96ea09f9340-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-jn8t7\" (UID: \"a9d47a49-cd8e-4a5f-88b4-a96ea09f9340\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-jn8t7"
Sep 16 14:16:00.611284 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:00.611240    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-666f668987-s624v"
Sep 16 14:16:00.611595 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:00.611336    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-666f668987-s624v"
Sep 16 14:16:00.616266 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:00.616243    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-666f668987-s624v"
Sep 16 14:16:00.723572 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:00.723540    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-666f668987-s624v"]
Sep 16 14:16:00.724826 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:00.724791    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4" podUID="8c16289f-192f-4eda-a7e5-3c8aef6f14e6" containerName="registry" containerID="cri-o://ef393ea5dd9fb91e7ae09653537b1d78359754dc0c0b0a90641b79a29a11b5a7" gracePeriod=30
Sep 16 14:16:00.793841 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:00.793761    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-jn8t7"
Sep 16 14:16:00.978595 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:00.978567    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:16:01.041885 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.041847    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-trusted-ca\") pod \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") "
Sep 16 14:16:01.042067 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.041898    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-installation-pull-secrets\") pod \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") "
Sep 16 14:16:01.042067 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.041931    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-ca-trust-extracted\") pod \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") "
Sep 16 14:16:01.042067 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.041959    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-registry-tls\") pod \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") "
Sep 16 14:16:01.042067 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.042007    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-bound-sa-token\") pod \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") "
Sep 16 14:16:01.042067 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.042058    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-registry-certificates\") pod \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") "
Sep 16 14:16:01.042348 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.042093    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-image-registry-private-configuration\") pod \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") "
Sep 16 14:16:01.042348 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.042146    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8xc7\" (UniqueName: \"kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-kube-api-access-s8xc7\") pod \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\" (UID: \"8c16289f-192f-4eda-a7e5-3c8aef6f14e6\") "
Sep 16 14:16:01.042348 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.042333    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8c16289f-192f-4eda-a7e5-3c8aef6f14e6" (UID: "8c16289f-192f-4eda-a7e5-3c8aef6f14e6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:16:01.042574 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.042468    2587 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-trusted-ca\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:16:01.043306 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.043273    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8c16289f-192f-4eda-a7e5-3c8aef6f14e6" (UID: "8c16289f-192f-4eda-a7e5-3c8aef6f14e6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:16:01.046442 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.046310    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8c16289f-192f-4eda-a7e5-3c8aef6f14e6" (UID: "8c16289f-192f-4eda-a7e5-3c8aef6f14e6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:16:01.046442 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.046315    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8c16289f-192f-4eda-a7e5-3c8aef6f14e6" (UID: "8c16289f-192f-4eda-a7e5-3c8aef6f14e6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:16:01.046709 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.046528    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-kube-api-access-s8xc7" (OuterVolumeSpecName: "kube-api-access-s8xc7") pod "8c16289f-192f-4eda-a7e5-3c8aef6f14e6" (UID: "8c16289f-192f-4eda-a7e5-3c8aef6f14e6"). InnerVolumeSpecName "kube-api-access-s8xc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:16:01.046986 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.046954    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8c16289f-192f-4eda-a7e5-3c8aef6f14e6" (UID: "8c16289f-192f-4eda-a7e5-3c8aef6f14e6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:16:01.047283 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.047265    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8c16289f-192f-4eda-a7e5-3c8aef6f14e6" (UID: "8c16289f-192f-4eda-a7e5-3c8aef6f14e6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:16:01.053314 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.053287    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8c16289f-192f-4eda-a7e5-3c8aef6f14e6" (UID: "8c16289f-192f-4eda-a7e5-3c8aef6f14e6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:16:01.077659 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.077631    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-jn8t7"]
Sep 16 14:16:01.081028 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:16:01.081000    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d47a49_cd8e_4a5f_88b4_a96ea09f9340.slice/crio-2acdc9c713aaf106982c312a821a5e0e6e837028173588ee1919c4f8aeb4b866 WatchSource:0}: Error finding container 2acdc9c713aaf106982c312a821a5e0e6e837028173588ee1919c4f8aeb4b866: Status 404 returned error can't find the container with id 2acdc9c713aaf106982c312a821a5e0e6e837028173588ee1919c4f8aeb4b866
Sep 16 14:16:01.143950 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.143919    2587 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-installation-pull-secrets\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:16:01.143950 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.143951    2587 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-ca-trust-extracted\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:16:01.144181 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.143967    2587 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-registry-tls\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:16:01.144181 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.143980    2587 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-bound-sa-token\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:16:01.144181 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.143995    2587 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-registry-certificates\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:16:01.144181 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.144008    2587 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-image-registry-private-configuration\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:16:01.144181 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.144017    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8xc7\" (UniqueName: \"kubernetes.io/projected/8c16289f-192f-4eda-a7e5-3c8aef6f14e6-kube-api-access-s8xc7\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:16:01.386438 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.386338    2587 generic.go:356] "Generic (PLEG): container finished" podID="8c16289f-192f-4eda-a7e5-3c8aef6f14e6" containerID="ef393ea5dd9fb91e7ae09653537b1d78359754dc0c0b0a90641b79a29a11b5a7" exitCode=0
Sep 16 14:16:01.386438 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.386411    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4"
Sep 16 14:16:01.386959 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.386432    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4" event={"ID":"8c16289f-192f-4eda-a7e5-3c8aef6f14e6","Type":"ContainerDied","Data":"ef393ea5dd9fb91e7ae09653537b1d78359754dc0c0b0a90641b79a29a11b5a7"}
Sep 16 14:16:01.386959 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.386494    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f74cc64bb-gpqp4" event={"ID":"8c16289f-192f-4eda-a7e5-3c8aef6f14e6","Type":"ContainerDied","Data":"eb1c32803edbece1d3f0961079d2075c71b434f77a2d147521b8c0184e31b733"}
Sep 16 14:16:01.386959 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.386523    2587 scope.go:117] "RemoveContainer" containerID="ef393ea5dd9fb91e7ae09653537b1d78359754dc0c0b0a90641b79a29a11b5a7"
Sep 16 14:16:01.389067 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.389041    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-jn8t7" event={"ID":"a9d47a49-cd8e-4a5f-88b4-a96ea09f9340","Type":"ContainerStarted","Data":"2acdc9c713aaf106982c312a821a5e0e6e837028173588ee1919c4f8aeb4b866"}
Sep 16 14:16:01.396975 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.396952    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-666f668987-s624v"
Sep 16 14:16:01.434798 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.434747    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f74cc64bb-gpqp4"]
Sep 16 14:16:01.438487 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.438455    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f74cc64bb-gpqp4"]
Sep 16 14:16:01.626824 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.626800    2587 scope.go:117] "RemoveContainer" containerID="ef393ea5dd9fb91e7ae09653537b1d78359754dc0c0b0a90641b79a29a11b5a7"
Sep 16 14:16:01.627232 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:16:01.627208    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef393ea5dd9fb91e7ae09653537b1d78359754dc0c0b0a90641b79a29a11b5a7\": container with ID starting with ef393ea5dd9fb91e7ae09653537b1d78359754dc0c0b0a90641b79a29a11b5a7 not found: ID does not exist" containerID="ef393ea5dd9fb91e7ae09653537b1d78359754dc0c0b0a90641b79a29a11b5a7"
Sep 16 14:16:01.629287 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.627245    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef393ea5dd9fb91e7ae09653537b1d78359754dc0c0b0a90641b79a29a11b5a7"} err="failed to get container status \"ef393ea5dd9fb91e7ae09653537b1d78359754dc0c0b0a90641b79a29a11b5a7\": rpc error: code = NotFound desc = could not find container \"ef393ea5dd9fb91e7ae09653537b1d78359754dc0c0b0a90641b79a29a11b5a7\": container with ID starting with ef393ea5dd9fb91e7ae09653537b1d78359754dc0c0b0a90641b79a29a11b5a7 not found: ID does not exist"
Sep 16 14:16:01.910632 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:01.910597    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c16289f-192f-4eda-a7e5-3c8aef6f14e6" path="/var/lib/kubelet/pods/8c16289f-192f-4eda-a7e5-3c8aef6f14e6/volumes"
Sep 16 14:16:02.395908 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:02.395874    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-78569c95d-l6whj" event={"ID":"380482d8-a477-4212-9214-ce63a801e2bc","Type":"ContainerStarted","Data":"74f80ae0605302cb695c1d02fa2cf3436f57aaa378d92181ae790bad0c34e4b7"}
Sep 16 14:16:02.423090 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:02.423031    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-78569c95d-l6whj" podStartSLOduration=1.8956680320000001 podStartE2EDuration="3.42300812s" totalImagesPullingTime="1.527340088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:16:02.420596502 +0000 UTC m=+111.024221534" watchObservedRunningTime="2026-09-16 14:16:02.42300812 +0000 UTC m=+111.026633152"
Sep 16 14:16:03.402183 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:03.402146    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-jn8t7" event={"ID":"a9d47a49-cd8e-4a5f-88b4-a96ea09f9340","Type":"ContainerStarted","Data":"b3a7a424c322449ab1da4f67d66354dd85672af081e6797d7771529eb3a2a666"}
Sep 16 14:16:03.421108 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:03.421053    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-jn8t7" podStartSLOduration=2.5245473389999997 podStartE2EDuration="4.42103843s" totalImagesPullingTime="1.896491091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:16:03.4200662 +0000 UTC m=+112.023691238" watchObservedRunningTime="2026-09-16 14:16:03.42103843 +0000 UTC m=+112.024663461"
Sep 16 14:16:04.405614 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:04.405582    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-jn8t7"
Sep 16 14:16:04.410695 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:04.410671    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-jn8t7"
Sep 16 14:16:07.368998 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:07.368966    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4ntk2"
Sep 16 14:16:19.865471 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:19.865433    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-78569c95d-l6whj"
Sep 16 14:16:19.865882 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:19.865500    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-78569c95d-l6whj"
Sep 16 14:16:21.722832 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:21.722781    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e10f563c-16f8-4699-a7d4-52e6bfba29a5-metrics-certs\") pod \"network-metrics-daemon-6qpvt\" (UID: \"e10f563c-16f8-4699-a7d4-52e6bfba29a5\") " pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:16:21.725302 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:21.725273    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e10f563c-16f8-4699-a7d4-52e6bfba29a5-metrics-certs\") pod \"network-metrics-daemon-6qpvt\" (UID: \"e10f563c-16f8-4699-a7d4-52e6bfba29a5\") " pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:16:22.009978 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:22.009896    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-999gw\""
Sep 16 14:16:22.017275 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:22.017250    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6qpvt"
Sep 16 14:16:22.291728 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:22.291646    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6qpvt"]
Sep 16 14:16:22.294908 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:16:22.294881    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode10f563c_16f8_4699_a7d4_52e6bfba29a5.slice/crio-5a0d4314e2f318d7b403944661fc49151c30ba146d512607653ad7b30814e3ac WatchSource:0}: Error finding container 5a0d4314e2f318d7b403944661fc49151c30ba146d512607653ad7b30814e3ac: Status 404 returned error can't find the container with id 5a0d4314e2f318d7b403944661fc49151c30ba146d512607653ad7b30814e3ac
Sep 16 14:16:22.474232 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:22.474196    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6qpvt" event={"ID":"e10f563c-16f8-4699-a7d4-52e6bfba29a5","Type":"ContainerStarted","Data":"5a0d4314e2f318d7b403944661fc49151c30ba146d512607653ad7b30814e3ac"}
Sep 16 14:16:22.475738 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:22.475711    2587 generic.go:356] "Generic (PLEG): container finished" podID="3e8e2284-b089-4525-80c5-93353f07252f" containerID="1cdedc16d60b66d3fd1fa8856370de0849d5a4fc9b2f1a2da0d3923ea7e7b2ff" exitCode=0
Sep 16 14:16:22.475860 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:22.475789    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-j674w" event={"ID":"3e8e2284-b089-4525-80c5-93353f07252f","Type":"ContainerDied","Data":"1cdedc16d60b66d3fd1fa8856370de0849d5a4fc9b2f1a2da0d3923ea7e7b2ff"}
Sep 16 14:16:22.476174 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:22.476159    2587 scope.go:117] "RemoveContainer" containerID="1cdedc16d60b66d3fd1fa8856370de0849d5a4fc9b2f1a2da0d3923ea7e7b2ff"
Sep 16 14:16:23.482362 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:23.482320    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6qpvt" event={"ID":"e10f563c-16f8-4699-a7d4-52e6bfba29a5","Type":"ContainerStarted","Data":"dd4980a497b1e801d5e5afef97596740cdc3b473d0786b48f41f019e0e46ffef"}
Sep 16 14:16:23.482824 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:23.482410    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6qpvt" event={"ID":"e10f563c-16f8-4699-a7d4-52e6bfba29a5","Type":"ContainerStarted","Data":"838a7779bfdb3301b559c221f5d34e7ac2ebbce2484f46f9c885f350cba8b0c7"}
Sep 16 14:16:23.484772 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:23.484739    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-j674w" event={"ID":"3e8e2284-b089-4525-80c5-93353f07252f","Type":"ContainerStarted","Data":"c4dd842dff844dbd361a0a237415596d13e4e10987d7298df0281f9d476be5d7"}
Sep 16 14:16:23.512696 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:23.512641    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6qpvt" podStartSLOduration=130.526341427 podStartE2EDuration="2m11.512625164s" totalImagesPullingTime="986.283737ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:16:23.511525426 +0000 UTC m=+132.115150460" watchObservedRunningTime="2026-09-16 14:16:23.512625164 +0000 UTC m=+132.116250196"
Sep 16 14:16:27.418709 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:27.418658    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-666f668987-s624v" podUID="0db79e86-e959-4025-a1cb-4c54d27e2d2f" containerName="console" containerID="cri-o://4b79c2a8f18bf06d3e71e09b2635661816004f48fe237b3bcc63e51ead5f2cce" gracePeriod=15
Sep 16 14:16:27.688203 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:27.688132    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-666f668987-s624v"
Sep 16 14:16:27.777213 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:27.777184    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0db79e86-e959-4025-a1cb-4c54d27e2d2f-service-ca\") pod \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\" (UID: \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\") "
Sep 16 14:16:27.777213 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:27.777218    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltl9g\" (UniqueName: \"kubernetes.io/projected/0db79e86-e959-4025-a1cb-4c54d27e2d2f-kube-api-access-ltl9g\") pod \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\" (UID: \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\") "
Sep 16 14:16:27.777511 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:27.777244    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0db79e86-e959-4025-a1cb-4c54d27e2d2f-console-oauth-config\") pod \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\" (UID: \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\") "
Sep 16 14:16:27.777511 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:27.777438    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0db79e86-e959-4025-a1cb-4c54d27e2d2f-console-config\") pod \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\" (UID: \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\") "
Sep 16 14:16:27.777511 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:27.777483    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0db79e86-e959-4025-a1cb-4c54d27e2d2f-console-serving-cert\") pod \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\" (UID: \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\") "
Sep 16 14:16:27.777511 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:27.777510    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0db79e86-e959-4025-a1cb-4c54d27e2d2f-oauth-serving-cert\") pod \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\" (UID: \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\") "
Sep 16 14:16:27.777708 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:27.777564    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0db79e86-e959-4025-a1cb-4c54d27e2d2f-trusted-ca-bundle\") pod \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\" (UID: \"0db79e86-e959-4025-a1cb-4c54d27e2d2f\") "
Sep 16 14:16:27.777835 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:27.777805    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0db79e86-e959-4025-a1cb-4c54d27e2d2f-service-ca" (OuterVolumeSpecName: "service-ca") pod "0db79e86-e959-4025-a1cb-4c54d27e2d2f" (UID: "0db79e86-e959-4025-a1cb-4c54d27e2d2f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:16:27.778041 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:27.778014    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0db79e86-e959-4025-a1cb-4c54d27e2d2f-console-config" (OuterVolumeSpecName: "console-config") pod "0db79e86-e959-4025-a1cb-4c54d27e2d2f" (UID: "0db79e86-e959-4025-a1cb-4c54d27e2d2f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:16:27.778112 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:27.778094    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0db79e86-e959-4025-a1cb-4c54d27e2d2f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0db79e86-e959-4025-a1cb-4c54d27e2d2f" (UID: "0db79e86-e959-4025-a1cb-4c54d27e2d2f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:16:27.778190 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:27.778162    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0db79e86-e959-4025-a1cb-4c54d27e2d2f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0db79e86-e959-4025-a1cb-4c54d27e2d2f" (UID: "0db79e86-e959-4025-a1cb-4c54d27e2d2f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:16:27.779665 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:27.779647    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0db79e86-e959-4025-a1cb-4c54d27e2d2f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0db79e86-e959-4025-a1cb-4c54d27e2d2f" (UID: "0db79e86-e959-4025-a1cb-4c54d27e2d2f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:16:27.780156 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:27.780133    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0db79e86-e959-4025-a1cb-4c54d27e2d2f-kube-api-access-ltl9g" (OuterVolumeSpecName: "kube-api-access-ltl9g") pod "0db79e86-e959-4025-a1cb-4c54d27e2d2f" (UID: "0db79e86-e959-4025-a1cb-4c54d27e2d2f"). InnerVolumeSpecName "kube-api-access-ltl9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:16:27.780156 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:27.780125    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0db79e86-e959-4025-a1cb-4c54d27e2d2f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0db79e86-e959-4025-a1cb-4c54d27e2d2f" (UID: "0db79e86-e959-4025-a1cb-4c54d27e2d2f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:16:27.878158 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:27.878123    2587 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0db79e86-e959-4025-a1cb-4c54d27e2d2f-console-config\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:16:27.878158 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:27.878153    2587 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0db79e86-e959-4025-a1cb-4c54d27e2d2f-console-serving-cert\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:16:27.878158 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:27.878163    2587 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0db79e86-e959-4025-a1cb-4c54d27e2d2f-oauth-serving-cert\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:16:27.878381 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:27.878172    2587 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0db79e86-e959-4025-a1cb-4c54d27e2d2f-trusted-ca-bundle\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:16:27.878381 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:27.878182    2587 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0db79e86-e959-4025-a1cb-4c54d27e2d2f-service-ca\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:16:27.878381 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:27.878190    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltl9g\" (UniqueName: \"kubernetes.io/projected/0db79e86-e959-4025-a1cb-4c54d27e2d2f-kube-api-access-ltl9g\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:16:27.878381 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:27.878198    2587 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0db79e86-e959-4025-a1cb-4c54d27e2d2f-console-oauth-config\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:16:28.502564 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:28.502530    2587 generic.go:356] "Generic (PLEG): container finished" podID="0db79e86-e959-4025-a1cb-4c54d27e2d2f" containerID="4b79c2a8f18bf06d3e71e09b2635661816004f48fe237b3bcc63e51ead5f2cce" exitCode=0
Sep 16 14:16:28.503052 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:28.502573    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-666f668987-s624v"
Sep 16 14:16:28.503052 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:28.502606    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-666f668987-s624v" event={"ID":"0db79e86-e959-4025-a1cb-4c54d27e2d2f","Type":"ContainerDied","Data":"4b79c2a8f18bf06d3e71e09b2635661816004f48fe237b3bcc63e51ead5f2cce"}
Sep 16 14:16:28.503052 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:28.502648    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-666f668987-s624v" event={"ID":"0db79e86-e959-4025-a1cb-4c54d27e2d2f","Type":"ContainerDied","Data":"73fa6f99604ef3303eff33c9bd497b729c8007788fcdc85f87611f5c336fc8b0"}
Sep 16 14:16:28.503052 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:28.502671    2587 scope.go:117] "RemoveContainer" containerID="4b79c2a8f18bf06d3e71e09b2635661816004f48fe237b3bcc63e51ead5f2cce"
Sep 16 14:16:28.511246 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:28.511226    2587 scope.go:117] "RemoveContainer" containerID="4b79c2a8f18bf06d3e71e09b2635661816004f48fe237b3bcc63e51ead5f2cce"
Sep 16 14:16:28.511609 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:16:28.511585    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b79c2a8f18bf06d3e71e09b2635661816004f48fe237b3bcc63e51ead5f2cce\": container with ID starting with 4b79c2a8f18bf06d3e71e09b2635661816004f48fe237b3bcc63e51ead5f2cce not found: ID does not exist" containerID="4b79c2a8f18bf06d3e71e09b2635661816004f48fe237b3bcc63e51ead5f2cce"
Sep 16 14:16:28.511665 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:28.511622    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b79c2a8f18bf06d3e71e09b2635661816004f48fe237b3bcc63e51ead5f2cce"} err="failed to get container status \"4b79c2a8f18bf06d3e71e09b2635661816004f48fe237b3bcc63e51ead5f2cce\": rpc error: code = NotFound desc = could not find container \"4b79c2a8f18bf06d3e71e09b2635661816004f48fe237b3bcc63e51ead5f2cce\": container with ID starting with 4b79c2a8f18bf06d3e71e09b2635661816004f48fe237b3bcc63e51ead5f2cce not found: ID does not exist"
Sep 16 14:16:28.529687 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:28.529655    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-666f668987-s624v"]
Sep 16 14:16:28.533673 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:28.533645    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-666f668987-s624v"]
Sep 16 14:16:29.901660 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:29.901632    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0db79e86-e959-4025-a1cb-4c54d27e2d2f" path="/var/lib/kubelet/pods/0db79e86-e959-4025-a1cb-4c54d27e2d2f/volumes"
Sep 16 14:16:33.523140 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:33.523098    2587 generic.go:356] "Generic (PLEG): container finished" podID="68975d6e-a86b-480f-94ba-b657b5b09fd1" containerID="a59b2311394f3abc12ac2393cd265e19213f0e062228642d0b4a10ff20c1aa63" exitCode=0
Sep 16 14:16:33.523542 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:33.523162    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w82jn" event={"ID":"68975d6e-a86b-480f-94ba-b657b5b09fd1","Type":"ContainerDied","Data":"a59b2311394f3abc12ac2393cd265e19213f0e062228642d0b4a10ff20c1aa63"}
Sep 16 14:16:33.523542 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:33.523527    2587 scope.go:117] "RemoveContainer" containerID="a59b2311394f3abc12ac2393cd265e19213f0e062228642d0b4a10ff20c1aa63"
Sep 16 14:16:34.533136 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:34.532270    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-w82jn" event={"ID":"68975d6e-a86b-480f-94ba-b657b5b09fd1","Type":"ContainerStarted","Data":"8051dad5197b51d9fcca803774c99426f449c6237f7d49e16bcdaaf27113dcf1"}
Sep 16 14:16:39.870469 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:39.870442    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-78569c95d-l6whj"
Sep 16 14:16:39.874344 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:16:39.874319    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-78569c95d-l6whj"
Sep 16 14:17:22.890322 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:22.890278    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-848dcff5fd-cmgl4"]
Sep 16 14:17:22.890750 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:22.890568    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0db79e86-e959-4025-a1cb-4c54d27e2d2f" containerName="console"
Sep 16 14:17:22.890750 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:22.890582    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="0db79e86-e959-4025-a1cb-4c54d27e2d2f" containerName="console"
Sep 16 14:17:22.890750 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:22.890600    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c16289f-192f-4eda-a7e5-3c8aef6f14e6" containerName="registry"
Sep 16 14:17:22.890750 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:22.890605    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c16289f-192f-4eda-a7e5-3c8aef6f14e6" containerName="registry"
Sep 16 14:17:22.890750 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:22.890648    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c16289f-192f-4eda-a7e5-3c8aef6f14e6" containerName="registry"
Sep 16 14:17:22.890750 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:22.890656    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="0db79e86-e959-4025-a1cb-4c54d27e2d2f" containerName="console"
Sep 16 14:17:22.894801 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:22.894783    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-848dcff5fd-cmgl4"
Sep 16 14:17:22.897647 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:22.897621    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 14:17:22.898835 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:22.898695    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 14:17:22.898835 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:22.898708    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 14:17:22.898835 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:22.898717    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 14:17:22.898835 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:22.898717    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 14:17:22.898835 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:22.898820    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-gqfcf\""
Sep 16 14:17:22.899098 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:22.898914    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 14:17:22.899132 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:22.899097    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 14:17:22.902948 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:22.902929    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 14:17:22.906066 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:22.906041    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-848dcff5fd-cmgl4"]
Sep 16 14:17:23.020779 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:23.020741    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbdvc\" (UniqueName: \"kubernetes.io/projected/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-kube-api-access-vbdvc\") pod \"console-848dcff5fd-cmgl4\" (UID: \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\") " pod="openshift-console/console-848dcff5fd-cmgl4"
Sep 16 14:17:23.020779 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:23.020781    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-console-config\") pod \"console-848dcff5fd-cmgl4\" (UID: \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\") " pod="openshift-console/console-848dcff5fd-cmgl4"
Sep 16 14:17:23.020986 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:23.020803    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-trusted-ca-bundle\") pod \"console-848dcff5fd-cmgl4\" (UID: \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\") " pod="openshift-console/console-848dcff5fd-cmgl4"
Sep 16 14:17:23.020986 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:23.020888    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-console-serving-cert\") pod \"console-848dcff5fd-cmgl4\" (UID: \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\") " pod="openshift-console/console-848dcff5fd-cmgl4"
Sep 16 14:17:23.020986 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:23.020922    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-console-oauth-config\") pod \"console-848dcff5fd-cmgl4\" (UID: \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\") " pod="openshift-console/console-848dcff5fd-cmgl4"
Sep 16 14:17:23.020986 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:23.020952    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-oauth-serving-cert\") pod \"console-848dcff5fd-cmgl4\" (UID: \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\") " pod="openshift-console/console-848dcff5fd-cmgl4"
Sep 16 14:17:23.021105 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:23.021012    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-service-ca\") pod \"console-848dcff5fd-cmgl4\" (UID: \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\") " pod="openshift-console/console-848dcff5fd-cmgl4"
Sep 16 14:17:23.121682 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:23.121647    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-oauth-serving-cert\") pod \"console-848dcff5fd-cmgl4\" (UID: \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\") " pod="openshift-console/console-848dcff5fd-cmgl4"
Sep 16 14:17:23.121851 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:23.121691    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-service-ca\") pod \"console-848dcff5fd-cmgl4\" (UID: \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\") " pod="openshift-console/console-848dcff5fd-cmgl4"
Sep 16 14:17:23.121851 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:23.121728    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbdvc\" (UniqueName: \"kubernetes.io/projected/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-kube-api-access-vbdvc\") pod \"console-848dcff5fd-cmgl4\" (UID: \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\") " pod="openshift-console/console-848dcff5fd-cmgl4"
Sep 16 14:17:23.121851 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:23.121750    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-console-config\") pod \"console-848dcff5fd-cmgl4\" (UID: \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\") " pod="openshift-console/console-848dcff5fd-cmgl4"
Sep 16 14:17:23.121851 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:23.121768    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-trusted-ca-bundle\") pod \"console-848dcff5fd-cmgl4\" (UID: \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\") " pod="openshift-console/console-848dcff5fd-cmgl4"
Sep 16 14:17:23.121851 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:23.121790    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-console-serving-cert\") pod \"console-848dcff5fd-cmgl4\" (UID: \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\") " pod="openshift-console/console-848dcff5fd-cmgl4"
Sep 16 14:17:23.122180 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:23.122135    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-console-oauth-config\") pod \"console-848dcff5fd-cmgl4\" (UID: \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\") " pod="openshift-console/console-848dcff5fd-cmgl4"
Sep 16 14:17:23.122598 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:23.122528    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-oauth-serving-cert\") pod \"console-848dcff5fd-cmgl4\" (UID: \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\") " pod="openshift-console/console-848dcff5fd-cmgl4"
Sep 16 14:17:23.122747 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:23.122528    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-service-ca\") pod \"console-848dcff5fd-cmgl4\" (UID: \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\") " pod="openshift-console/console-848dcff5fd-cmgl4"
Sep 16 14:17:23.127434 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:23.122885    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-console-config\") pod \"console-848dcff5fd-cmgl4\" (UID: \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\") " pod="openshift-console/console-848dcff5fd-cmgl4"
Sep 16 14:17:23.127434 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:23.123151    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-trusted-ca-bundle\") pod \"console-848dcff5fd-cmgl4\" (UID: \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\") " pod="openshift-console/console-848dcff5fd-cmgl4"
Sep 16 14:17:23.127434 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:23.125071    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-console-oauth-config\") pod \"console-848dcff5fd-cmgl4\" (UID: \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\") " pod="openshift-console/console-848dcff5fd-cmgl4"
Sep 16 14:17:23.128055 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:23.128030    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-console-serving-cert\") pod \"console-848dcff5fd-cmgl4\" (UID: \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\") " pod="openshift-console/console-848dcff5fd-cmgl4"
Sep 16 14:17:23.130760 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:23.130738    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbdvc\" (UniqueName: \"kubernetes.io/projected/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-kube-api-access-vbdvc\") pod \"console-848dcff5fd-cmgl4\" (UID: \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\") " pod="openshift-console/console-848dcff5fd-cmgl4"
Sep 16 14:17:23.205782 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:23.205744    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-848dcff5fd-cmgl4"
Sep 16 14:17:23.478890 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:23.478805    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-848dcff5fd-cmgl4"]
Sep 16 14:17:23.483624 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:17:23.483585    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46d5806_3c6b_49c0_aeb1_8134cafa87ff.slice/crio-e474575380b6574ad69e8264cfb54c7652cd355c83c0e82b29c8808e959772fc WatchSource:0}: Error finding container e474575380b6574ad69e8264cfb54c7652cd355c83c0e82b29c8808e959772fc: Status 404 returned error can't find the container with id e474575380b6574ad69e8264cfb54c7652cd355c83c0e82b29c8808e959772fc
Sep 16 14:17:23.722853 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:23.722819    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-848dcff5fd-cmgl4" event={"ID":"f46d5806-3c6b-49c0-aeb1-8134cafa87ff","Type":"ContainerStarted","Data":"3a963dcfdcb9e9a4e7511f98503f4fa6b9de4fe0afe0ea3012c29ae32b400fb8"}
Sep 16 14:17:23.723054 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:23.722871    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-848dcff5fd-cmgl4" event={"ID":"f46d5806-3c6b-49c0-aeb1-8134cafa87ff","Type":"ContainerStarted","Data":"e474575380b6574ad69e8264cfb54c7652cd355c83c0e82b29c8808e959772fc"}
Sep 16 14:17:23.742532 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:23.742420    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-848dcff5fd-cmgl4" podStartSLOduration=1.742383062 podStartE2EDuration="1.742383062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:17:23.74105008 +0000 UTC m=+192.344675112" watchObservedRunningTime="2026-09-16 14:17:23.742383062 +0000 UTC m=+192.346008094"
Sep 16 14:17:32.402407 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:32.402356    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-848dcff5fd-cmgl4"]
Sep 16 14:17:32.432825 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:32.432788    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78db8d7859-njbk9"]
Sep 16 14:17:32.435990 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:32.435964    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:17:32.448644 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:32.448617    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78db8d7859-njbk9"]
Sep 16 14:17:32.503557 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:32.503519    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-console-serving-cert\") pod \"console-78db8d7859-njbk9\" (UID: \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\") " pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:17:32.503557 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:32.503555    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-console-oauth-config\") pod \"console-78db8d7859-njbk9\" (UID: \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\") " pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:17:32.503755 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:32.503596    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkxt6\" (UniqueName: \"kubernetes.io/projected/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-kube-api-access-rkxt6\") pod \"console-78db8d7859-njbk9\" (UID: \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\") " pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:17:32.503755 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:32.503625    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-trusted-ca-bundle\") pod \"console-78db8d7859-njbk9\" (UID: \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\") " pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:17:32.503755 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:32.503650    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-service-ca\") pod \"console-78db8d7859-njbk9\" (UID: \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\") " pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:17:32.503755 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:32.503686    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-oauth-serving-cert\") pod \"console-78db8d7859-njbk9\" (UID: \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\") " pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:17:32.503755 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:32.503718    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-console-config\") pod \"console-78db8d7859-njbk9\" (UID: \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\") " pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:17:32.604988 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:32.604955    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-trusted-ca-bundle\") pod \"console-78db8d7859-njbk9\" (UID: \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\") " pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:17:32.605141 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:32.605000    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-service-ca\") pod \"console-78db8d7859-njbk9\" (UID: \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\") " pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:17:32.605141 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:32.605019    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-oauth-serving-cert\") pod \"console-78db8d7859-njbk9\" (UID: \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\") " pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:17:32.605141 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:32.605036    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-console-config\") pod \"console-78db8d7859-njbk9\" (UID: \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\") " pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:17:32.605141 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:32.605078    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-console-serving-cert\") pod \"console-78db8d7859-njbk9\" (UID: \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\") " pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:17:32.605141 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:32.605097    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-console-oauth-config\") pod \"console-78db8d7859-njbk9\" (UID: \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\") " pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:17:32.605368 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:32.605146    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkxt6\" (UniqueName: \"kubernetes.io/projected/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-kube-api-access-rkxt6\") pod \"console-78db8d7859-njbk9\" (UID: \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\") " pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:17:32.605779 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:32.605750    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-oauth-serving-cert\") pod \"console-78db8d7859-njbk9\" (UID: \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\") " pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:17:32.605909 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:32.605869    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-service-ca\") pod \"console-78db8d7859-njbk9\" (UID: \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\") " pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:17:32.606204 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:32.606179    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-console-config\") pod \"console-78db8d7859-njbk9\" (UID: \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\") " pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:17:32.606240 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:32.606219    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-trusted-ca-bundle\") pod \"console-78db8d7859-njbk9\" (UID: \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\") " pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:17:32.607757 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:32.607735    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-console-oauth-config\") pod \"console-78db8d7859-njbk9\" (UID: \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\") " pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:17:32.607961 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:32.607941    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-console-serving-cert\") pod \"console-78db8d7859-njbk9\" (UID: \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\") " pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:17:32.619602 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:32.619574    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkxt6\" (UniqueName: \"kubernetes.io/projected/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-kube-api-access-rkxt6\") pod \"console-78db8d7859-njbk9\" (UID: \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\") " pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:17:32.745630 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:32.745525    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:17:33.023872 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:33.023797    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78db8d7859-njbk9"]
Sep 16 14:17:33.026683 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:17:33.026657    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cfe4ef5_e18d_47e0_ab03_4ca3df3d98dc.slice/crio-0b1fd45aa8a5b4a3b076afd800cc01bc876d072213ae9ed6e0f048c807d311f5 WatchSource:0}: Error finding container 0b1fd45aa8a5b4a3b076afd800cc01bc876d072213ae9ed6e0f048c807d311f5: Status 404 returned error can't find the container with id 0b1fd45aa8a5b4a3b076afd800cc01bc876d072213ae9ed6e0f048c807d311f5
Sep 16 14:17:33.206472 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:33.206433    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-848dcff5fd-cmgl4"
Sep 16 14:17:33.762133 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:33.762097    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78db8d7859-njbk9" event={"ID":"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc","Type":"ContainerStarted","Data":"280834a559a7f9414ad9dfd07b6a9492c6a027a4edb75ed57e66ff06ad291e3d"}
Sep 16 14:17:33.762550 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:33.762176    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78db8d7859-njbk9" event={"ID":"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc","Type":"ContainerStarted","Data":"0b1fd45aa8a5b4a3b076afd800cc01bc876d072213ae9ed6e0f048c807d311f5"}
Sep 16 14:17:33.780184 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:33.780125    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78db8d7859-njbk9" podStartSLOduration=1.780108343 podStartE2EDuration="1.780108343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:17:33.779600789 +0000 UTC m=+202.383225846" watchObservedRunningTime="2026-09-16 14:17:33.780108343 +0000 UTC m=+202.383733375"
Sep 16 14:17:42.746686 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:42.746646    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:17:42.747142 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:42.746719    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:17:42.751473 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:42.751446    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:17:42.797949 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:42.797924    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:17:57.421562 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.421492    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-848dcff5fd-cmgl4" podUID="f46d5806-3c6b-49c0-aeb1-8134cafa87ff" containerName="console" containerID="cri-o://3a963dcfdcb9e9a4e7511f98503f4fa6b9de4fe0afe0ea3012c29ae32b400fb8" gracePeriod=15
Sep 16 14:17:57.661592 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.661568    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-848dcff5fd-cmgl4"
Sep 16 14:17:57.717058 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.716969    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-console-config\") pod \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\" (UID: \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\") "
Sep 16 14:17:57.717058 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.717016    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-console-serving-cert\") pod \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\" (UID: \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\") "
Sep 16 14:17:57.717058 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.717049    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-trusted-ca-bundle\") pod \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\" (UID: \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\") "
Sep 16 14:17:57.717304 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.717113    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-oauth-serving-cert\") pod \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\" (UID: \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\") "
Sep 16 14:17:57.717304 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.717139    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-console-oauth-config\") pod \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\" (UID: \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\") "
Sep 16 14:17:57.717304 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.717183    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbdvc\" (UniqueName: \"kubernetes.io/projected/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-kube-api-access-vbdvc\") pod \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\" (UID: \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\") "
Sep 16 14:17:57.717304 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.717217    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-service-ca\") pod \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\" (UID: \"f46d5806-3c6b-49c0-aeb1-8134cafa87ff\") "
Sep 16 14:17:57.717578 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.717548    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-console-config" (OuterVolumeSpecName: "console-config") pod "f46d5806-3c6b-49c0-aeb1-8134cafa87ff" (UID: "f46d5806-3c6b-49c0-aeb1-8134cafa87ff"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:17:57.717682 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.717649    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f46d5806-3c6b-49c0-aeb1-8134cafa87ff" (UID: "f46d5806-3c6b-49c0-aeb1-8134cafa87ff"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:17:57.717735 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.717713    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-service-ca" (OuterVolumeSpecName: "service-ca") pod "f46d5806-3c6b-49c0-aeb1-8134cafa87ff" (UID: "f46d5806-3c6b-49c0-aeb1-8134cafa87ff"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:17:57.717790 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.717765    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f46d5806-3c6b-49c0-aeb1-8134cafa87ff" (UID: "f46d5806-3c6b-49c0-aeb1-8134cafa87ff"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:17:57.719455 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.719426    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f46d5806-3c6b-49c0-aeb1-8134cafa87ff" (UID: "f46d5806-3c6b-49c0-aeb1-8134cafa87ff"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:17:57.720164 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.720136    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f46d5806-3c6b-49c0-aeb1-8134cafa87ff" (UID: "f46d5806-3c6b-49c0-aeb1-8134cafa87ff"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:17:57.720164 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.720146    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-kube-api-access-vbdvc" (OuterVolumeSpecName: "kube-api-access-vbdvc") pod "f46d5806-3c6b-49c0-aeb1-8134cafa87ff" (UID: "f46d5806-3c6b-49c0-aeb1-8134cafa87ff"). InnerVolumeSpecName "kube-api-access-vbdvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:17:57.818016 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.817972    2587 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-console-config\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:17:57.818016 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.818005    2587 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-console-serving-cert\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:17:57.818016 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.818017    2587 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-trusted-ca-bundle\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:17:57.818016 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.818026    2587 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-oauth-serving-cert\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:17:57.818263 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.818034    2587 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-console-oauth-config\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:17:57.818263 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.818043    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbdvc\" (UniqueName: \"kubernetes.io/projected/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-kube-api-access-vbdvc\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:17:57.818263 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.818052    2587 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f46d5806-3c6b-49c0-aeb1-8134cafa87ff-service-ca\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:17:57.852351 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.852313    2587 generic.go:356] "Generic (PLEG): container finished" podID="f46d5806-3c6b-49c0-aeb1-8134cafa87ff" containerID="3a963dcfdcb9e9a4e7511f98503f4fa6b9de4fe0afe0ea3012c29ae32b400fb8" exitCode=0
Sep 16 14:17:57.852535 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.852360    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-848dcff5fd-cmgl4"
Sep 16 14:17:57.852535 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.852431    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-848dcff5fd-cmgl4" event={"ID":"f46d5806-3c6b-49c0-aeb1-8134cafa87ff","Type":"ContainerDied","Data":"3a963dcfdcb9e9a4e7511f98503f4fa6b9de4fe0afe0ea3012c29ae32b400fb8"}
Sep 16 14:17:57.852535 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.852523    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-848dcff5fd-cmgl4" event={"ID":"f46d5806-3c6b-49c0-aeb1-8134cafa87ff","Type":"ContainerDied","Data":"e474575380b6574ad69e8264cfb54c7652cd355c83c0e82b29c8808e959772fc"}
Sep 16 14:17:57.852702 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.852550    2587 scope.go:117] "RemoveContainer" containerID="3a963dcfdcb9e9a4e7511f98503f4fa6b9de4fe0afe0ea3012c29ae32b400fb8"
Sep 16 14:17:57.861430 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.861413    2587 scope.go:117] "RemoveContainer" containerID="3a963dcfdcb9e9a4e7511f98503f4fa6b9de4fe0afe0ea3012c29ae32b400fb8"
Sep 16 14:17:57.861819 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:17:57.861797    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a963dcfdcb9e9a4e7511f98503f4fa6b9de4fe0afe0ea3012c29ae32b400fb8\": container with ID starting with 3a963dcfdcb9e9a4e7511f98503f4fa6b9de4fe0afe0ea3012c29ae32b400fb8 not found: ID does not exist" containerID="3a963dcfdcb9e9a4e7511f98503f4fa6b9de4fe0afe0ea3012c29ae32b400fb8"
Sep 16 14:17:57.861870 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.861836    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a963dcfdcb9e9a4e7511f98503f4fa6b9de4fe0afe0ea3012c29ae32b400fb8"} err="failed to get container status \"3a963dcfdcb9e9a4e7511f98503f4fa6b9de4fe0afe0ea3012c29ae32b400fb8\": rpc error: code = NotFound desc = could not find container \"3a963dcfdcb9e9a4e7511f98503f4fa6b9de4fe0afe0ea3012c29ae32b400fb8\": container with ID starting with 3a963dcfdcb9e9a4e7511f98503f4fa6b9de4fe0afe0ea3012c29ae32b400fb8 not found: ID does not exist"
Sep 16 14:17:57.873802 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.873773    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-848dcff5fd-cmgl4"]
Sep 16 14:17:57.877310 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.877287    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-848dcff5fd-cmgl4"]
Sep 16 14:17:57.902163 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:17:57.902131    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f46d5806-3c6b-49c0-aeb1-8134cafa87ff" path="/var/lib/kubelet/pods/f46d5806-3c6b-49c0-aeb1-8134cafa87ff/volumes"
Sep 16 14:19:08.856277 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:08.856242    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-tg856"]
Sep 16 14:19:08.856707 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:08.856570    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f46d5806-3c6b-49c0-aeb1-8134cafa87ff" containerName="console"
Sep 16 14:19:08.856707 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:08.856582    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="f46d5806-3c6b-49c0-aeb1-8134cafa87ff" containerName="console"
Sep 16 14:19:08.856707 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:08.856634    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="f46d5806-3c6b-49c0-aeb1-8134cafa87ff" containerName="console"
Sep 16 14:19:08.859355 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:08.859341    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tg856"
Sep 16 14:19:08.861986 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:08.861966    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 14:19:08.862874 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:08.862856    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-q5bg5\""
Sep 16 14:19:08.862977 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:08.862856    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 14:19:08.867295 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:08.867271    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-tg856"]
Sep 16 14:19:08.979901 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:08.979855    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4096783-a575-4801-ae38-bf2bade65fa7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-tg856\" (UID: \"c4096783-a575-4801-ae38-bf2bade65fa7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tg856"
Sep 16 14:19:08.980093 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:08.979923    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crbp6\" (UniqueName: \"kubernetes.io/projected/c4096783-a575-4801-ae38-bf2bade65fa7-kube-api-access-crbp6\") pod \"cert-manager-cainjector-559cdd8c87-tg856\" (UID: \"c4096783-a575-4801-ae38-bf2bade65fa7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tg856"
Sep 16 14:19:09.080718 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:09.080674    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4096783-a575-4801-ae38-bf2bade65fa7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-tg856\" (UID: \"c4096783-a575-4801-ae38-bf2bade65fa7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tg856"
Sep 16 14:19:09.080905 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:09.080751    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crbp6\" (UniqueName: \"kubernetes.io/projected/c4096783-a575-4801-ae38-bf2bade65fa7-kube-api-access-crbp6\") pod \"cert-manager-cainjector-559cdd8c87-tg856\" (UID: \"c4096783-a575-4801-ae38-bf2bade65fa7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tg856"
Sep 16 14:19:09.088763 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:09.088733    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4096783-a575-4801-ae38-bf2bade65fa7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-tg856\" (UID: \"c4096783-a575-4801-ae38-bf2bade65fa7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tg856"
Sep 16 14:19:09.089282 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:09.089264    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crbp6\" (UniqueName: \"kubernetes.io/projected/c4096783-a575-4801-ae38-bf2bade65fa7-kube-api-access-crbp6\") pod \"cert-manager-cainjector-559cdd8c87-tg856\" (UID: \"c4096783-a575-4801-ae38-bf2bade65fa7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tg856"
Sep 16 14:19:09.176699 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:09.176669    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tg856"
Sep 16 14:19:09.445428 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:09.445385    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-tg856"]
Sep 16 14:19:09.447500 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:19:09.447473    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4096783_a575_4801_ae38_bf2bade65fa7.slice/crio-bf1d49b8c1bf8dccf7d878c1b232d255636ff4a1e03182211331c935b64d6210 WatchSource:0}: Error finding container bf1d49b8c1bf8dccf7d878c1b232d255636ff4a1e03182211331c935b64d6210: Status 404 returned error can't find the container with id bf1d49b8c1bf8dccf7d878c1b232d255636ff4a1e03182211331c935b64d6210
Sep 16 14:19:10.121940 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:10.121907    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tg856" event={"ID":"c4096783-a575-4801-ae38-bf2bade65fa7","Type":"ContainerStarted","Data":"bf1d49b8c1bf8dccf7d878c1b232d255636ff4a1e03182211331c935b64d6210"}
Sep 16 14:19:11.041699 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:11.041666    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-wrrl8"]
Sep 16 14:19:11.050619 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:11.050559    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-wrrl8"
Sep 16 14:19:11.054484 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:11.054268    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-qpt8k\""
Sep 16 14:19:11.056274 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:11.056231    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-wrrl8"]
Sep 16 14:19:11.198500 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:11.198465    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/761eedfc-d59a-440a-bbf5-f8ad991290d2-bound-sa-token\") pod \"cert-manager-55b4d954d5-wrrl8\" (UID: \"761eedfc-d59a-440a-bbf5-f8ad991290d2\") " pod="cert-manager/cert-manager-55b4d954d5-wrrl8"
Sep 16 14:19:11.198919 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:11.198525    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vczck\" (UniqueName: \"kubernetes.io/projected/761eedfc-d59a-440a-bbf5-f8ad991290d2-kube-api-access-vczck\") pod \"cert-manager-55b4d954d5-wrrl8\" (UID: \"761eedfc-d59a-440a-bbf5-f8ad991290d2\") " pod="cert-manager/cert-manager-55b4d954d5-wrrl8"
Sep 16 14:19:11.299534 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:11.299442    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/761eedfc-d59a-440a-bbf5-f8ad991290d2-bound-sa-token\") pod \"cert-manager-55b4d954d5-wrrl8\" (UID: \"761eedfc-d59a-440a-bbf5-f8ad991290d2\") " pod="cert-manager/cert-manager-55b4d954d5-wrrl8"
Sep 16 14:19:11.299534 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:11.299510    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vczck\" (UniqueName: \"kubernetes.io/projected/761eedfc-d59a-440a-bbf5-f8ad991290d2-kube-api-access-vczck\") pod \"cert-manager-55b4d954d5-wrrl8\" (UID: \"761eedfc-d59a-440a-bbf5-f8ad991290d2\") " pod="cert-manager/cert-manager-55b4d954d5-wrrl8"
Sep 16 14:19:11.308960 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:11.308893    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/761eedfc-d59a-440a-bbf5-f8ad991290d2-bound-sa-token\") pod \"cert-manager-55b4d954d5-wrrl8\" (UID: \"761eedfc-d59a-440a-bbf5-f8ad991290d2\") " pod="cert-manager/cert-manager-55b4d954d5-wrrl8"
Sep 16 14:19:11.309172 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:11.309143    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vczck\" (UniqueName: \"kubernetes.io/projected/761eedfc-d59a-440a-bbf5-f8ad991290d2-kube-api-access-vczck\") pod \"cert-manager-55b4d954d5-wrrl8\" (UID: \"761eedfc-d59a-440a-bbf5-f8ad991290d2\") " pod="cert-manager/cert-manager-55b4d954d5-wrrl8"
Sep 16 14:19:11.364008 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:11.363966    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-wrrl8"
Sep 16 14:19:11.911800 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:11.909797    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-wrrl8"]
Sep 16 14:19:12.620428 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:19:12.620374    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod761eedfc_d59a_440a_bbf5_f8ad991290d2.slice/crio-c21e741f0ed04940e0387b7c7dbce4a84bdeeb4a7b69866e37ead8b270ad670e WatchSource:0}: Error finding container c21e741f0ed04940e0387b7c7dbce4a84bdeeb4a7b69866e37ead8b270ad670e: Status 404 returned error can't find the container with id c21e741f0ed04940e0387b7c7dbce4a84bdeeb4a7b69866e37ead8b270ad670e
Sep 16 14:19:12.643711 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:12.643688    2587 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 14:19:13.136944 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:13.136907    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-wrrl8" event={"ID":"761eedfc-d59a-440a-bbf5-f8ad991290d2","Type":"ContainerStarted","Data":"c21e741f0ed04940e0387b7c7dbce4a84bdeeb4a7b69866e37ead8b270ad670e"}
Sep 16 14:19:13.138502 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:13.138473    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tg856" event={"ID":"c4096783-a575-4801-ae38-bf2bade65fa7","Type":"ContainerStarted","Data":"d21276051ed749640ff05930e7dc1f66e76642bb3fc8a3c94044d22770396fa3"}
Sep 16 14:19:13.155333 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:13.155280    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tg856" podStartSLOduration=1.922754652 podStartE2EDuration="5.155264136s" totalImagesPullingTime="3.232509484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:19:13.153298444 +0000 UTC m=+301.756923478" watchObservedRunningTime="2026-09-16 14:19:13.155264136 +0000 UTC m=+301.758889168"
Sep 16 14:19:14.143600 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:14.143564    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-wrrl8" event={"ID":"761eedfc-d59a-440a-bbf5-f8ad991290d2","Type":"ContainerStarted","Data":"cde35b1c6a6ea96db97b04716025eacfe3b1cd8ac1a5d3b82435cb285bbb4464"}
Sep 16 14:19:14.160862 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:14.160800    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-wrrl8" podStartSLOduration=2.4571453659999998 podStartE2EDuration="3.160784564s" totalImagesPullingTime="703.639198ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:19:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:19:14.160010159 +0000 UTC m=+302.763635204" watchObservedRunningTime="2026-09-16 14:19:14.160784564 +0000 UTC m=+302.764409596"
Sep 16 14:19:38.030705 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:38.030667    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68b85944db-9bv9d"]
Sep 16 14:19:38.033469 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:38.033451    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-9bv9d"
Sep 16 14:19:38.036339 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:38.036312    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 14:19:38.036339 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:38.036317    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 14:19:38.036559 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:38.036357    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 14:19:38.036559 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:38.036510    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 14:19:38.036559 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:38.036509    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-k7vhv\""
Sep 16 14:19:38.052022 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:38.051997    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68b85944db-9bv9d"]
Sep 16 14:19:38.125251 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:38.125216    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/910e25fa-e7b6-4b60-a5f0-255d7fc3ca65-tmp\") pod \"opendatahub-operator-controller-manager-68b85944db-9bv9d\" (UID: \"910e25fa-e7b6-4b60-a5f0-255d7fc3ca65\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-9bv9d"
Sep 16 14:19:38.125484 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:38.125259    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfl95\" (UniqueName: \"kubernetes.io/projected/910e25fa-e7b6-4b60-a5f0-255d7fc3ca65-kube-api-access-jfl95\") pod \"opendatahub-operator-controller-manager-68b85944db-9bv9d\" (UID: \"910e25fa-e7b6-4b60-a5f0-255d7fc3ca65\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-9bv9d"
Sep 16 14:19:38.125484 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:38.125346    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/910e25fa-e7b6-4b60-a5f0-255d7fc3ca65-webhook-cert\") pod \"opendatahub-operator-controller-manager-68b85944db-9bv9d\" (UID: \"910e25fa-e7b6-4b60-a5f0-255d7fc3ca65\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-9bv9d"
Sep 16 14:19:38.125484 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:38.125432    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/910e25fa-e7b6-4b60-a5f0-255d7fc3ca65-manifests\") pod \"opendatahub-operator-controller-manager-68b85944db-9bv9d\" (UID: \"910e25fa-e7b6-4b60-a5f0-255d7fc3ca65\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-9bv9d"
Sep 16 14:19:38.125611 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:38.125496    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/910e25fa-e7b6-4b60-a5f0-255d7fc3ca65-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68b85944db-9bv9d\" (UID: \"910e25fa-e7b6-4b60-a5f0-255d7fc3ca65\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-9bv9d"
Sep 16 14:19:38.226843 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:38.226810    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/910e25fa-e7b6-4b60-a5f0-255d7fc3ca65-tmp\") pod \"opendatahub-operator-controller-manager-68b85944db-9bv9d\" (UID: \"910e25fa-e7b6-4b60-a5f0-255d7fc3ca65\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-9bv9d"
Sep 16 14:19:38.226995 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:38.226849    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfl95\" (UniqueName: \"kubernetes.io/projected/910e25fa-e7b6-4b60-a5f0-255d7fc3ca65-kube-api-access-jfl95\") pod \"opendatahub-operator-controller-manager-68b85944db-9bv9d\" (UID: \"910e25fa-e7b6-4b60-a5f0-255d7fc3ca65\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-9bv9d"
Sep 16 14:19:38.226995 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:38.226894    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/910e25fa-e7b6-4b60-a5f0-255d7fc3ca65-webhook-cert\") pod \"opendatahub-operator-controller-manager-68b85944db-9bv9d\" (UID: \"910e25fa-e7b6-4b60-a5f0-255d7fc3ca65\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-9bv9d"
Sep 16 14:19:38.226995 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:38.226923    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/910e25fa-e7b6-4b60-a5f0-255d7fc3ca65-manifests\") pod \"opendatahub-operator-controller-manager-68b85944db-9bv9d\" (UID: \"910e25fa-e7b6-4b60-a5f0-255d7fc3ca65\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-9bv9d"
Sep 16 14:19:38.226995 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:38.226948    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/910e25fa-e7b6-4b60-a5f0-255d7fc3ca65-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68b85944db-9bv9d\" (UID: \"910e25fa-e7b6-4b60-a5f0-255d7fc3ca65\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-9bv9d"
Sep 16 14:19:38.227334 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:38.227314    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/910e25fa-e7b6-4b60-a5f0-255d7fc3ca65-manifests\") pod \"opendatahub-operator-controller-manager-68b85944db-9bv9d\" (UID: \"910e25fa-e7b6-4b60-a5f0-255d7fc3ca65\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-9bv9d"
Sep 16 14:19:38.229488 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:38.229454    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/910e25fa-e7b6-4b60-a5f0-255d7fc3ca65-tmp\") pod \"opendatahub-operator-controller-manager-68b85944db-9bv9d\" (UID: \"910e25fa-e7b6-4b60-a5f0-255d7fc3ca65\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-9bv9d"
Sep 16 14:19:38.229586 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:38.229562    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/910e25fa-e7b6-4b60-a5f0-255d7fc3ca65-webhook-cert\") pod \"opendatahub-operator-controller-manager-68b85944db-9bv9d\" (UID: \"910e25fa-e7b6-4b60-a5f0-255d7fc3ca65\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-9bv9d"
Sep 16 14:19:38.229646 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:38.229627    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/910e25fa-e7b6-4b60-a5f0-255d7fc3ca65-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68b85944db-9bv9d\" (UID: \"910e25fa-e7b6-4b60-a5f0-255d7fc3ca65\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-9bv9d"
Sep 16 14:19:38.239021 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:38.238996    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfl95\" (UniqueName: \"kubernetes.io/projected/910e25fa-e7b6-4b60-a5f0-255d7fc3ca65-kube-api-access-jfl95\") pod \"opendatahub-operator-controller-manager-68b85944db-9bv9d\" (UID: \"910e25fa-e7b6-4b60-a5f0-255d7fc3ca65\") " pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-9bv9d"
Sep 16 14:19:38.344281 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:38.344193    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-9bv9d"
Sep 16 14:19:38.632565 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:38.632535    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68b85944db-9bv9d"]
Sep 16 14:19:38.634471 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:19:38.634445    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod910e25fa_e7b6_4b60_a5f0_255d7fc3ca65.slice/crio-e235b1cc36175b8c9232e8bf80fd060495d338b62f4d521b0ff8b2cb3d7dcc57 WatchSource:0}: Error finding container e235b1cc36175b8c9232e8bf80fd060495d338b62f4d521b0ff8b2cb3d7dcc57: Status 404 returned error can't find the container with id e235b1cc36175b8c9232e8bf80fd060495d338b62f4d521b0ff8b2cb3d7dcc57
Sep 16 14:19:38.636219 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:38.636203    2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 14:19:39.243976 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:39.243936    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-9bv9d" event={"ID":"910e25fa-e7b6-4b60-a5f0-255d7fc3ca65","Type":"ContainerStarted","Data":"e235b1cc36175b8c9232e8bf80fd060495d338b62f4d521b0ff8b2cb3d7dcc57"}
Sep 16 14:19:42.259302 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:42.259263    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-9bv9d" event={"ID":"910e25fa-e7b6-4b60-a5f0-255d7fc3ca65","Type":"ContainerStarted","Data":"ca6795efd019ad6695a33e452e345e97157623c61079b8d89626b6c8363cbfc2"}
Sep 16 14:19:43.265155 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:43.265118    2587 generic.go:356] "Generic (PLEG): container finished" podID="910e25fa-e7b6-4b60-a5f0-255d7fc3ca65" containerID="ca6795efd019ad6695a33e452e345e97157623c61079b8d89626b6c8363cbfc2" exitCode=0
Sep 16 14:19:43.265563 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:43.265208    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-9bv9d" event={"ID":"910e25fa-e7b6-4b60-a5f0-255d7fc3ca65","Type":"ContainerDied","Data":"ca6795efd019ad6695a33e452e345e97157623c61079b8d89626b6c8363cbfc2"}
Sep 16 14:19:44.276621 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:44.276590    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-9bv9d" event={"ID":"910e25fa-e7b6-4b60-a5f0-255d7fc3ca65","Type":"ContainerStarted","Data":"346809f8db1489030216dfe8d0dde8811ba38a037e60b9d864a0bfb8d678f037"}
Sep 16 14:19:44.277034 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:44.276686    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-9bv9d"
Sep 16 14:19:44.301991 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:44.301898    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-9bv9d" podStartSLOduration=1.664379085 podStartE2EDuration="6.301882759s" totalImagesPullingTime="3.615058335s" totalInitContainerRuntime="1.022445339s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:19:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:19:44.301624227 +0000 UTC m=+332.905249272" watchObservedRunningTime="2026-09-16 14:19:44.301882759 +0000 UTC m=+332.905507791"
Sep 16 14:19:55.283210 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:55.283179    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-68b85944db-9bv9d"
Sep 16 14:19:58.831586 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:58.831557    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-75f575964-h6spt"]
Sep 16 14:19:58.841411 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:58.841376    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-75f575964-h6spt"
Sep 16 14:19:58.850638 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:58.850619    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8cklk\""
Sep 16 14:19:58.850721 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:58.850619    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 16 14:19:58.873059 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:58.873030    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 16 14:19:58.876369 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:58.876344    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-75f575964-h6spt"]
Sep 16 14:19:58.910982 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:58.910933    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/293cc966-ff29-490d-8886-0a734d91f817-tls-certs\") pod \"kube-auth-proxy-75f575964-h6spt\" (UID: \"293cc966-ff29-490d-8886-0a734d91f817\") " pod="openshift-ingress/kube-auth-proxy-75f575964-h6spt"
Sep 16 14:19:58.911183 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:58.911027    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/293cc966-ff29-490d-8886-0a734d91f817-tmp\") pod \"kube-auth-proxy-75f575964-h6spt\" (UID: \"293cc966-ff29-490d-8886-0a734d91f817\") " pod="openshift-ingress/kube-auth-proxy-75f575964-h6spt"
Sep 16 14:19:58.911183 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:58.911067    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27n6g\" (UniqueName: \"kubernetes.io/projected/293cc966-ff29-490d-8886-0a734d91f817-kube-api-access-27n6g\") pod \"kube-auth-proxy-75f575964-h6spt\" (UID: \"293cc966-ff29-490d-8886-0a734d91f817\") " pod="openshift-ingress/kube-auth-proxy-75f575964-h6spt"
Sep 16 14:19:59.012514 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:59.012452    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/293cc966-ff29-490d-8886-0a734d91f817-tls-certs\") pod \"kube-auth-proxy-75f575964-h6spt\" (UID: \"293cc966-ff29-490d-8886-0a734d91f817\") " pod="openshift-ingress/kube-auth-proxy-75f575964-h6spt"
Sep 16 14:19:59.012722 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:59.012536    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/293cc966-ff29-490d-8886-0a734d91f817-tmp\") pod \"kube-auth-proxy-75f575964-h6spt\" (UID: \"293cc966-ff29-490d-8886-0a734d91f817\") " pod="openshift-ingress/kube-auth-proxy-75f575964-h6spt"
Sep 16 14:19:59.012722 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:59.012572    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27n6g\" (UniqueName: \"kubernetes.io/projected/293cc966-ff29-490d-8886-0a734d91f817-kube-api-access-27n6g\") pod \"kube-auth-proxy-75f575964-h6spt\" (UID: \"293cc966-ff29-490d-8886-0a734d91f817\") " pod="openshift-ingress/kube-auth-proxy-75f575964-h6spt"
Sep 16 14:19:59.015094 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:59.015059    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/293cc966-ff29-490d-8886-0a734d91f817-tmp\") pod \"kube-auth-proxy-75f575964-h6spt\" (UID: \"293cc966-ff29-490d-8886-0a734d91f817\") " pod="openshift-ingress/kube-auth-proxy-75f575964-h6spt"
Sep 16 14:19:59.015212 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:59.015193    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/293cc966-ff29-490d-8886-0a734d91f817-tls-certs\") pod \"kube-auth-proxy-75f575964-h6spt\" (UID: \"293cc966-ff29-490d-8886-0a734d91f817\") " pod="openshift-ingress/kube-auth-proxy-75f575964-h6spt"
Sep 16 14:19:59.033746 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:59.033719    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27n6g\" (UniqueName: \"kubernetes.io/projected/293cc966-ff29-490d-8886-0a734d91f817-kube-api-access-27n6g\") pod \"kube-auth-proxy-75f575964-h6spt\" (UID: \"293cc966-ff29-490d-8886-0a734d91f817\") " pod="openshift-ingress/kube-auth-proxy-75f575964-h6spt"
Sep 16 14:19:59.150620 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:59.150590    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-75f575964-h6spt"
Sep 16 14:19:59.435065 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:19:59.434901    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-75f575964-h6spt"]
Sep 16 14:19:59.437709 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:19:59.437681    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod293cc966_ff29_490d_8886_0a734d91f817.slice/crio-6ea1cf4c6b531b4d0cea196dc3121be0724387d6b1b1f385fa5044d9456664bc WatchSource:0}: Error finding container 6ea1cf4c6b531b4d0cea196dc3121be0724387d6b1b1f385fa5044d9456664bc: Status 404 returned error can't find the container with id 6ea1cf4c6b531b4d0cea196dc3121be0724387d6b1b1f385fa5044d9456664bc
Sep 16 14:20:00.349112 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:00.349053    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-75f575964-h6spt" event={"ID":"293cc966-ff29-490d-8886-0a734d91f817","Type":"ContainerStarted","Data":"6ea1cf4c6b531b4d0cea196dc3121be0724387d6b1b1f385fa5044d9456664bc"}
Sep 16 14:20:00.786722 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:00.786683    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"]
Sep 16 14:20:00.790496 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:00.790467    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"
Sep 16 14:20:00.793136 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:00.792969    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 16 14:20:00.793263 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:00.793205    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 16 14:20:00.793263 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:00.793228    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-pdjgm\""
Sep 16 14:20:00.793535 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:00.793510    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 16 14:20:00.793829 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:00.793812    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 14:20:00.808493 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:00.808462    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"]
Sep 16 14:20:00.930225 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:00.930185    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88dkb\" (UniqueName: \"kubernetes.io/projected/5ba46f8d-ef90-4711-b159-b0ec52154814-kube-api-access-88dkb\") pod \"istiod-openshift-gateway-7b7995d5cd-hx9c8\" (UID: \"5ba46f8d-ef90-4711-b159-b0ec52154814\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"
Sep 16 14:20:00.930501 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:00.930249    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba46f8d-ef90-4711-b159-b0ec52154814-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-hx9c8\" (UID: \"5ba46f8d-ef90-4711-b159-b0ec52154814\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"
Sep 16 14:20:00.930501 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:00.930427    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5ba46f8d-ef90-4711-b159-b0ec52154814-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-hx9c8\" (UID: \"5ba46f8d-ef90-4711-b159-b0ec52154814\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"
Sep 16 14:20:00.930501 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:00.930454    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5ba46f8d-ef90-4711-b159-b0ec52154814-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-hx9c8\" (UID: \"5ba46f8d-ef90-4711-b159-b0ec52154814\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"
Sep 16 14:20:00.930672 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:00.930506    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5ba46f8d-ef90-4711-b159-b0ec52154814-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-hx9c8\" (UID: \"5ba46f8d-ef90-4711-b159-b0ec52154814\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"
Sep 16 14:20:00.930672 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:00.930543    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ba46f8d-ef90-4711-b159-b0ec52154814-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-hx9c8\" (UID: \"5ba46f8d-ef90-4711-b159-b0ec52154814\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"
Sep 16 14:20:00.930672 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:00.930573    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba46f8d-ef90-4711-b159-b0ec52154814-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-hx9c8\" (UID: \"5ba46f8d-ef90-4711-b159-b0ec52154814\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"
Sep 16 14:20:01.031796 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:01.031753    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5ba46f8d-ef90-4711-b159-b0ec52154814-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-hx9c8\" (UID: \"5ba46f8d-ef90-4711-b159-b0ec52154814\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"
Sep 16 14:20:01.031984 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:01.031809    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5ba46f8d-ef90-4711-b159-b0ec52154814-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-hx9c8\" (UID: \"5ba46f8d-ef90-4711-b159-b0ec52154814\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"
Sep 16 14:20:01.031984 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:01.031846    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5ba46f8d-ef90-4711-b159-b0ec52154814-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-hx9c8\" (UID: \"5ba46f8d-ef90-4711-b159-b0ec52154814\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"
Sep 16 14:20:01.031984 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:01.031876    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ba46f8d-ef90-4711-b159-b0ec52154814-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-hx9c8\" (UID: \"5ba46f8d-ef90-4711-b159-b0ec52154814\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"
Sep 16 14:20:01.031984 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:01.031900    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba46f8d-ef90-4711-b159-b0ec52154814-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-hx9c8\" (UID: \"5ba46f8d-ef90-4711-b159-b0ec52154814\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"
Sep 16 14:20:01.031984 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:01.031944    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88dkb\" (UniqueName: \"kubernetes.io/projected/5ba46f8d-ef90-4711-b159-b0ec52154814-kube-api-access-88dkb\") pod \"istiod-openshift-gateway-7b7995d5cd-hx9c8\" (UID: \"5ba46f8d-ef90-4711-b159-b0ec52154814\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"
Sep 16 14:20:01.031984 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:01.031971    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba46f8d-ef90-4711-b159-b0ec52154814-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-hx9c8\" (UID: \"5ba46f8d-ef90-4711-b159-b0ec52154814\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"
Sep 16 14:20:01.032887 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:01.032821    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5ba46f8d-ef90-4711-b159-b0ec52154814-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-hx9c8\" (UID: \"5ba46f8d-ef90-4711-b159-b0ec52154814\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"
Sep 16 14:20:01.034718 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:01.034670    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5ba46f8d-ef90-4711-b159-b0ec52154814-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-hx9c8\" (UID: \"5ba46f8d-ef90-4711-b159-b0ec52154814\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"
Sep 16 14:20:01.034905 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:01.034882    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5ba46f8d-ef90-4711-b159-b0ec52154814-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-hx9c8\" (UID: \"5ba46f8d-ef90-4711-b159-b0ec52154814\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"
Sep 16 14:20:01.038633 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:01.038577    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba46f8d-ef90-4711-b159-b0ec52154814-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-hx9c8\" (UID: \"5ba46f8d-ef90-4711-b159-b0ec52154814\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"
Sep 16 14:20:01.038775 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:01.038743    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ba46f8d-ef90-4711-b159-b0ec52154814-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-hx9c8\" (UID: \"5ba46f8d-ef90-4711-b159-b0ec52154814\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"
Sep 16 14:20:01.044038 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:01.044012    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba46f8d-ef90-4711-b159-b0ec52154814-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-hx9c8\" (UID: \"5ba46f8d-ef90-4711-b159-b0ec52154814\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"
Sep 16 14:20:01.045783 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:01.045762    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88dkb\" (UniqueName: \"kubernetes.io/projected/5ba46f8d-ef90-4711-b159-b0ec52154814-kube-api-access-88dkb\") pod \"istiod-openshift-gateway-7b7995d5cd-hx9c8\" (UID: \"5ba46f8d-ef90-4711-b159-b0ec52154814\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"
Sep 16 14:20:01.104143 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:01.104110    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"
Sep 16 14:20:02.543882 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:02.543801    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"]
Sep 16 14:20:02.546619 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:20:02.546585    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba46f8d_ef90_4711_b159_b0ec52154814.slice/crio-3026f40d9b9a0df9b2f525ce219f92eba7f7580ace5868c1cfd1780fa1f139c6 WatchSource:0}: Error finding container 3026f40d9b9a0df9b2f525ce219f92eba7f7580ace5868c1cfd1780fa1f139c6: Status 404 returned error can't find the container with id 3026f40d9b9a0df9b2f525ce219f92eba7f7580ace5868c1cfd1780fa1f139c6
Sep 16 14:20:03.365080 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:03.365043    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8" event={"ID":"5ba46f8d-ef90-4711-b159-b0ec52154814","Type":"ContainerStarted","Data":"3026f40d9b9a0df9b2f525ce219f92eba7f7580ace5868c1cfd1780fa1f139c6"}
Sep 16 14:20:03.366534 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:03.366506    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-75f575964-h6spt" event={"ID":"293cc966-ff29-490d-8886-0a734d91f817","Type":"ContainerStarted","Data":"2814bd2c5ffdc6b29920ed1e98f7bf406aae3a798e4858653a8f46b66674d954"}
Sep 16 14:20:03.386752 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:03.386699    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-75f575964-h6spt" podStartSLOduration=2.484755331 podStartE2EDuration="5.38668287s" totalImagesPullingTime="2.901927539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:20:03.385594382 +0000 UTC m=+351.989219415" watchObservedRunningTime="2026-09-16 14:20:03.38668287 +0000 UTC m=+351.990307906"
Sep 16 14:20:06.407309 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:06.407270    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 14:20:06.407662 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:06.407343    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 16 14:20:07.384579 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:07.384540    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8" event={"ID":"5ba46f8d-ef90-4711-b159-b0ec52154814","Type":"ContainerStarted","Data":"2a468ff3dfb51e0319ba6207671f9a0cc62db9cd67e5effc33837c94eb761837"}
Sep 16 14:20:07.384771 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:07.384721    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"
Sep 16 14:20:07.386545 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:07.386520    2587 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-hx9c8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 14:20:07.386662 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:07.386568    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8" podUID="5ba46f8d-ef90-4711-b159-b0ec52154814" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 14:20:07.413720 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:07.413654    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8" podStartSLOduration=3.555340381 podStartE2EDuration="7.413630786s" totalImagesPullingTime="3.858290405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:20:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:20:07.41070635 +0000 UTC m=+356.014331383" watchObservedRunningTime="2026-09-16 14:20:07.413630786 +0000 UTC m=+356.017255819"
Sep 16 14:20:08.389488 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:08.389458    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"
Sep 16 14:20:18.423762 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:18.423721    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-bn5dg"]
Sep 16 14:20:18.431547 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:18.431521    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-bn5dg"
Sep 16 14:20:18.434851 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:18.434828    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-7mzc6\""
Sep 16 14:20:18.435858 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:18.435830    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 14:20:18.436000 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:18.435878    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 14:20:18.436000 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:18.435878    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 14:20:18.436000 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:18.435929    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 14:20:18.436281 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:18.436261    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 14:20:18.438416 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:18.438380    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-bn5dg"]
Sep 16 14:20:18.481685 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:18.481649    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a322e55-06c1-44e7-993a-331af159e4e1-cert\") pod \"lws-controller-manager-5cbd5cd6f8-bn5dg\" (UID: \"7a322e55-06c1-44e7-993a-331af159e4e1\") " pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-bn5dg"
Sep 16 14:20:18.481685 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:18.481683    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7a322e55-06c1-44e7-993a-331af159e4e1-manager-config\") pod \"lws-controller-manager-5cbd5cd6f8-bn5dg\" (UID: \"7a322e55-06c1-44e7-993a-331af159e4e1\") " pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-bn5dg"
Sep 16 14:20:18.481907 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:18.481744    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7wmk\" (UniqueName: \"kubernetes.io/projected/7a322e55-06c1-44e7-993a-331af159e4e1-kube-api-access-z7wmk\") pod \"lws-controller-manager-5cbd5cd6f8-bn5dg\" (UID: \"7a322e55-06c1-44e7-993a-331af159e4e1\") " pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-bn5dg"
Sep 16 14:20:18.481907 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:18.481786    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7a322e55-06c1-44e7-993a-331af159e4e1-metrics-cert\") pod \"lws-controller-manager-5cbd5cd6f8-bn5dg\" (UID: \"7a322e55-06c1-44e7-993a-331af159e4e1\") " pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-bn5dg"
Sep 16 14:20:18.583071 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:18.583034    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7a322e55-06c1-44e7-993a-331af159e4e1-metrics-cert\") pod \"lws-controller-manager-5cbd5cd6f8-bn5dg\" (UID: \"7a322e55-06c1-44e7-993a-331af159e4e1\") " pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-bn5dg"
Sep 16 14:20:18.583267 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:18.583079    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a322e55-06c1-44e7-993a-331af159e4e1-cert\") pod \"lws-controller-manager-5cbd5cd6f8-bn5dg\" (UID: \"7a322e55-06c1-44e7-993a-331af159e4e1\") " pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-bn5dg"
Sep 16 14:20:18.583267 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:18.583100    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7a322e55-06c1-44e7-993a-331af159e4e1-manager-config\") pod \"lws-controller-manager-5cbd5cd6f8-bn5dg\" (UID: \"7a322e55-06c1-44e7-993a-331af159e4e1\") " pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-bn5dg"
Sep 16 14:20:18.583267 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:18.583138    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7wmk\" (UniqueName: \"kubernetes.io/projected/7a322e55-06c1-44e7-993a-331af159e4e1-kube-api-access-z7wmk\") pod \"lws-controller-manager-5cbd5cd6f8-bn5dg\" (UID: \"7a322e55-06c1-44e7-993a-331af159e4e1\") " pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-bn5dg"
Sep 16 14:20:18.583885 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:18.583862    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7a322e55-06c1-44e7-993a-331af159e4e1-manager-config\") pod \"lws-controller-manager-5cbd5cd6f8-bn5dg\" (UID: \"7a322e55-06c1-44e7-993a-331af159e4e1\") " pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-bn5dg"
Sep 16 14:20:18.585678 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:18.585649    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a322e55-06c1-44e7-993a-331af159e4e1-cert\") pod \"lws-controller-manager-5cbd5cd6f8-bn5dg\" (UID: \"7a322e55-06c1-44e7-993a-331af159e4e1\") " pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-bn5dg"
Sep 16 14:20:18.585802 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:18.585784    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7a322e55-06c1-44e7-993a-331af159e4e1-metrics-cert\") pod \"lws-controller-manager-5cbd5cd6f8-bn5dg\" (UID: \"7a322e55-06c1-44e7-993a-331af159e4e1\") " pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-bn5dg"
Sep 16 14:20:18.594561 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:18.594529    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7wmk\" (UniqueName: \"kubernetes.io/projected/7a322e55-06c1-44e7-993a-331af159e4e1-kube-api-access-z7wmk\") pod \"lws-controller-manager-5cbd5cd6f8-bn5dg\" (UID: \"7a322e55-06c1-44e7-993a-331af159e4e1\") " pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-bn5dg"
Sep 16 14:20:18.742238 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:18.742159    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-bn5dg"
Sep 16 14:20:19.024223 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:19.024197    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-bn5dg"]
Sep 16 14:20:19.025926 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:20:19.025900    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a322e55_06c1_44e7_993a_331af159e4e1.slice/crio-70dc7e29c6e383e51c57bd2f42c470971e86f8c5a2638b9f73a74e79a8c41efd WatchSource:0}: Error finding container 70dc7e29c6e383e51c57bd2f42c470971e86f8c5a2638b9f73a74e79a8c41efd: Status 404 returned error can't find the container with id 70dc7e29c6e383e51c57bd2f42c470971e86f8c5a2638b9f73a74e79a8c41efd
Sep 16 14:20:19.440559 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:19.440520    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-bn5dg" event={"ID":"7a322e55-06c1-44e7-993a-331af159e4e1","Type":"ContainerStarted","Data":"70dc7e29c6e383e51c57bd2f42c470971e86f8c5a2638b9f73a74e79a8c41efd"}
Sep 16 14:20:21.692096 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:21.692057    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"]
Sep 16 14:20:21.692563 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:21.692437    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8" podUID="5ba46f8d-ef90-4711-b159-b0ec52154814" containerName="discovery" containerID="cri-o://2a468ff3dfb51e0319ba6207671f9a0cc62db9cd67e5effc33837c94eb761837" gracePeriod=30
Sep 16 14:20:22.459635 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:22.459600    2587 generic.go:356] "Generic (PLEG): container finished" podID="5ba46f8d-ef90-4711-b159-b0ec52154814" containerID="2a468ff3dfb51e0319ba6207671f9a0cc62db9cd67e5effc33837c94eb761837" exitCode=0
Sep 16 14:20:22.459812 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:22.459665    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8" event={"ID":"5ba46f8d-ef90-4711-b159-b0ec52154814","Type":"ContainerDied","Data":"2a468ff3dfb51e0319ba6207671f9a0cc62db9cd67e5effc33837c94eb761837"}
Sep 16 14:20:22.878498 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:22.878473    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"
Sep 16 14:20:22.923155 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:22.923131    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ba46f8d-ef90-4711-b159-b0ec52154814-istio-kubeconfig\") pod \"5ba46f8d-ef90-4711-b159-b0ec52154814\" (UID: \"5ba46f8d-ef90-4711-b159-b0ec52154814\") "
Sep 16 14:20:22.923473 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:22.923172    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88dkb\" (UniqueName: \"kubernetes.io/projected/5ba46f8d-ef90-4711-b159-b0ec52154814-kube-api-access-88dkb\") pod \"5ba46f8d-ef90-4711-b159-b0ec52154814\" (UID: \"5ba46f8d-ef90-4711-b159-b0ec52154814\") "
Sep 16 14:20:22.923473 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:22.923220    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5ba46f8d-ef90-4711-b159-b0ec52154814-istio-csr-ca-configmap\") pod \"5ba46f8d-ef90-4711-b159-b0ec52154814\" (UID: \"5ba46f8d-ef90-4711-b159-b0ec52154814\") "
Sep 16 14:20:22.923473 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:22.923275    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba46f8d-ef90-4711-b159-b0ec52154814-local-certs\") pod \"5ba46f8d-ef90-4711-b159-b0ec52154814\" (UID: \"5ba46f8d-ef90-4711-b159-b0ec52154814\") "
Sep 16 14:20:22.923473 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:22.923301    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba46f8d-ef90-4711-b159-b0ec52154814-istio-token\") pod \"5ba46f8d-ef90-4711-b159-b0ec52154814\" (UID: \"5ba46f8d-ef90-4711-b159-b0ec52154814\") "
Sep 16 14:20:22.923473 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:22.923325    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5ba46f8d-ef90-4711-b159-b0ec52154814-cacerts\") pod \"5ba46f8d-ef90-4711-b159-b0ec52154814\" (UID: \"5ba46f8d-ef90-4711-b159-b0ec52154814\") "
Sep 16 14:20:22.923473 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:22.923347    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5ba46f8d-ef90-4711-b159-b0ec52154814-istio-csr-dns-cert\") pod \"5ba46f8d-ef90-4711-b159-b0ec52154814\" (UID: \"5ba46f8d-ef90-4711-b159-b0ec52154814\") "
Sep 16 14:20:22.924541 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:22.924497    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba46f8d-ef90-4711-b159-b0ec52154814-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "5ba46f8d-ef90-4711-b159-b0ec52154814" (UID: "5ba46f8d-ef90-4711-b159-b0ec52154814"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:20:22.926239 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:22.926197    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba46f8d-ef90-4711-b159-b0ec52154814-kube-api-access-88dkb" (OuterVolumeSpecName: "kube-api-access-88dkb") pod "5ba46f8d-ef90-4711-b159-b0ec52154814" (UID: "5ba46f8d-ef90-4711-b159-b0ec52154814"). InnerVolumeSpecName "kube-api-access-88dkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:20:22.926642 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:22.926604    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ba46f8d-ef90-4711-b159-b0ec52154814-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "5ba46f8d-ef90-4711-b159-b0ec52154814" (UID: "5ba46f8d-ef90-4711-b159-b0ec52154814"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:20:22.926883 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:22.926827    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ba46f8d-ef90-4711-b159-b0ec52154814-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "5ba46f8d-ef90-4711-b159-b0ec52154814" (UID: "5ba46f8d-ef90-4711-b159-b0ec52154814"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:20:22.926883 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:22.926832    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba46f8d-ef90-4711-b159-b0ec52154814-istio-token" (OuterVolumeSpecName: "istio-token") pod "5ba46f8d-ef90-4711-b159-b0ec52154814" (UID: "5ba46f8d-ef90-4711-b159-b0ec52154814"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:20:22.927908 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:22.927874    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba46f8d-ef90-4711-b159-b0ec52154814-local-certs" (OuterVolumeSpecName: "local-certs") pod "5ba46f8d-ef90-4711-b159-b0ec52154814" (UID: "5ba46f8d-ef90-4711-b159-b0ec52154814"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:20:22.928659 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:22.928634    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ba46f8d-ef90-4711-b159-b0ec52154814-cacerts" (OuterVolumeSpecName: "cacerts") pod "5ba46f8d-ef90-4711-b159-b0ec52154814" (UID: "5ba46f8d-ef90-4711-b159-b0ec52154814"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:20:23.024787 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:23.024749    2587 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ba46f8d-ef90-4711-b159-b0ec52154814-istio-kubeconfig\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:20:23.024787 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:23.024787    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88dkb\" (UniqueName: \"kubernetes.io/projected/5ba46f8d-ef90-4711-b159-b0ec52154814-kube-api-access-88dkb\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:20:23.025023 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:23.024805    2587 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5ba46f8d-ef90-4711-b159-b0ec52154814-istio-csr-ca-configmap\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:20:23.025023 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:23.024819    2587 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba46f8d-ef90-4711-b159-b0ec52154814-local-certs\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:20:23.025023 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:23.024832    2587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba46f8d-ef90-4711-b159-b0ec52154814-istio-token\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:20:23.025023 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:23.024845    2587 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5ba46f8d-ef90-4711-b159-b0ec52154814-cacerts\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:20:23.025023 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:23.024859    2587 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5ba46f8d-ef90-4711-b159-b0ec52154814-istio-csr-dns-cert\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:20:23.464775 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:23.464745    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"
Sep 16 14:20:23.464775 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:23.464763    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8" event={"ID":"5ba46f8d-ef90-4711-b159-b0ec52154814","Type":"ContainerDied","Data":"3026f40d9b9a0df9b2f525ce219f92eba7f7580ace5868c1cfd1780fa1f139c6"}
Sep 16 14:20:23.465006 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:23.464817    2587 scope.go:117] "RemoveContainer" containerID="2a468ff3dfb51e0319ba6207671f9a0cc62db9cd67e5effc33837c94eb761837"
Sep 16 14:20:23.466577 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:23.466486    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-bn5dg" event={"ID":"7a322e55-06c1-44e7-993a-331af159e4e1","Type":"ContainerStarted","Data":"4349ee7780eb2b03259b435be5a582942e1beacf06bbe9f4ddc53a8cdc1e0bc3"}
Sep 16 14:20:23.466577 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:23.466543    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-bn5dg"
Sep 16 14:20:23.510771 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:23.510721    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-bn5dg" podStartSLOduration=1.616180316 podStartE2EDuration="5.510705288s" totalImagesPullingTime="3.894524972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:20:23.504803945 +0000 UTC m=+372.108428979" watchObservedRunningTime="2026-09-16 14:20:23.510705288 +0000 UTC m=+372.114330320"
Sep 16 14:20:23.538404 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:23.538355    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"]
Sep 16 14:20:23.546008 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:23.545979    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hx9c8"]
Sep 16 14:20:23.902626 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:23.902595    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ba46f8d-ef90-4711-b159-b0ec52154814" path="/var/lib/kubelet/pods/5ba46f8d-ef90-4711-b159-b0ec52154814/volumes"
Sep 16 14:20:34.474446 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:34.474385    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5cbd5cd6f8-bn5dg"
Sep 16 14:20:59.853083 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:20:59.853047    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78db8d7859-njbk9"]
Sep 16 14:21:24.873496 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:24.873422    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-78db8d7859-njbk9" podUID="7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc" containerName="console" containerID="cri-o://280834a559a7f9414ad9dfd07b6a9492c6a027a4edb75ed57e66ff06ad291e3d" gracePeriod=15
Sep 16 14:21:25.124373 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.124312    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:21:25.156718 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.156687    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-console-serving-cert\") pod \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\" (UID: \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\") "
Sep 16 14:21:25.156889 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.156724    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-console-config\") pod \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\" (UID: \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\") "
Sep 16 14:21:25.156889 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.156778    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-service-ca\") pod \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\" (UID: \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\") "
Sep 16 14:21:25.156889 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.156816    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-trusted-ca-bundle\") pod \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\" (UID: \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\") "
Sep 16 14:21:25.156889 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.156853    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-console-oauth-config\") pod \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\" (UID: \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\") "
Sep 16 14:21:25.156889 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.156880    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-oauth-serving-cert\") pod \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\" (UID: \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\") "
Sep 16 14:21:25.157180 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.156943    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkxt6\" (UniqueName: \"kubernetes.io/projected/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-kube-api-access-rkxt6\") pod \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\" (UID: \"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc\") "
Sep 16 14:21:25.157311 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.157276    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-service-ca" (OuterVolumeSpecName: "service-ca") pod "7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc" (UID: "7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:21:25.157311 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.157283    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-console-config" (OuterVolumeSpecName: "console-config") pod "7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc" (UID: "7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:21:25.157503 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.157310    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc" (UID: "7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:21:25.157503 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.157323    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc" (UID: "7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:21:25.159159 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.159133    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc" (UID: "7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:21:25.159259 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.159162    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc" (UID: "7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:21:25.159337 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.159322    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-kube-api-access-rkxt6" (OuterVolumeSpecName: "kube-api-access-rkxt6") pod "7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc" (UID: "7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc"). InnerVolumeSpecName "kube-api-access-rkxt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:21:25.257564 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.257520    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkxt6\" (UniqueName: \"kubernetes.io/projected/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-kube-api-access-rkxt6\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:21:25.257564 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.257558    2587 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-console-serving-cert\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:21:25.257564 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.257571    2587 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-console-config\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:21:25.257804 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.257580    2587 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-service-ca\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:21:25.257804 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.257588    2587 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-trusted-ca-bundle\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:21:25.257804 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.257597    2587 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-console-oauth-config\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:21:25.257804 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.257605    2587 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc-oauth-serving-cert\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:21:25.731781 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.731752    2587 generic.go:356] "Generic (PLEG): container finished" podID="7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc" containerID="280834a559a7f9414ad9dfd07b6a9492c6a027a4edb75ed57e66ff06ad291e3d" exitCode=0
Sep 16 14:21:25.731987 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.731791    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78db8d7859-njbk9" event={"ID":"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc","Type":"ContainerDied","Data":"280834a559a7f9414ad9dfd07b6a9492c6a027a4edb75ed57e66ff06ad291e3d"}
Sep 16 14:21:25.731987 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.731803    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78db8d7859-njbk9"
Sep 16 14:21:25.731987 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.731824    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78db8d7859-njbk9" event={"ID":"7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc","Type":"ContainerDied","Data":"0b1fd45aa8a5b4a3b076afd800cc01bc876d072213ae9ed6e0f048c807d311f5"}
Sep 16 14:21:25.731987 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.731842    2587 scope.go:117] "RemoveContainer" containerID="280834a559a7f9414ad9dfd07b6a9492c6a027a4edb75ed57e66ff06ad291e3d"
Sep 16 14:21:25.740936 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.740919    2587 scope.go:117] "RemoveContainer" containerID="280834a559a7f9414ad9dfd07b6a9492c6a027a4edb75ed57e66ff06ad291e3d"
Sep 16 14:21:25.741253 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:21:25.741235    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"280834a559a7f9414ad9dfd07b6a9492c6a027a4edb75ed57e66ff06ad291e3d\": container with ID starting with 280834a559a7f9414ad9dfd07b6a9492c6a027a4edb75ed57e66ff06ad291e3d not found: ID does not exist" containerID="280834a559a7f9414ad9dfd07b6a9492c6a027a4edb75ed57e66ff06ad291e3d"
Sep 16 14:21:25.741295 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.741263    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"280834a559a7f9414ad9dfd07b6a9492c6a027a4edb75ed57e66ff06ad291e3d"} err="failed to get container status \"280834a559a7f9414ad9dfd07b6a9492c6a027a4edb75ed57e66ff06ad291e3d\": rpc error: code = NotFound desc = could not find container \"280834a559a7f9414ad9dfd07b6a9492c6a027a4edb75ed57e66ff06ad291e3d\": container with ID starting with 280834a559a7f9414ad9dfd07b6a9492c6a027a4edb75ed57e66ff06ad291e3d not found: ID does not exist"
Sep 16 14:21:25.771842 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.771809    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78db8d7859-njbk9"]
Sep 16 14:21:25.779866 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.779838    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78db8d7859-njbk9"]
Sep 16 14:21:25.901572 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:25.901539    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc" path="/var/lib/kubelet/pods/7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc/volumes"
Sep 16 14:21:35.180317 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:35.180236    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-m7tvq"]
Sep 16 14:21:35.180688 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:35.180611    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc" containerName="console"
Sep 16 14:21:35.180688 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:35.180627    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc" containerName="console"
Sep 16 14:21:35.180688 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:35.180636    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ba46f8d-ef90-4711-b159-b0ec52154814" containerName="discovery"
Sep 16 14:21:35.180688 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:35.180641    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ba46f8d-ef90-4711-b159-b0ec52154814" containerName="discovery"
Sep 16 14:21:35.180857 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:35.180702    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cfe4ef5-e18d-47e0-ab03-4ca3df3d98dc" containerName="console"
Sep 16 14:21:35.180857 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:35.180714    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ba46f8d-ef90-4711-b159-b0ec52154814" containerName="discovery"
Sep 16 14:21:35.183615 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:35.183597    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m7tvq"
Sep 16 14:21:35.187875 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:21:35.187846    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-console-plugin-fbdd4f445-m7tvq\" is forbidden: User \"system:node:ip-10-0-132-217.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-217.ec2.internal' and this object" podUID="40ef16cf-9f50-4e74-9e55-c8a6399d47fb" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m7tvq"
Sep 16 14:21:35.187991 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:21:35.187941    2587 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-132-217.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-217.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 16 14:21:35.188082 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:21:35.188063    2587 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-2f64c\" is forbidden: User \"system:node:ip-10-0-132-217.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-217.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2f64c\"" type="*v1.Secret"
Sep 16 14:21:35.188126 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:21:35.188095    2587 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"plugin-serving-cert\" is forbidden: User \"system:node:ip-10-0-132-217.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-217.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" type="*v1.Secret"
Sep 16 14:21:35.188269 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:21:35.188242    2587 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kuadrant-console-nginx-conf\" is forbidden: User \"system:node:ip-10-0-132-217.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-217.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" type="*v1.ConfigMap"
Sep 16 14:21:35.188316 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:21:35.188272    2587 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-132-217.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-217.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 16 14:21:35.201381 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:35.201353    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-m7tvq"]
Sep 16 14:21:35.235621 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:35.235594    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/40ef16cf-9f50-4e74-9e55-c8a6399d47fb-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-m7tvq\" (UID: \"40ef16cf-9f50-4e74-9e55-c8a6399d47fb\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m7tvq"
Sep 16 14:21:35.235785 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:35.235637    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ctcj\" (UniqueName: \"kubernetes.io/projected/40ef16cf-9f50-4e74-9e55-c8a6399d47fb-kube-api-access-5ctcj\") pod \"kuadrant-console-plugin-fbdd4f445-m7tvq\" (UID: \"40ef16cf-9f50-4e74-9e55-c8a6399d47fb\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m7tvq"
Sep 16 14:21:35.235785 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:35.235739    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/40ef16cf-9f50-4e74-9e55-c8a6399d47fb-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-m7tvq\" (UID: \"40ef16cf-9f50-4e74-9e55-c8a6399d47fb\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m7tvq"
Sep 16 14:21:35.337047 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:35.337011    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/40ef16cf-9f50-4e74-9e55-c8a6399d47fb-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-m7tvq\" (UID: \"40ef16cf-9f50-4e74-9e55-c8a6399d47fb\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m7tvq"
Sep 16 14:21:35.337047 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:35.337058    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/40ef16cf-9f50-4e74-9e55-c8a6399d47fb-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-m7tvq\" (UID: \"40ef16cf-9f50-4e74-9e55-c8a6399d47fb\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m7tvq"
Sep 16 14:21:35.337247 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:35.337088    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ctcj\" (UniqueName: \"kubernetes.io/projected/40ef16cf-9f50-4e74-9e55-c8a6399d47fb-kube-api-access-5ctcj\") pod \"kuadrant-console-plugin-fbdd4f445-m7tvq\" (UID: \"40ef16cf-9f50-4e74-9e55-c8a6399d47fb\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m7tvq"
Sep 16 14:21:36.035915 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:36.035881    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2f64c\""
Sep 16 14:21:36.108458 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:36.108421    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 14:21:36.187903 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:36.187869    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 14:21:36.199045 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:36.199013    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ctcj\" (UniqueName: \"kubernetes.io/projected/40ef16cf-9f50-4e74-9e55-c8a6399d47fb-kube-api-access-5ctcj\") pod \"kuadrant-console-plugin-fbdd4f445-m7tvq\" (UID: \"40ef16cf-9f50-4e74-9e55-c8a6399d47fb\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m7tvq"
Sep 16 14:21:36.337803 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:21:36.337730    2587 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: failed to sync secret cache: timed out waiting for the condition
Sep 16 14:21:36.337803 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:21:36.337743    2587 configmap.go:193] Couldn't get configMap kuadrant-system/kuadrant-console-nginx-conf: failed to sync configmap cache: timed out waiting for the condition
Sep 16 14:21:36.337964 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:21:36.337810    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40ef16cf-9f50-4e74-9e55-c8a6399d47fb-plugin-serving-cert podName:40ef16cf-9f50-4e74-9e55-c8a6399d47fb nodeName:}" failed. No retries permitted until 2026-09-16 14:21:36.837794215 +0000 UTC m=+445.441419225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/40ef16cf-9f50-4e74-9e55-c8a6399d47fb-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-m7tvq" (UID: "40ef16cf-9f50-4e74-9e55-c8a6399d47fb") : failed to sync secret cache: timed out waiting for the condition
Sep 16 14:21:36.337964 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:21:36.337824    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/40ef16cf-9f50-4e74-9e55-c8a6399d47fb-nginx-conf podName:40ef16cf-9f50-4e74-9e55-c8a6399d47fb nodeName:}" failed. No retries permitted until 2026-09-16 14:21:36.837818108 +0000 UTC m=+445.441443118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "nginx-conf" (UniqueName: "kubernetes.io/configmap/40ef16cf-9f50-4e74-9e55-c8a6399d47fb-nginx-conf") pod "kuadrant-console-plugin-fbdd4f445-m7tvq" (UID: "40ef16cf-9f50-4e74-9e55-c8a6399d47fb") : failed to sync configmap cache: timed out waiting for the condition
Sep 16 14:21:36.496197 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:36.496169    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 16 14:21:36.650436 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:36.650416    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 16 14:21:36.849707 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:36.849674    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/40ef16cf-9f50-4e74-9e55-c8a6399d47fb-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-m7tvq\" (UID: \"40ef16cf-9f50-4e74-9e55-c8a6399d47fb\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m7tvq"
Sep 16 14:21:36.849871 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:36.849718    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/40ef16cf-9f50-4e74-9e55-c8a6399d47fb-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-m7tvq\" (UID: \"40ef16cf-9f50-4e74-9e55-c8a6399d47fb\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m7tvq"
Sep 16 14:21:36.850427 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:36.850380    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/40ef16cf-9f50-4e74-9e55-c8a6399d47fb-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-m7tvq\" (UID: \"40ef16cf-9f50-4e74-9e55-c8a6399d47fb\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m7tvq"
Sep 16 14:21:36.852373 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:36.852352    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/40ef16cf-9f50-4e74-9e55-c8a6399d47fb-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-m7tvq\" (UID: \"40ef16cf-9f50-4e74-9e55-c8a6399d47fb\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m7tvq"
Sep 16 14:21:36.992615 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:36.992533    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m7tvq"
Sep 16 14:21:37.266011 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:37.265984    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-m7tvq"]
Sep 16 14:21:37.268473 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:21:37.268449    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40ef16cf_9f50_4e74_9e55_c8a6399d47fb.slice/crio-f03e81e35d57047514a1e5af9955425a61f1eabe4aa9da620bd859edf8a73434 WatchSource:0}: Error finding container f03e81e35d57047514a1e5af9955425a61f1eabe4aa9da620bd859edf8a73434: Status 404 returned error can't find the container with id f03e81e35d57047514a1e5af9955425a61f1eabe4aa9da620bd859edf8a73434
Sep 16 14:21:37.787918 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:37.787881    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m7tvq" event={"ID":"40ef16cf-9f50-4e74-9e55-c8a6399d47fb","Type":"ContainerStarted","Data":"f03e81e35d57047514a1e5af9955425a61f1eabe4aa9da620bd859edf8a73434"}
Sep 16 14:21:41.809032 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:21:41.808955    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m7tvq" event={"ID":"40ef16cf-9f50-4e74-9e55-c8a6399d47fb","Type":"ContainerStarted","Data":"e354a7c66602c9d71f8aed3ef3206a61118be4eabe736a428f77df7a67169e1e"}
Sep 16 14:22:21.232012 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:21.231954    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m7tvq" podStartSLOduration=42.073764992 podStartE2EDuration="46.231937145s" totalImagesPullingTime="4.158172153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:21:41.85354357 +0000 UTC m=+450.457168601" watchObservedRunningTime="2026-09-16 14:22:21.231937145 +0000 UTC m=+489.835562176"
Sep 16 14:22:21.233081 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:21.233062    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-j9gm4"]
Sep 16 14:22:21.255512 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:21.255486    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-j9gm4"]
Sep 16 14:22:21.255653 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:21.255577    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-j9gm4"
Sep 16 14:22:21.258252 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:21.258227    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wfb5q\""
Sep 16 14:22:21.322657 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:21.322621    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs2qt\" (UniqueName: \"kubernetes.io/projected/2c459ebd-023a-45a9-80e0-c1f67078dc62-kube-api-access-xs2qt\") pod \"authorino-7bcfd8f9cd-j9gm4\" (UID: \"2c459ebd-023a-45a9-80e0-c1f67078dc62\") " pod="kuadrant-system/authorino-7bcfd8f9cd-j9gm4"
Sep 16 14:22:21.402704 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:21.402672    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-2fzwr"]
Sep 16 14:22:21.406044 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:21.406028    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2fzwr"
Sep 16 14:22:21.414900 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:21.414874    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2fzwr"]
Sep 16 14:22:21.423651 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:21.423619    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs2qt\" (UniqueName: \"kubernetes.io/projected/2c459ebd-023a-45a9-80e0-c1f67078dc62-kube-api-access-xs2qt\") pod \"authorino-7bcfd8f9cd-j9gm4\" (UID: \"2c459ebd-023a-45a9-80e0-c1f67078dc62\") " pod="kuadrant-system/authorino-7bcfd8f9cd-j9gm4"
Sep 16 14:22:21.423829 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:21.423710    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pz76\" (UniqueName: \"kubernetes.io/projected/ec5a4b62-1ee5-45d4-a301-047cec60244b-kube-api-access-8pz76\") pod \"authorino-59c974bd48-2fzwr\" (UID: \"ec5a4b62-1ee5-45d4-a301-047cec60244b\") " pod="kuadrant-system/authorino-59c974bd48-2fzwr"
Sep 16 14:22:21.432132 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:21.432111    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs2qt\" (UniqueName: \"kubernetes.io/projected/2c459ebd-023a-45a9-80e0-c1f67078dc62-kube-api-access-xs2qt\") pod \"authorino-7bcfd8f9cd-j9gm4\" (UID: \"2c459ebd-023a-45a9-80e0-c1f67078dc62\") " pod="kuadrant-system/authorino-7bcfd8f9cd-j9gm4"
Sep 16 14:22:21.525050 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:21.524955    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pz76\" (UniqueName: \"kubernetes.io/projected/ec5a4b62-1ee5-45d4-a301-047cec60244b-kube-api-access-8pz76\") pod \"authorino-59c974bd48-2fzwr\" (UID: \"ec5a4b62-1ee5-45d4-a301-047cec60244b\") " pod="kuadrant-system/authorino-59c974bd48-2fzwr"
Sep 16 14:22:21.535145 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:21.535116    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pz76\" (UniqueName: \"kubernetes.io/projected/ec5a4b62-1ee5-45d4-a301-047cec60244b-kube-api-access-8pz76\") pod \"authorino-59c974bd48-2fzwr\" (UID: \"ec5a4b62-1ee5-45d4-a301-047cec60244b\") " pod="kuadrant-system/authorino-59c974bd48-2fzwr"
Sep 16 14:22:21.564811 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:21.564772    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-j9gm4"
Sep 16 14:22:21.714967 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:21.714926    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2fzwr"
Sep 16 14:22:21.836550 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:21.836523    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-j9gm4"]
Sep 16 14:22:21.839094 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:22:21.839062    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c459ebd_023a_45a9_80e0_c1f67078dc62.slice/crio-e5f930d2e9d0c9be3a3451d634b501f1dc8802af62c29edf00af7def6705d06a WatchSource:0}: Error finding container e5f930d2e9d0c9be3a3451d634b501f1dc8802af62c29edf00af7def6705d06a: Status 404 returned error can't find the container with id e5f930d2e9d0c9be3a3451d634b501f1dc8802af62c29edf00af7def6705d06a
Sep 16 14:22:21.981717 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:21.981687    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-j9gm4" event={"ID":"2c459ebd-023a-45a9-80e0-c1f67078dc62","Type":"ContainerStarted","Data":"e5f930d2e9d0c9be3a3451d634b501f1dc8802af62c29edf00af7def6705d06a"}
Sep 16 14:22:21.989137 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:22:21.989073    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec5a4b62_1ee5_45d4_a301_047cec60244b.slice/crio-751aabceb767cf3fa9466a41c1de4cb506b41312fed8dec68854e26ad5dbc77b WatchSource:0}: Error finding container 751aabceb767cf3fa9466a41c1de4cb506b41312fed8dec68854e26ad5dbc77b: Status 404 returned error can't find the container with id 751aabceb767cf3fa9466a41c1de4cb506b41312fed8dec68854e26ad5dbc77b
Sep 16 14:22:21.991503 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:21.991463    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2fzwr"]
Sep 16 14:22:22.989903 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:22.989857    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2fzwr" event={"ID":"ec5a4b62-1ee5-45d4-a301-047cec60244b","Type":"ContainerStarted","Data":"751aabceb767cf3fa9466a41c1de4cb506b41312fed8dec68854e26ad5dbc77b"}
Sep 16 14:22:26.004265 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:26.004225    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2fzwr" event={"ID":"ec5a4b62-1ee5-45d4-a301-047cec60244b","Type":"ContainerStarted","Data":"4dc6989f3f58a0ec01986c8846c5ab929b237267844364b5f30b75220406e6e5"}
Sep 16 14:22:26.022245 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:26.022195    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-2fzwr" podStartSLOduration=1.565046105 podStartE2EDuration="5.022179629s" totalImagesPullingTime="3.457133524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:22:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:22:26.020486469 +0000 UTC m=+494.624111503" watchObservedRunningTime="2026-09-16 14:22:26.022179629 +0000 UTC m=+494.625804661"
Sep 16 14:22:26.049539 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:26.049505    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-j9gm4"]
Sep 16 14:22:35.043363 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:35.043279    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-j9gm4" event={"ID":"2c459ebd-023a-45a9-80e0-c1f67078dc62","Type":"ContainerStarted","Data":"44cb3a488c53ca0a61d2265e024b9f253f910121a95040d0b83ccbd86e31d3a3"}
Sep 16 14:22:35.043779 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:35.043362    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-j9gm4" podUID="2c459ebd-023a-45a9-80e0-c1f67078dc62" containerName="authorino" containerID="cri-o://44cb3a488c53ca0a61d2265e024b9f253f910121a95040d0b83ccbd86e31d3a3" gracePeriod=30
Sep 16 14:22:35.067597 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:35.067548    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-j9gm4" podStartSLOduration=1.246633851 podStartE2EDuration="14.067530787s" totalImagesPullingTime="12.820896936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:22:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:22:35.066352286 +0000 UTC m=+503.669977320" watchObservedRunningTime="2026-09-16 14:22:35.067530787 +0000 UTC m=+503.671155819"
Sep 16 14:22:35.279849 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:35.279824    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-j9gm4"
Sep 16 14:22:35.336358 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:35.336277    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xs2qt\" (UniqueName: \"kubernetes.io/projected/2c459ebd-023a-45a9-80e0-c1f67078dc62-kube-api-access-xs2qt\") pod \"2c459ebd-023a-45a9-80e0-c1f67078dc62\" (UID: \"2c459ebd-023a-45a9-80e0-c1f67078dc62\") "
Sep 16 14:22:35.338703 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:35.338678    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c459ebd-023a-45a9-80e0-c1f67078dc62-kube-api-access-xs2qt" (OuterVolumeSpecName: "kube-api-access-xs2qt") pod "2c459ebd-023a-45a9-80e0-c1f67078dc62" (UID: "2c459ebd-023a-45a9-80e0-c1f67078dc62"). InnerVolumeSpecName "kube-api-access-xs2qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:22:35.437711 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:35.437680    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs2qt\" (UniqueName: \"kubernetes.io/projected/2c459ebd-023a-45a9-80e0-c1f67078dc62-kube-api-access-xs2qt\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:22:36.049086 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:36.049049    2587 generic.go:356] "Generic (PLEG): container finished" podID="2c459ebd-023a-45a9-80e0-c1f67078dc62" containerID="44cb3a488c53ca0a61d2265e024b9f253f910121a95040d0b83ccbd86e31d3a3" exitCode=0
Sep 16 14:22:36.049086 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:36.049091    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-j9gm4"
Sep 16 14:22:36.049607 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:36.049137    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-j9gm4" event={"ID":"2c459ebd-023a-45a9-80e0-c1f67078dc62","Type":"ContainerDied","Data":"44cb3a488c53ca0a61d2265e024b9f253f910121a95040d0b83ccbd86e31d3a3"}
Sep 16 14:22:36.049607 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:36.049176    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-j9gm4" event={"ID":"2c459ebd-023a-45a9-80e0-c1f67078dc62","Type":"ContainerDied","Data":"e5f930d2e9d0c9be3a3451d634b501f1dc8802af62c29edf00af7def6705d06a"}
Sep 16 14:22:36.049607 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:36.049192    2587 scope.go:117] "RemoveContainer" containerID="44cb3a488c53ca0a61d2265e024b9f253f910121a95040d0b83ccbd86e31d3a3"
Sep 16 14:22:36.057377 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:36.057358    2587 scope.go:117] "RemoveContainer" containerID="44cb3a488c53ca0a61d2265e024b9f253f910121a95040d0b83ccbd86e31d3a3"
Sep 16 14:22:36.057658 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:22:36.057641    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44cb3a488c53ca0a61d2265e024b9f253f910121a95040d0b83ccbd86e31d3a3\": container with ID starting with 44cb3a488c53ca0a61d2265e024b9f253f910121a95040d0b83ccbd86e31d3a3 not found: ID does not exist" containerID="44cb3a488c53ca0a61d2265e024b9f253f910121a95040d0b83ccbd86e31d3a3"
Sep 16 14:22:36.057703 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:36.057668    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44cb3a488c53ca0a61d2265e024b9f253f910121a95040d0b83ccbd86e31d3a3"} err="failed to get container status \"44cb3a488c53ca0a61d2265e024b9f253f910121a95040d0b83ccbd86e31d3a3\": rpc error: code = NotFound desc = could not find container \"44cb3a488c53ca0a61d2265e024b9f253f910121a95040d0b83ccbd86e31d3a3\": container with ID starting with 44cb3a488c53ca0a61d2265e024b9f253f910121a95040d0b83ccbd86e31d3a3 not found: ID does not exist"
Sep 16 14:22:36.089579 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:36.086706    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-j9gm4"]
Sep 16 14:22:36.093203 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:36.093176    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-j9gm4"]
Sep 16 14:22:37.901883 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:37.901848    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c459ebd-023a-45a9-80e0-c1f67078dc62" path="/var/lib/kubelet/pods/2c459ebd-023a-45a9-80e0-c1f67078dc62/volumes"
Sep 16 14:22:54.861633 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:54.861597    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-mh5lh"]
Sep 16 14:22:54.862026 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:54.861939    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c459ebd-023a-45a9-80e0-c1f67078dc62" containerName="authorino"
Sep 16 14:22:54.862026 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:54.861951    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c459ebd-023a-45a9-80e0-c1f67078dc62" containerName="authorino"
Sep 16 14:22:54.862026 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:54.862016    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c459ebd-023a-45a9-80e0-c1f67078dc62" containerName="authorino"
Sep 16 14:22:54.866519 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:54.866503    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-mh5lh"
Sep 16 14:22:54.877429 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:54.877407    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-mh5lh"]
Sep 16 14:22:54.991119 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:54.991091    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwqqd\" (UniqueName: \"kubernetes.io/projected/33f17e55-113e-4bf4-8114-589ad0cbeedd-kube-api-access-pwqqd\") pod \"authorino-557f876f69-mh5lh\" (UID: \"33f17e55-113e-4bf4-8114-589ad0cbeedd\") " pod="kuadrant-system/authorino-557f876f69-mh5lh"
Sep 16 14:22:55.028842 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.028812    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-mh5lh"]
Sep 16 14:22:55.029027 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:22:55.029010    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-pwqqd], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-mh5lh" podUID="33f17e55-113e-4bf4-8114-589ad0cbeedd"
Sep 16 14:22:55.060205 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.060179    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-qjk2g"]
Sep 16 14:22:55.063316 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.063301    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-qjk2g"
Sep 16 14:22:55.066686 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.066668    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 14:22:55.092316 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.092290    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwqqd\" (UniqueName: \"kubernetes.io/projected/33f17e55-113e-4bf4-8114-589ad0cbeedd-kube-api-access-pwqqd\") pod \"authorino-557f876f69-mh5lh\" (UID: \"33f17e55-113e-4bf4-8114-589ad0cbeedd\") " pod="kuadrant-system/authorino-557f876f69-mh5lh"
Sep 16 14:22:55.092723 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.092706    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-qjk2g"]
Sep 16 14:22:55.124344 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.124296    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwqqd\" (UniqueName: \"kubernetes.io/projected/33f17e55-113e-4bf4-8114-589ad0cbeedd-kube-api-access-pwqqd\") pod \"authorino-557f876f69-mh5lh\" (UID: \"33f17e55-113e-4bf4-8114-589ad0cbeedd\") " pod="kuadrant-system/authorino-557f876f69-mh5lh"
Sep 16 14:22:55.130681 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.130651    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-mh5lh"
Sep 16 14:22:55.138609 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.138590    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-mh5lh"
Sep 16 14:22:55.192858 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.192835    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jxgm\" (UniqueName: \"kubernetes.io/projected/0cfd2f86-5a0c-4dc0-b055-20022c875915-kube-api-access-6jxgm\") pod \"authorino-6d4dfc649f-qjk2g\" (UID: \"0cfd2f86-5a0c-4dc0-b055-20022c875915\") " pod="kuadrant-system/authorino-6d4dfc649f-qjk2g"
Sep 16 14:22:55.192970 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.192903    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0cfd2f86-5a0c-4dc0-b055-20022c875915-tls-cert\") pod \"authorino-6d4dfc649f-qjk2g\" (UID: \"0cfd2f86-5a0c-4dc0-b055-20022c875915\") " pod="kuadrant-system/authorino-6d4dfc649f-qjk2g"
Sep 16 14:22:55.199635 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.199605    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-qjk2g"]
Sep 16 14:22:55.200176 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:22:55.200145    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-6jxgm tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-qjk2g" podUID="0cfd2f86-5a0c-4dc0-b055-20022c875915"
Sep 16 14:22:55.240070 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.240044    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5954586d97-n5t7c"]
Sep 16 14:22:55.243042 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.243028    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5954586d97-n5t7c"
Sep 16 14:22:55.255718 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.255700    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5954586d97-n5t7c"]
Sep 16 14:22:55.294001 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.293972    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwqqd\" (UniqueName: \"kubernetes.io/projected/33f17e55-113e-4bf4-8114-589ad0cbeedd-kube-api-access-pwqqd\") pod \"33f17e55-113e-4bf4-8114-589ad0cbeedd\" (UID: \"33f17e55-113e-4bf4-8114-589ad0cbeedd\") "
Sep 16 14:22:55.294171 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.294155    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0cfd2f86-5a0c-4dc0-b055-20022c875915-tls-cert\") pod \"authorino-6d4dfc649f-qjk2g\" (UID: \"0cfd2f86-5a0c-4dc0-b055-20022c875915\") " pod="kuadrant-system/authorino-6d4dfc649f-qjk2g"
Sep 16 14:22:55.294242 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.294226    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jxgm\" (UniqueName: \"kubernetes.io/projected/0cfd2f86-5a0c-4dc0-b055-20022c875915-kube-api-access-6jxgm\") pod \"authorino-6d4dfc649f-qjk2g\" (UID: \"0cfd2f86-5a0c-4dc0-b055-20022c875915\") " pod="kuadrant-system/authorino-6d4dfc649f-qjk2g"
Sep 16 14:22:55.296254 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.296233    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f17e55-113e-4bf4-8114-589ad0cbeedd-kube-api-access-pwqqd" (OuterVolumeSpecName: "kube-api-access-pwqqd") pod "33f17e55-113e-4bf4-8114-589ad0cbeedd" (UID: "33f17e55-113e-4bf4-8114-589ad0cbeedd"). InnerVolumeSpecName "kube-api-access-pwqqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:22:55.296705 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.296687    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0cfd2f86-5a0c-4dc0-b055-20022c875915-tls-cert\") pod \"authorino-6d4dfc649f-qjk2g\" (UID: \"0cfd2f86-5a0c-4dc0-b055-20022c875915\") " pod="kuadrant-system/authorino-6d4dfc649f-qjk2g"
Sep 16 14:22:55.303780 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.303762    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jxgm\" (UniqueName: \"kubernetes.io/projected/0cfd2f86-5a0c-4dc0-b055-20022c875915-kube-api-access-6jxgm\") pod \"authorino-6d4dfc649f-qjk2g\" (UID: \"0cfd2f86-5a0c-4dc0-b055-20022c875915\") " pod="kuadrant-system/authorino-6d4dfc649f-qjk2g"
Sep 16 14:22:55.394664 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.394604    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d1f8393-db8f-4305-b8be-59cf0ffc3578-tls-cert\") pod \"authorino-5954586d97-n5t7c\" (UID: \"7d1f8393-db8f-4305-b8be-59cf0ffc3578\") " pod="kuadrant-system/authorino-5954586d97-n5t7c"
Sep 16 14:22:55.394747 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.394662    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjf89\" (UniqueName: \"kubernetes.io/projected/7d1f8393-db8f-4305-b8be-59cf0ffc3578-kube-api-access-tjf89\") pod \"authorino-5954586d97-n5t7c\" (UID: \"7d1f8393-db8f-4305-b8be-59cf0ffc3578\") " pod="kuadrant-system/authorino-5954586d97-n5t7c"
Sep 16 14:22:55.394747 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.394717    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwqqd\" (UniqueName: \"kubernetes.io/projected/33f17e55-113e-4bf4-8114-589ad0cbeedd-kube-api-access-pwqqd\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:22:55.496116 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.496071    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d1f8393-db8f-4305-b8be-59cf0ffc3578-tls-cert\") pod \"authorino-5954586d97-n5t7c\" (UID: \"7d1f8393-db8f-4305-b8be-59cf0ffc3578\") " pod="kuadrant-system/authorino-5954586d97-n5t7c"
Sep 16 14:22:55.496288 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.496166    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjf89\" (UniqueName: \"kubernetes.io/projected/7d1f8393-db8f-4305-b8be-59cf0ffc3578-kube-api-access-tjf89\") pod \"authorino-5954586d97-n5t7c\" (UID: \"7d1f8393-db8f-4305-b8be-59cf0ffc3578\") " pod="kuadrant-system/authorino-5954586d97-n5t7c"
Sep 16 14:22:55.498638 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.498617    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d1f8393-db8f-4305-b8be-59cf0ffc3578-tls-cert\") pod \"authorino-5954586d97-n5t7c\" (UID: \"7d1f8393-db8f-4305-b8be-59cf0ffc3578\") " pod="kuadrant-system/authorino-5954586d97-n5t7c"
Sep 16 14:22:55.508220 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.508200    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjf89\" (UniqueName: \"kubernetes.io/projected/7d1f8393-db8f-4305-b8be-59cf0ffc3578-kube-api-access-tjf89\") pod \"authorino-5954586d97-n5t7c\" (UID: \"7d1f8393-db8f-4305-b8be-59cf0ffc3578\") " pod="kuadrant-system/authorino-5954586d97-n5t7c"
Sep 16 14:22:55.551336 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.551307    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5954586d97-n5t7c"
Sep 16 14:22:55.832668 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:55.832644    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5954586d97-n5t7c"]
Sep 16 14:22:55.835036 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:22:55.835012    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d1f8393_db8f_4305_b8be_59cf0ffc3578.slice/crio-88c109226cba014ddd39856af59398169312a175e102ecf788ee53de7b0f60da WatchSource:0}: Error finding container 88c109226cba014ddd39856af59398169312a175e102ecf788ee53de7b0f60da: Status 404 returned error can't find the container with id 88c109226cba014ddd39856af59398169312a175e102ecf788ee53de7b0f60da
Sep 16 14:22:56.136434 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:56.136343    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-qjk2g"
Sep 16 14:22:56.136434 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:56.136343    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5954586d97-n5t7c" event={"ID":"7d1f8393-db8f-4305-b8be-59cf0ffc3578","Type":"ContainerStarted","Data":"88c109226cba014ddd39856af59398169312a175e102ecf788ee53de7b0f60da"}
Sep 16 14:22:56.136890 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:56.136430    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-mh5lh"
Sep 16 14:22:56.145189 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:56.145169    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-qjk2g"
Sep 16 14:22:56.195323 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:56.195299    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-mh5lh"]
Sep 16 14:22:56.200507 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:56.200488    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-mh5lh"]
Sep 16 14:22:56.302717 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:56.302690    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0cfd2f86-5a0c-4dc0-b055-20022c875915-tls-cert\") pod \"0cfd2f86-5a0c-4dc0-b055-20022c875915\" (UID: \"0cfd2f86-5a0c-4dc0-b055-20022c875915\") "
Sep 16 14:22:56.302851 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:56.302744    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jxgm\" (UniqueName: \"kubernetes.io/projected/0cfd2f86-5a0c-4dc0-b055-20022c875915-kube-api-access-6jxgm\") pod \"0cfd2f86-5a0c-4dc0-b055-20022c875915\" (UID: \"0cfd2f86-5a0c-4dc0-b055-20022c875915\") "
Sep 16 14:22:56.304946 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:56.304924    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cfd2f86-5a0c-4dc0-b055-20022c875915-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "0cfd2f86-5a0c-4dc0-b055-20022c875915" (UID: "0cfd2f86-5a0c-4dc0-b055-20022c875915"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:22:56.305063 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:56.304999    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cfd2f86-5a0c-4dc0-b055-20022c875915-kube-api-access-6jxgm" (OuterVolumeSpecName: "kube-api-access-6jxgm") pod "0cfd2f86-5a0c-4dc0-b055-20022c875915" (UID: "0cfd2f86-5a0c-4dc0-b055-20022c875915"). InnerVolumeSpecName "kube-api-access-6jxgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:22:56.403806 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:56.403786    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jxgm\" (UniqueName: \"kubernetes.io/projected/0cfd2f86-5a0c-4dc0-b055-20022c875915-kube-api-access-6jxgm\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:22:56.403806 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:56.403806    2587 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0cfd2f86-5a0c-4dc0-b055-20022c875915-tls-cert\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:22:57.143166 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:57.143129    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5954586d97-n5t7c" event={"ID":"7d1f8393-db8f-4305-b8be-59cf0ffc3578","Type":"ContainerStarted","Data":"95c4d94b63e0c90f0043d7774d04ee0b5c8cc0e9d10163075006a557b772d2cb"}
Sep 16 14:22:57.143545 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:57.143187    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-qjk2g"
Sep 16 14:22:57.164422 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:57.164358    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5954586d97-n5t7c" podStartSLOduration=1.502608345 podStartE2EDuration="2.164342778s" totalImagesPullingTime="661.734433ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:22:57.16129383 +0000 UTC m=+525.764918862" watchObservedRunningTime="2026-09-16 14:22:57.164342778 +0000 UTC m=+525.767967810"
Sep 16 14:22:57.202803 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:57.202762    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-qjk2g"]
Sep 16 14:22:57.210630 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:57.210600    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-qjk2g"]
Sep 16 14:22:57.217989 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:57.217965    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2fzwr"]
Sep 16 14:22:57.218143 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:57.218123    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-2fzwr" podUID="ec5a4b62-1ee5-45d4-a301-047cec60244b" containerName="authorino" containerID="cri-o://4dc6989f3f58a0ec01986c8846c5ab929b237267844364b5f30b75220406e6e5" gracePeriod=30
Sep 16 14:22:57.450672 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:57.450652    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2fzwr"
Sep 16 14:22:57.613873 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:57.613844    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pz76\" (UniqueName: \"kubernetes.io/projected/ec5a4b62-1ee5-45d4-a301-047cec60244b-kube-api-access-8pz76\") pod \"ec5a4b62-1ee5-45d4-a301-047cec60244b\" (UID: \"ec5a4b62-1ee5-45d4-a301-047cec60244b\") "
Sep 16 14:22:57.616031 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:57.616010    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec5a4b62-1ee5-45d4-a301-047cec60244b-kube-api-access-8pz76" (OuterVolumeSpecName: "kube-api-access-8pz76") pod "ec5a4b62-1ee5-45d4-a301-047cec60244b" (UID: "ec5a4b62-1ee5-45d4-a301-047cec60244b"). InnerVolumeSpecName "kube-api-access-8pz76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:22:57.714785 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:57.714715    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pz76\" (UniqueName: \"kubernetes.io/projected/ec5a4b62-1ee5-45d4-a301-047cec60244b-kube-api-access-8pz76\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:22:57.901738 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:57.901708    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cfd2f86-5a0c-4dc0-b055-20022c875915" path="/var/lib/kubelet/pods/0cfd2f86-5a0c-4dc0-b055-20022c875915/volumes"
Sep 16 14:22:57.901923 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:57.901911    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33f17e55-113e-4bf4-8114-589ad0cbeedd" path="/var/lib/kubelet/pods/33f17e55-113e-4bf4-8114-589ad0cbeedd/volumes"
Sep 16 14:22:58.148647 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:58.148620    2587 generic.go:356] "Generic (PLEG): container finished" podID="ec5a4b62-1ee5-45d4-a301-047cec60244b" containerID="4dc6989f3f58a0ec01986c8846c5ab929b237267844364b5f30b75220406e6e5" exitCode=0
Sep 16 14:22:58.149056 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:58.148666    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2fzwr"
Sep 16 14:22:58.149056 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:58.148709    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2fzwr" event={"ID":"ec5a4b62-1ee5-45d4-a301-047cec60244b","Type":"ContainerDied","Data":"4dc6989f3f58a0ec01986c8846c5ab929b237267844364b5f30b75220406e6e5"}
Sep 16 14:22:58.149056 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:58.148751    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2fzwr" event={"ID":"ec5a4b62-1ee5-45d4-a301-047cec60244b","Type":"ContainerDied","Data":"751aabceb767cf3fa9466a41c1de4cb506b41312fed8dec68854e26ad5dbc77b"}
Sep 16 14:22:58.149056 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:58.148770    2587 scope.go:117] "RemoveContainer" containerID="4dc6989f3f58a0ec01986c8846c5ab929b237267844364b5f30b75220406e6e5"
Sep 16 14:22:58.157137 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:58.157089    2587 scope.go:117] "RemoveContainer" containerID="4dc6989f3f58a0ec01986c8846c5ab929b237267844364b5f30b75220406e6e5"
Sep 16 14:22:58.157356 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:22:58.157337    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dc6989f3f58a0ec01986c8846c5ab929b237267844364b5f30b75220406e6e5\": container with ID starting with 4dc6989f3f58a0ec01986c8846c5ab929b237267844364b5f30b75220406e6e5 not found: ID does not exist" containerID="4dc6989f3f58a0ec01986c8846c5ab929b237267844364b5f30b75220406e6e5"
Sep 16 14:22:58.157356 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:58.157368    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc6989f3f58a0ec01986c8846c5ab929b237267844364b5f30b75220406e6e5"} err="failed to get container status \"4dc6989f3f58a0ec01986c8846c5ab929b237267844364b5f30b75220406e6e5\": rpc error: code = NotFound desc = could not find container \"4dc6989f3f58a0ec01986c8846c5ab929b237267844364b5f30b75220406e6e5\": container with ID starting with 4dc6989f3f58a0ec01986c8846c5ab929b237267844364b5f30b75220406e6e5 not found: ID does not exist"
Sep 16 14:22:58.175122 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:58.175100    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2fzwr"]
Sep 16 14:22:58.176433 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:58.176415    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2fzwr"]
Sep 16 14:22:59.901636 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:22:59.901608    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec5a4b62-1ee5-45d4-a301-047cec60244b" path="/var/lib/kubelet/pods/ec5a4b62-1ee5-45d4-a301-047cec60244b/volumes"
Sep 16 14:23:02.885517 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:02.885438    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d46d94b58-wxggj"]
Sep 16 14:23:02.886127 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:02.885797    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec5a4b62-1ee5-45d4-a301-047cec60244b" containerName="authorino"
Sep 16 14:23:02.886127 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:02.885810    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec5a4b62-1ee5-45d4-a301-047cec60244b" containerName="authorino"
Sep 16 14:23:02.886127 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:02.885856    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec5a4b62-1ee5-45d4-a301-047cec60244b" containerName="authorino"
Sep 16 14:23:02.890155 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:02.890138    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d46d94b58-wxggj"
Sep 16 14:23:02.894899 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:02.894874    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 16 14:23:02.894899 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:02.894869    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 16 14:23:02.895084 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:02.894913    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-s8vl9\""
Sep 16 14:23:02.895150 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:02.895136    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 16 14:23:02.902671 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:02.902649    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d46d94b58-wxggj"]
Sep 16 14:23:03.061216 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:03.061183    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da5ee3c7-9059-4506-9246-1d395a659bf2-metrics-certs\") pod \"maas-controller-6d46d94b58-wxggj\" (UID: \"da5ee3c7-9059-4506-9246-1d395a659bf2\") " pod="opendatahub/maas-controller-6d46d94b58-wxggj"
Sep 16 14:23:03.061378 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:03.061239    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da5ee3c7-9059-4506-9246-1d395a659bf2-webhook-cert\") pod \"maas-controller-6d46d94b58-wxggj\" (UID: \"da5ee3c7-9059-4506-9246-1d395a659bf2\") " pod="opendatahub/maas-controller-6d46d94b58-wxggj"
Sep 16 14:23:03.061378 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:03.061264    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfgsp\" (UniqueName: \"kubernetes.io/projected/da5ee3c7-9059-4506-9246-1d395a659bf2-kube-api-access-pfgsp\") pod \"maas-controller-6d46d94b58-wxggj\" (UID: \"da5ee3c7-9059-4506-9246-1d395a659bf2\") " pod="opendatahub/maas-controller-6d46d94b58-wxggj"
Sep 16 14:23:03.161957 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:03.161930    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da5ee3c7-9059-4506-9246-1d395a659bf2-metrics-certs\") pod \"maas-controller-6d46d94b58-wxggj\" (UID: \"da5ee3c7-9059-4506-9246-1d395a659bf2\") " pod="opendatahub/maas-controller-6d46d94b58-wxggj"
Sep 16 14:23:03.162090 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:03.161981    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da5ee3c7-9059-4506-9246-1d395a659bf2-webhook-cert\") pod \"maas-controller-6d46d94b58-wxggj\" (UID: \"da5ee3c7-9059-4506-9246-1d395a659bf2\") " pod="opendatahub/maas-controller-6d46d94b58-wxggj"
Sep 16 14:23:03.162090 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:03.162009    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfgsp\" (UniqueName: \"kubernetes.io/projected/da5ee3c7-9059-4506-9246-1d395a659bf2-kube-api-access-pfgsp\") pod \"maas-controller-6d46d94b58-wxggj\" (UID: \"da5ee3c7-9059-4506-9246-1d395a659bf2\") " pod="opendatahub/maas-controller-6d46d94b58-wxggj"
Sep 16 14:23:03.164568 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:03.164540    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da5ee3c7-9059-4506-9246-1d395a659bf2-metrics-certs\") pod \"maas-controller-6d46d94b58-wxggj\" (UID: \"da5ee3c7-9059-4506-9246-1d395a659bf2\") " pod="opendatahub/maas-controller-6d46d94b58-wxggj"
Sep 16 14:23:03.164690 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:03.164580    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da5ee3c7-9059-4506-9246-1d395a659bf2-webhook-cert\") pod \"maas-controller-6d46d94b58-wxggj\" (UID: \"da5ee3c7-9059-4506-9246-1d395a659bf2\") " pod="opendatahub/maas-controller-6d46d94b58-wxggj"
Sep 16 14:23:03.180931 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:03.180910    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfgsp\" (UniqueName: \"kubernetes.io/projected/da5ee3c7-9059-4506-9246-1d395a659bf2-kube-api-access-pfgsp\") pod \"maas-controller-6d46d94b58-wxggj\" (UID: \"da5ee3c7-9059-4506-9246-1d395a659bf2\") " pod="opendatahub/maas-controller-6d46d94b58-wxggj"
Sep 16 14:23:03.199915 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:03.199885    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d46d94b58-wxggj"
Sep 16 14:23:03.484910 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:03.484883    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d46d94b58-wxggj"]
Sep 16 14:23:03.486717 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:23:03.486682    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda5ee3c7_9059_4506_9246_1d395a659bf2.slice/crio-54fa3c135af56311026f887fe13062a188baba377e14b04cae5d0cb7c4e6e66e WatchSource:0}: Error finding container 54fa3c135af56311026f887fe13062a188baba377e14b04cae5d0cb7c4e6e66e: Status 404 returned error can't find the container with id 54fa3c135af56311026f887fe13062a188baba377e14b04cae5d0cb7c4e6e66e
Sep 16 14:23:04.176112 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:04.176076    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d46d94b58-wxggj" event={"ID":"da5ee3c7-9059-4506-9246-1d395a659bf2","Type":"ContainerStarted","Data":"54fa3c135af56311026f887fe13062a188baba377e14b04cae5d0cb7c4e6e66e"}
Sep 16 14:23:06.185947 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:06.185865    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d46d94b58-wxggj" event={"ID":"da5ee3c7-9059-4506-9246-1d395a659bf2","Type":"ContainerStarted","Data":"1d87dc0114649c5cb3b601b5fa6cd6f5f0199ab13f20ec951408d881c63adcfc"}
Sep 16 14:23:06.186288 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:06.185955    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d46d94b58-wxggj"
Sep 16 14:23:06.227680 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:06.227634    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6d46d94b58-wxggj" podStartSLOduration=1.8087586180000002 podStartE2EDuration="4.227619995s" totalImagesPullingTime="2.418861377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:23:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:23:06.22548088 +0000 UTC m=+534.829105912" watchObservedRunningTime="2026-09-16 14:23:06.227619995 +0000 UTC m=+534.831245027"
Sep 16 14:23:17.192439 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:17.192409    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d46d94b58-wxggj"
Sep 16 14:23:39.910534 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:39.910502    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-x4c4q"]
Sep 16 14:23:39.916898 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:39.916878    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x4c4q"
Sep 16 14:23:39.919993 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:39.919970    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 16 14:23:39.930100 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:39.930073    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-x4c4q"]
Sep 16 14:23:39.971833 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:39.971798    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a2432ed-9c4b-4fdf-8f68-bcff18302d6a-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-x4c4q\" (UID: \"4a2432ed-9c4b-4fdf-8f68-bcff18302d6a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x4c4q"
Sep 16 14:23:39.972011 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:39.971851    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79hjk\" (UniqueName: \"kubernetes.io/projected/4a2432ed-9c4b-4fdf-8f68-bcff18302d6a-kube-api-access-79hjk\") pod \"payload-pre-processing-59fbc47bf8-x4c4q\" (UID: \"4a2432ed-9c4b-4fdf-8f68-bcff18302d6a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x4c4q"
Sep 16 14:23:40.073143 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:40.073103    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a2432ed-9c4b-4fdf-8f68-bcff18302d6a-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-x4c4q\" (UID: \"4a2432ed-9c4b-4fdf-8f68-bcff18302d6a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x4c4q"
Sep 16 14:23:40.073322 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:40.073154    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79hjk\" (UniqueName: \"kubernetes.io/projected/4a2432ed-9c4b-4fdf-8f68-bcff18302d6a-kube-api-access-79hjk\") pod \"payload-pre-processing-59fbc47bf8-x4c4q\" (UID: \"4a2432ed-9c4b-4fdf-8f68-bcff18302d6a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x4c4q"
Sep 16 14:23:40.074096 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:40.074073    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a2432ed-9c4b-4fdf-8f68-bcff18302d6a-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-x4c4q\" (UID: \"4a2432ed-9c4b-4fdf-8f68-bcff18302d6a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x4c4q"
Sep 16 14:23:40.082824 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:40.082795    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79hjk\" (UniqueName: \"kubernetes.io/projected/4a2432ed-9c4b-4fdf-8f68-bcff18302d6a-kube-api-access-79hjk\") pod \"payload-pre-processing-59fbc47bf8-x4c4q\" (UID: \"4a2432ed-9c4b-4fdf-8f68-bcff18302d6a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x4c4q"
Sep 16 14:23:40.226765 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:40.226678    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x4c4q"
Sep 16 14:23:40.502204 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:40.502119    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-x4c4q"]
Sep 16 14:23:40.505275 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:23:40.505248    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2432ed_9c4b_4fdf_8f68_bcff18302d6a.slice/crio-7d68126fc2a96a71010fc98e36cf83e3dfcbff17c994421de3d5e2d44eee3505 WatchSource:0}: Error finding container 7d68126fc2a96a71010fc98e36cf83e3dfcbff17c994421de3d5e2d44eee3505: Status 404 returned error can't find the container with id 7d68126fc2a96a71010fc98e36cf83e3dfcbff17c994421de3d5e2d44eee3505
Sep 16 14:23:41.341028 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:41.340989    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x4c4q" event={"ID":"4a2432ed-9c4b-4fdf-8f68-bcff18302d6a","Type":"ContainerStarted","Data":"7d68126fc2a96a71010fc98e36cf83e3dfcbff17c994421de3d5e2d44eee3505"}
Sep 16 14:23:42.347034 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:42.346996    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x4c4q" event={"ID":"4a2432ed-9c4b-4fdf-8f68-bcff18302d6a","Type":"ContainerStarted","Data":"ae523ac2b6cf0ba3ea72110357c2c702fb6b311727e1db004e313fe6c5a48283"}
Sep 16 14:23:42.347514 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:42.347064    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x4c4q"
Sep 16 14:23:53.353323 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:53.353294    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x4c4q"
Sep 16 14:23:53.382250 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:23:53.382200    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-x4c4q" podStartSLOduration=13.117160683 podStartE2EDuration="14.382184982s" totalImagesPullingTime="1.265024299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:23:42.367952498 +0000 UTC m=+570.971577531" watchObservedRunningTime="2026-09-16 14:23:53.382184982 +0000 UTC m=+581.985810059"
Sep 16 14:24:02.123223 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:02.123193    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2"]
Sep 16 14:24:02.126732 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:02.126712    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2"
Sep 16 14:24:02.129627 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:02.129601    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 14:24:02.129730 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:02.129708    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 14:24:02.130440 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:02.130424    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bm6t2\""
Sep 16 14:24:02.130514 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:02.130460    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 16 14:24:02.137212 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:02.137193    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2"]
Sep 16 14:24:02.248411 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:02.248359    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tv9x\" (UniqueName: \"kubernetes.io/projected/7c741af3-761f-4a25-9752-64324d62cb5f-kube-api-access-8tv9x\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2\" (UID: \"7c741af3-761f-4a25-9752-64324d62cb5f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2"
Sep 16 14:24:02.248572 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:02.248421    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c741af3-761f-4a25-9752-64324d62cb5f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2\" (UID: \"7c741af3-761f-4a25-9752-64324d62cb5f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2"
Sep 16 14:24:02.248572 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:02.248467    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c741af3-761f-4a25-9752-64324d62cb5f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2\" (UID: \"7c741af3-761f-4a25-9752-64324d62cb5f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2"
Sep 16 14:24:02.248572 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:02.248488    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c741af3-761f-4a25-9752-64324d62cb5f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2\" (UID: \"7c741af3-761f-4a25-9752-64324d62cb5f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2"
Sep 16 14:24:02.248572 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:02.248503    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c741af3-761f-4a25-9752-64324d62cb5f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2\" (UID: \"7c741af3-761f-4a25-9752-64324d62cb5f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2"
Sep 16 14:24:02.248572 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:02.248522    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c741af3-761f-4a25-9752-64324d62cb5f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2\" (UID: \"7c741af3-761f-4a25-9752-64324d62cb5f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2"
Sep 16 14:24:02.349636 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:02.349602    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tv9x\" (UniqueName: \"kubernetes.io/projected/7c741af3-761f-4a25-9752-64324d62cb5f-kube-api-access-8tv9x\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2\" (UID: \"7c741af3-761f-4a25-9752-64324d62cb5f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2"
Sep 16 14:24:02.349795 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:02.349647    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c741af3-761f-4a25-9752-64324d62cb5f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2\" (UID: \"7c741af3-761f-4a25-9752-64324d62cb5f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2"
Sep 16 14:24:02.349795 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:02.349770    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c741af3-761f-4a25-9752-64324d62cb5f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2\" (UID: \"7c741af3-761f-4a25-9752-64324d62cb5f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2"
Sep 16 14:24:02.349918 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:02.349807    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c741af3-761f-4a25-9752-64324d62cb5f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2\" (UID: \"7c741af3-761f-4a25-9752-64324d62cb5f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2"
Sep 16 14:24:02.349918 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:02.349825    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c741af3-761f-4a25-9752-64324d62cb5f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2\" (UID: \"7c741af3-761f-4a25-9752-64324d62cb5f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2"
Sep 16 14:24:02.349918 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:02.349846    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c741af3-761f-4a25-9752-64324d62cb5f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2\" (UID: \"7c741af3-761f-4a25-9752-64324d62cb5f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2"
Sep 16 14:24:02.350090 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:02.350069    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c741af3-761f-4a25-9752-64324d62cb5f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2\" (UID: \"7c741af3-761f-4a25-9752-64324d62cb5f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2"
Sep 16 14:24:02.350237 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:02.350211    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c741af3-761f-4a25-9752-64324d62cb5f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2\" (UID: \"7c741af3-761f-4a25-9752-64324d62cb5f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2"
Sep 16 14:24:02.350237 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:02.350232    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c741af3-761f-4a25-9752-64324d62cb5f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2\" (UID: \"7c741af3-761f-4a25-9752-64324d62cb5f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2"
Sep 16 14:24:02.352247 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:02.352218    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c741af3-761f-4a25-9752-64324d62cb5f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2\" (UID: \"7c741af3-761f-4a25-9752-64324d62cb5f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2"
Sep 16 14:24:02.352494 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:02.352477    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c741af3-761f-4a25-9752-64324d62cb5f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2\" (UID: \"7c741af3-761f-4a25-9752-64324d62cb5f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2"
Sep 16 14:24:02.359893 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:02.359869    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tv9x\" (UniqueName: \"kubernetes.io/projected/7c741af3-761f-4a25-9752-64324d62cb5f-kube-api-access-8tv9x\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2\" (UID: \"7c741af3-761f-4a25-9752-64324d62cb5f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2"
Sep 16 14:24:02.438978 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:02.438947    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2"
Sep 16 14:24:02.731251 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:02.731225    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2"]
Sep 16 14:24:02.733688 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:24:02.733661    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c741af3_761f_4a25_9752_64324d62cb5f.slice/crio-a79d68026c79fea9bfd31724dfa2c44eda5f6a32bf81459c3a300dc2c7988acf WatchSource:0}: Error finding container a79d68026c79fea9bfd31724dfa2c44eda5f6a32bf81459c3a300dc2c7988acf: Status 404 returned error can't find the container with id a79d68026c79fea9bfd31724dfa2c44eda5f6a32bf81459c3a300dc2c7988acf
Sep 16 14:24:03.436682 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:03.436639    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2" event={"ID":"7c741af3-761f-4a25-9752-64324d62cb5f","Type":"ContainerStarted","Data":"a79d68026c79fea9bfd31724dfa2c44eda5f6a32bf81459c3a300dc2c7988acf"}
Sep 16 14:24:04.442431 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:04.442374    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2" event={"ID":"7c741af3-761f-4a25-9752-64324d62cb5f","Type":"ContainerStarted","Data":"1300b0e805b2c1eb8fadf7b60b59a145959d7f6d17bdb8ddbdee22991ce65627"}
Sep 16 14:24:04.462636 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:04.462585    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2" podStartSLOduration=1.264438009 podStartE2EDuration="2.462570296s" totalImagesPullingTime="1.198132287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:24:04.461916154 +0000 UTC m=+593.065541187" watchObservedRunningTime="2026-09-16 14:24:04.462570296 +0000 UTC m=+593.066195368"
Sep 16 14:24:12.439317 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:12.439280    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2"
Sep 16 14:24:12.441881 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:12.439349    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2"
Sep 16 14:24:12.452105 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:12.452080    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2"
Sep 16 14:24:12.487521 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:24:12.487492    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2"
Sep 16 14:25:27.386854 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:27.386823    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv"]
Sep 16 14:25:27.390059 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:27.390040    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv"
Sep 16 14:25:27.395546 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:27.395525    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 16 14:25:27.416756 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:27.416726    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv"]
Sep 16 14:25:27.459368 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:27.459331    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a04036-268e-45c5-815a-12b48c81be4b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mw7qv\" (UID: \"23a04036-268e-45c5-815a-12b48c81be4b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv"
Sep 16 14:25:27.459368 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:27.459369    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23a04036-268e-45c5-815a-12b48c81be4b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mw7qv\" (UID: \"23a04036-268e-45c5-815a-12b48c81be4b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv"
Sep 16 14:25:27.459614 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:27.459413    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23a04036-268e-45c5-815a-12b48c81be4b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mw7qv\" (UID: \"23a04036-268e-45c5-815a-12b48c81be4b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv"
Sep 16 14:25:27.459614 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:27.459514    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6rkv\" (UniqueName: \"kubernetes.io/projected/23a04036-268e-45c5-815a-12b48c81be4b-kube-api-access-x6rkv\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mw7qv\" (UID: \"23a04036-268e-45c5-815a-12b48c81be4b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv"
Sep 16 14:25:27.459614 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:27.459553    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23a04036-268e-45c5-815a-12b48c81be4b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mw7qv\" (UID: \"23a04036-268e-45c5-815a-12b48c81be4b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv"
Sep 16 14:25:27.459614 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:27.459601    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a04036-268e-45c5-815a-12b48c81be4b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mw7qv\" (UID: \"23a04036-268e-45c5-815a-12b48c81be4b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv"
Sep 16 14:25:27.560784 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:27.560688    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23a04036-268e-45c5-815a-12b48c81be4b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mw7qv\" (UID: \"23a04036-268e-45c5-815a-12b48c81be4b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv"
Sep 16 14:25:27.560938 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:27.560839    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a04036-268e-45c5-815a-12b48c81be4b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mw7qv\" (UID: \"23a04036-268e-45c5-815a-12b48c81be4b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv"
Sep 16 14:25:27.560938 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:27.560894    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a04036-268e-45c5-815a-12b48c81be4b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mw7qv\" (UID: \"23a04036-268e-45c5-815a-12b48c81be4b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv"
Sep 16 14:25:27.560938 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:27.560925    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23a04036-268e-45c5-815a-12b48c81be4b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mw7qv\" (UID: \"23a04036-268e-45c5-815a-12b48c81be4b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv"
Sep 16 14:25:27.561049 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:27.560954    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23a04036-268e-45c5-815a-12b48c81be4b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mw7qv\" (UID: \"23a04036-268e-45c5-815a-12b48c81be4b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv"
Sep 16 14:25:27.561049 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:27.561012    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6rkv\" (UniqueName: \"kubernetes.io/projected/23a04036-268e-45c5-815a-12b48c81be4b-kube-api-access-x6rkv\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mw7qv\" (UID: \"23a04036-268e-45c5-815a-12b48c81be4b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv"
Sep 16 14:25:27.561268 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:27.561242    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a04036-268e-45c5-815a-12b48c81be4b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mw7qv\" (UID: \"23a04036-268e-45c5-815a-12b48c81be4b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv"
Sep 16 14:25:27.561268 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:27.561258    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23a04036-268e-45c5-815a-12b48c81be4b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mw7qv\" (UID: \"23a04036-268e-45c5-815a-12b48c81be4b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv"
Sep 16 14:25:27.561449 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:27.561345    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23a04036-268e-45c5-815a-12b48c81be4b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mw7qv\" (UID: \"23a04036-268e-45c5-815a-12b48c81be4b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv"
Sep 16 14:25:27.562996 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:27.562974    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23a04036-268e-45c5-815a-12b48c81be4b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mw7qv\" (UID: \"23a04036-268e-45c5-815a-12b48c81be4b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv"
Sep 16 14:25:27.563417 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:27.563377    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a04036-268e-45c5-815a-12b48c81be4b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mw7qv\" (UID: \"23a04036-268e-45c5-815a-12b48c81be4b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv"
Sep 16 14:25:27.572669 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:27.572642    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6rkv\" (UniqueName: \"kubernetes.io/projected/23a04036-268e-45c5-815a-12b48c81be4b-kube-api-access-x6rkv\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mw7qv\" (UID: \"23a04036-268e-45c5-815a-12b48c81be4b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv"
Sep 16 14:25:27.700226 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:27.700198    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv"
Sep 16 14:25:27.974955 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:27.974929    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv"]
Sep 16 14:25:27.976991 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:25:27.976966    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a04036_268e_45c5_815a_12b48c81be4b.slice/crio-0d3310b6c81b1651070d16586f6c31b79365fb37228d0f6c083903b50c9df742 WatchSource:0}: Error finding container 0d3310b6c81b1651070d16586f6c31b79365fb37228d0f6c083903b50c9df742: Status 404 returned error can't find the container with id 0d3310b6c81b1651070d16586f6c31b79365fb37228d0f6c083903b50c9df742
Sep 16 14:25:27.978262 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:27.978241    2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 14:25:28.807105 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:28.807071    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv" event={"ID":"23a04036-268e-45c5-815a-12b48c81be4b","Type":"ContainerStarted","Data":"c8d79e72d755d91d73ed257aeac6f27d7b2380f40501a285a4d9c7ef34ffcb43"}
Sep 16 14:25:28.807486 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:28.807120    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv" event={"ID":"23a04036-268e-45c5-815a-12b48c81be4b","Type":"ContainerStarted","Data":"0d3310b6c81b1651070d16586f6c31b79365fb37228d0f6c083903b50c9df742"}
Sep 16 14:25:28.828327 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:28.828259    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv" podStartSLOduration=1.604283763 podStartE2EDuration="1.828235863s" totalImagesPullingTime="223.9521ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:25:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:25:28.826557987 +0000 UTC m=+677.430183019" watchObservedRunningTime="2026-09-16 14:25:28.828235863 +0000 UTC m=+677.431860900"
Sep 16 14:25:37.701251 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:37.701211    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv"
Sep 16 14:25:37.701707 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:37.701284    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv"
Sep 16 14:25:37.713676 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:37.713653    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv"
Sep 16 14:25:37.856976 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:37.856942    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mw7qv"
Sep 16 14:25:37.955887 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:37.955811    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh"]
Sep 16 14:25:37.959319 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:37.959297    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh"
Sep 16 14:25:37.965559 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:37.965540    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 16 14:25:37.980024 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:37.979997    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh"]
Sep 16 14:25:38.056903 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:38.056871    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a32ca17-c160-4fa0-b9a2-67c13d1ffe00-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh\" (UID: \"8a32ca17-c160-4fa0-b9a2-67c13d1ffe00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh"
Sep 16 14:25:38.056903 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:38.056917    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a32ca17-c160-4fa0-b9a2-67c13d1ffe00-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh\" (UID: \"8a32ca17-c160-4fa0-b9a2-67c13d1ffe00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh"
Sep 16 14:25:38.057119 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:38.056941    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a32ca17-c160-4fa0-b9a2-67c13d1ffe00-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh\" (UID: \"8a32ca17-c160-4fa0-b9a2-67c13d1ffe00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh"
Sep 16 14:25:38.057119 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:38.057059    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m9sj\" (UniqueName: \"kubernetes.io/projected/8a32ca17-c160-4fa0-b9a2-67c13d1ffe00-kube-api-access-5m9sj\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh\" (UID: \"8a32ca17-c160-4fa0-b9a2-67c13d1ffe00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh"
Sep 16 14:25:38.057119 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:38.057104    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a32ca17-c160-4fa0-b9a2-67c13d1ffe00-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh\" (UID: \"8a32ca17-c160-4fa0-b9a2-67c13d1ffe00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh"
Sep 16 14:25:38.057245 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:38.057121    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a32ca17-c160-4fa0-b9a2-67c13d1ffe00-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh\" (UID: \"8a32ca17-c160-4fa0-b9a2-67c13d1ffe00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh"
Sep 16 14:25:38.158343 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:38.158292    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m9sj\" (UniqueName: \"kubernetes.io/projected/8a32ca17-c160-4fa0-b9a2-67c13d1ffe00-kube-api-access-5m9sj\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh\" (UID: \"8a32ca17-c160-4fa0-b9a2-67c13d1ffe00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh"
Sep 16 14:25:38.158343 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:38.158353    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a32ca17-c160-4fa0-b9a2-67c13d1ffe00-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh\" (UID: \"8a32ca17-c160-4fa0-b9a2-67c13d1ffe00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh"
Sep 16 14:25:38.158641 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:38.158511    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a32ca17-c160-4fa0-b9a2-67c13d1ffe00-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh\" (UID: \"8a32ca17-c160-4fa0-b9a2-67c13d1ffe00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh"
Sep 16 14:25:38.158641 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:38.158546    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a32ca17-c160-4fa0-b9a2-67c13d1ffe00-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh\" (UID: \"8a32ca17-c160-4fa0-b9a2-67c13d1ffe00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh"
Sep 16 14:25:38.158641 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:38.158584    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a32ca17-c160-4fa0-b9a2-67c13d1ffe00-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh\" (UID: \"8a32ca17-c160-4fa0-b9a2-67c13d1ffe00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh"
Sep 16 14:25:38.158641 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:38.158615    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a32ca17-c160-4fa0-b9a2-67c13d1ffe00-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh\" (UID: \"8a32ca17-c160-4fa0-b9a2-67c13d1ffe00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh"
Sep 16 14:25:38.158963 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:38.158940    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a32ca17-c160-4fa0-b9a2-67c13d1ffe00-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh\" (UID: \"8a32ca17-c160-4fa0-b9a2-67c13d1ffe00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh"
Sep 16 14:25:38.159037 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:38.158972    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a32ca17-c160-4fa0-b9a2-67c13d1ffe00-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh\" (UID: \"8a32ca17-c160-4fa0-b9a2-67c13d1ffe00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh"
Sep 16 14:25:38.159037 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:38.159001    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a32ca17-c160-4fa0-b9a2-67c13d1ffe00-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh\" (UID: \"8a32ca17-c160-4fa0-b9a2-67c13d1ffe00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh"
Sep 16 14:25:38.160681 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:38.160662    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a32ca17-c160-4fa0-b9a2-67c13d1ffe00-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh\" (UID: \"8a32ca17-c160-4fa0-b9a2-67c13d1ffe00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh"
Sep 16 14:25:38.160994 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:38.160975    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a32ca17-c160-4fa0-b9a2-67c13d1ffe00-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh\" (UID: \"8a32ca17-c160-4fa0-b9a2-67c13d1ffe00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh"
Sep 16 14:25:38.171202 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:38.171171    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m9sj\" (UniqueName: \"kubernetes.io/projected/8a32ca17-c160-4fa0-b9a2-67c13d1ffe00-kube-api-access-5m9sj\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh\" (UID: \"8a32ca17-c160-4fa0-b9a2-67c13d1ffe00\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh"
Sep 16 14:25:38.268726 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:38.268634    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh"
Sep 16 14:25:38.550997 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:38.550972    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh"]
Sep 16 14:25:38.552552 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:25:38.552509    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a32ca17_c160_4fa0_b9a2_67c13d1ffe00.slice/crio-5720397c15b4e16710e112a8ca2eb906662907f768e358134c542323fa27fb18 WatchSource:0}: Error finding container 5720397c15b4e16710e112a8ca2eb906662907f768e358134c542323fa27fb18: Status 404 returned error can't find the container with id 5720397c15b4e16710e112a8ca2eb906662907f768e358134c542323fa27fb18
Sep 16 14:25:38.851847 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:38.851815    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh" event={"ID":"8a32ca17-c160-4fa0-b9a2-67c13d1ffe00","Type":"ContainerStarted","Data":"9b1c8b62eedfb2f3e1309b8a34f98669484bd3f3fdd496000e206d67bac00d3e"}
Sep 16 14:25:38.851847 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:38.851857    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh" event={"ID":"8a32ca17-c160-4fa0-b9a2-67c13d1ffe00","Type":"ContainerStarted","Data":"5720397c15b4e16710e112a8ca2eb906662907f768e358134c542323fa27fb18"}
Sep 16 14:25:38.877304 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:38.877252    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh" podStartSLOduration=1.6765756330000001 podStartE2EDuration="1.877233651s" totalImagesPullingTime="200.658018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:25:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:25:38.875704864 +0000 UTC m=+687.479329899" watchObservedRunningTime="2026-09-16 14:25:38.877233651 +0000 UTC m=+687.480858722"
Sep 16 14:25:48.269550 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:48.269514    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh"
Sep 16 14:25:48.269550 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:48.269560    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh"
Sep 16 14:25:48.282315 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:48.282290    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh"
Sep 16 14:25:48.908082 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:25:48.908048    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh"
Sep 16 14:26:55.061165 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:55.061127    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7597cddb56-6jn88"]
Sep 16 14:26:55.064586 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:55.064564    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7597cddb56-6jn88"
Sep 16 14:26:55.072025 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:55.072000    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7597cddb56-6jn88"]
Sep 16 14:26:55.147314 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:55.147281    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24zpr\" (UniqueName: \"kubernetes.io/projected/1dbe5ecc-4d35-41ae-b95c-bd86da658146-kube-api-access-24zpr\") pod \"authorino-7597cddb56-6jn88\" (UID: \"1dbe5ecc-4d35-41ae-b95c-bd86da658146\") " pod="kuadrant-system/authorino-7597cddb56-6jn88"
Sep 16 14:26:55.147499 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:55.147320    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1dbe5ecc-4d35-41ae-b95c-bd86da658146-tls-cert\") pod \"authorino-7597cddb56-6jn88\" (UID: \"1dbe5ecc-4d35-41ae-b95c-bd86da658146\") " pod="kuadrant-system/authorino-7597cddb56-6jn88"
Sep 16 14:26:55.248716 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:55.248683    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24zpr\" (UniqueName: \"kubernetes.io/projected/1dbe5ecc-4d35-41ae-b95c-bd86da658146-kube-api-access-24zpr\") pod \"authorino-7597cddb56-6jn88\" (UID: \"1dbe5ecc-4d35-41ae-b95c-bd86da658146\") " pod="kuadrant-system/authorino-7597cddb56-6jn88"
Sep 16 14:26:55.248716 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:55.248717    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1dbe5ecc-4d35-41ae-b95c-bd86da658146-tls-cert\") pod \"authorino-7597cddb56-6jn88\" (UID: \"1dbe5ecc-4d35-41ae-b95c-bd86da658146\") " pod="kuadrant-system/authorino-7597cddb56-6jn88"
Sep 16 14:26:55.251269 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:55.251240    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1dbe5ecc-4d35-41ae-b95c-bd86da658146-tls-cert\") pod \"authorino-7597cddb56-6jn88\" (UID: \"1dbe5ecc-4d35-41ae-b95c-bd86da658146\") " pod="kuadrant-system/authorino-7597cddb56-6jn88"
Sep 16 14:26:55.258174 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:55.258147    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24zpr\" (UniqueName: \"kubernetes.io/projected/1dbe5ecc-4d35-41ae-b95c-bd86da658146-kube-api-access-24zpr\") pod \"authorino-7597cddb56-6jn88\" (UID: \"1dbe5ecc-4d35-41ae-b95c-bd86da658146\") " pod="kuadrant-system/authorino-7597cddb56-6jn88"
Sep 16 14:26:55.373510 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:55.373434    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7597cddb56-6jn88"
Sep 16 14:26:55.643968 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:55.643940    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7597cddb56-6jn88"]
Sep 16 14:26:55.645885 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:26:55.645856    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1dbe5ecc_4d35_41ae_b95c_bd86da658146.slice/crio-0508b3e18564d3eb899d81da16b86fe01c0d107f834ed8e22372aa5a40c27cbc WatchSource:0}: Error finding container 0508b3e18564d3eb899d81da16b86fe01c0d107f834ed8e22372aa5a40c27cbc: Status 404 returned error can't find the container with id 0508b3e18564d3eb899d81da16b86fe01c0d107f834ed8e22372aa5a40c27cbc
Sep 16 14:26:56.196224 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:56.196184    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7597cddb56-6jn88" event={"ID":"1dbe5ecc-4d35-41ae-b95c-bd86da658146","Type":"ContainerStarted","Data":"0508b3e18564d3eb899d81da16b86fe01c0d107f834ed8e22372aa5a40c27cbc"}
Sep 16 14:26:57.204468 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:57.204423    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7597cddb56-6jn88" event={"ID":"1dbe5ecc-4d35-41ae-b95c-bd86da658146","Type":"ContainerStarted","Data":"629e1ffc827478a0873a11eca85f8a7615298af242b69553610033c22d63ea53"}
Sep 16 14:26:57.227468 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:57.227410    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7597cddb56-6jn88" podStartSLOduration=0.837171607 podStartE2EDuration="2.227373826s" totalImagesPullingTime="1.390202219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:26:57.226424395 +0000 UTC m=+765.830049430" watchObservedRunningTime="2026-09-16 14:26:57.227373826 +0000 UTC m=+765.830998859"
Sep 16 14:26:57.292212 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:57.292117    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5954586d97-n5t7c"]
Sep 16 14:26:57.292377 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:57.292344    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5954586d97-n5t7c" podUID="7d1f8393-db8f-4305-b8be-59cf0ffc3578" containerName="authorino" containerID="cri-o://95c4d94b63e0c90f0043d7774d04ee0b5c8cc0e9d10163075006a557b772d2cb" gracePeriod=30
Sep 16 14:26:57.562262 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:57.562232    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5954586d97-n5t7c"
Sep 16 14:26:57.668821 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:57.668791    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjf89\" (UniqueName: \"kubernetes.io/projected/7d1f8393-db8f-4305-b8be-59cf0ffc3578-kube-api-access-tjf89\") pod \"7d1f8393-db8f-4305-b8be-59cf0ffc3578\" (UID: \"7d1f8393-db8f-4305-b8be-59cf0ffc3578\") "
Sep 16 14:26:57.668980 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:57.668854    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d1f8393-db8f-4305-b8be-59cf0ffc3578-tls-cert\") pod \"7d1f8393-db8f-4305-b8be-59cf0ffc3578\" (UID: \"7d1f8393-db8f-4305-b8be-59cf0ffc3578\") "
Sep 16 14:26:57.671486 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:57.671459    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d1f8393-db8f-4305-b8be-59cf0ffc3578-kube-api-access-tjf89" (OuterVolumeSpecName: "kube-api-access-tjf89") pod "7d1f8393-db8f-4305-b8be-59cf0ffc3578" (UID: "7d1f8393-db8f-4305-b8be-59cf0ffc3578"). InnerVolumeSpecName "kube-api-access-tjf89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:26:57.680043 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:57.680017    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d1f8393-db8f-4305-b8be-59cf0ffc3578-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "7d1f8393-db8f-4305-b8be-59cf0ffc3578" (UID: "7d1f8393-db8f-4305-b8be-59cf0ffc3578"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:26:57.770337 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:57.770304    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjf89\" (UniqueName: \"kubernetes.io/projected/7d1f8393-db8f-4305-b8be-59cf0ffc3578-kube-api-access-tjf89\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:26:57.770337 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:57.770336    2587 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d1f8393-db8f-4305-b8be-59cf0ffc3578-tls-cert\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:26:58.210578 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:58.210502    2587 generic.go:356] "Generic (PLEG): container finished" podID="7d1f8393-db8f-4305-b8be-59cf0ffc3578" containerID="95c4d94b63e0c90f0043d7774d04ee0b5c8cc0e9d10163075006a557b772d2cb" exitCode=0
Sep 16 14:26:58.210578 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:58.210546    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5954586d97-n5t7c"
Sep 16 14:26:58.210978 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:58.210604    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5954586d97-n5t7c" event={"ID":"7d1f8393-db8f-4305-b8be-59cf0ffc3578","Type":"ContainerDied","Data":"95c4d94b63e0c90f0043d7774d04ee0b5c8cc0e9d10163075006a557b772d2cb"}
Sep 16 14:26:58.210978 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:58.210657    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5954586d97-n5t7c" event={"ID":"7d1f8393-db8f-4305-b8be-59cf0ffc3578","Type":"ContainerDied","Data":"88c109226cba014ddd39856af59398169312a175e102ecf788ee53de7b0f60da"}
Sep 16 14:26:58.210978 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:58.210682    2587 scope.go:117] "RemoveContainer" containerID="95c4d94b63e0c90f0043d7774d04ee0b5c8cc0e9d10163075006a557b772d2cb"
Sep 16 14:26:58.223589 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:58.223565    2587 scope.go:117] "RemoveContainer" containerID="95c4d94b63e0c90f0043d7774d04ee0b5c8cc0e9d10163075006a557b772d2cb"
Sep 16 14:26:58.224223 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:26:58.224200    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c4d94b63e0c90f0043d7774d04ee0b5c8cc0e9d10163075006a557b772d2cb\": container with ID starting with 95c4d94b63e0c90f0043d7774d04ee0b5c8cc0e9d10163075006a557b772d2cb not found: ID does not exist" containerID="95c4d94b63e0c90f0043d7774d04ee0b5c8cc0e9d10163075006a557b772d2cb"
Sep 16 14:26:58.224309 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:58.224233    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c4d94b63e0c90f0043d7774d04ee0b5c8cc0e9d10163075006a557b772d2cb"} err="failed to get container status \"95c4d94b63e0c90f0043d7774d04ee0b5c8cc0e9d10163075006a557b772d2cb\": rpc error: code = NotFound desc = could not find container \"95c4d94b63e0c90f0043d7774d04ee0b5c8cc0e9d10163075006a557b772d2cb\": container with ID starting with 95c4d94b63e0c90f0043d7774d04ee0b5c8cc0e9d10163075006a557b772d2cb not found: ID does not exist"
Sep 16 14:26:58.228232 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:58.228208    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5954586d97-n5t7c"]
Sep 16 14:26:58.231344 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:58.231321    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5954586d97-n5t7c"]
Sep 16 14:26:59.901731 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:26:59.901698    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d1f8393-db8f-4305-b8be-59cf0ffc3578" path="/var/lib/kubelet/pods/7d1f8393-db8f-4305-b8be-59cf0ffc3578/volumes"
Sep 16 14:27:20.276620 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:20.276589    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7617-f48ce1"]
Sep 16 14:27:20.276993 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:20.276984    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d1f8393-db8f-4305-b8be-59cf0ffc3578" containerName="authorino"
Sep 16 14:27:20.277035 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:20.276997    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d1f8393-db8f-4305-b8be-59cf0ffc3578" containerName="authorino"
Sep 16 14:27:20.277072 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:20.277056    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d1f8393-db8f-4305-b8be-59cf0ffc3578" containerName="authorino"
Sep 16 14:27:20.281125 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:20.281108    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7617-f48ce1"
Sep 16 14:27:20.283583 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:20.283561    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gn749\""
Sep 16 14:27:20.284963 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:20.284942    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7617-f48ce1"]
Sep 16 14:27:20.375724 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:20.375694    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpcvk\" (UniqueName: \"kubernetes.io/projected/79671c57-57dc-4913-bc5a-acf06a4a5758-kube-api-access-rpcvk\") pod \"test-curl-7617-f48ce1\" (UID: \"79671c57-57dc-4913-bc5a-acf06a4a5758\") " pod="opendatahub/test-curl-7617-f48ce1"
Sep 16 14:27:20.476267 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:20.476226    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpcvk\" (UniqueName: \"kubernetes.io/projected/79671c57-57dc-4913-bc5a-acf06a4a5758-kube-api-access-rpcvk\") pod \"test-curl-7617-f48ce1\" (UID: \"79671c57-57dc-4913-bc5a-acf06a4a5758\") " pod="opendatahub/test-curl-7617-f48ce1"
Sep 16 14:27:20.487501 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:20.487470    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpcvk\" (UniqueName: \"kubernetes.io/projected/79671c57-57dc-4913-bc5a-acf06a4a5758-kube-api-access-rpcvk\") pod \"test-curl-7617-f48ce1\" (UID: \"79671c57-57dc-4913-bc5a-acf06a4a5758\") " pod="opendatahub/test-curl-7617-f48ce1"
Sep 16 14:27:20.591476 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:20.591374    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7617-f48ce1"
Sep 16 14:27:20.871546 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:20.871512    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7617-f48ce1"]
Sep 16 14:27:20.874284 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:27:20.874251    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79671c57_57dc_4913_bc5a_acf06a4a5758.slice/crio-c2d1d1af9bd8d3173f407d379c4d8ce6414f694800540a3d9f697c94f620b788 WatchSource:0}: Error finding container c2d1d1af9bd8d3173f407d379c4d8ce6414f694800540a3d9f697c94f620b788: Status 404 returned error can't find the container with id c2d1d1af9bd8d3173f407d379c4d8ce6414f694800540a3d9f697c94f620b788
Sep 16 14:27:21.321982 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:21.321949    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7617-f48ce1" event={"ID":"79671c57-57dc-4913-bc5a-acf06a4a5758","Type":"ContainerStarted","Data":"c2d1d1af9bd8d3173f407d379c4d8ce6414f694800540a3d9f697c94f620b788"}
Sep 16 14:27:23.333336 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:23.333248    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7617-f48ce1" event={"ID":"79671c57-57dc-4913-bc5a-acf06a4a5758","Type":"ContainerStarted","Data":"195381616c651d0457f9c399beca7c26cea996f44d084d9f5829882aed523486"}
Sep 16 14:27:23.348992 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:23.348928    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7617-f48ce1" podStartSLOduration=1.198899767 podStartE2EDuration="3.348913055s" totalImagesPullingTime="2.150013288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:27:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:27:23.346845772 +0000 UTC m=+791.950470804" watchObservedRunningTime="2026-09-16 14:27:23.348913055 +0000 UTC m=+791.952538087"
Sep 16 14:27:23.984315 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:23.984281    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55"]
Sep 16 14:27:23.987549 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:23.987531    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55"
Sep 16 14:27:23.994306 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:23.994284    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-094dab-dockercfg-5kl7n\""
Sep 16 14:27:23.994423 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:23.994318    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-094dab\""
Sep 16 14:27:24.010553 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.010530    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55"]
Sep 16 14:27:24.050270 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.050242    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf"]
Sep 16 14:27:24.053537 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.053518    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf"
Sep 16 14:27:24.073225 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.072847    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf"]
Sep 16 14:27:24.109207 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.109183    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdcr7\" (UniqueName: \"kubernetes.io/projected/608bb3a3-74d2-4557-8f6a-5961db6fad36-kube-api-access-rdcr7\") pod \"payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55\" (UID: \"608bb3a3-74d2-4557-8f6a-5961db6fad36\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55"
Sep 16 14:27:24.109327 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.109232    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/608bb3a3-74d2-4557-8f6a-5961db6fad36-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55\" (UID: \"608bb3a3-74d2-4557-8f6a-5961db6fad36\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55"
Sep 16 14:27:24.199022 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.198950    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7617-f48ce1"]
Sep 16 14:27:24.204930 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.204896    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7617-f48ce1"]
Sep 16 14:27:24.209948 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.209914    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b3132a-66c9-4a5d-b47b-e3cff8a2b644-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf\" (UID: \"a7b3132a-66c9-4a5d-b47b-e3cff8a2b644\") " pod="openshift-ingress/payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf"
Sep 16 14:27:24.210141 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.210119    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdcr7\" (UniqueName: \"kubernetes.io/projected/608bb3a3-74d2-4557-8f6a-5961db6fad36-kube-api-access-rdcr7\") pod \"payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55\" (UID: \"608bb3a3-74d2-4557-8f6a-5961db6fad36\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55"
Sep 16 14:27:24.210217 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.210159    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5vcm\" (UniqueName: \"kubernetes.io/projected/a7b3132a-66c9-4a5d-b47b-e3cff8a2b644-kube-api-access-q5vcm\") pod \"payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf\" (UID: \"a7b3132a-66c9-4a5d-b47b-e3cff8a2b644\") " pod="openshift-ingress/payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf"
Sep 16 14:27:24.210271 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.210216    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/608bb3a3-74d2-4557-8f6a-5961db6fad36-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55\" (UID: \"608bb3a3-74d2-4557-8f6a-5961db6fad36\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55"
Sep 16 14:27:24.211007 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.210987    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/608bb3a3-74d2-4557-8f6a-5961db6fad36-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55\" (UID: \"608bb3a3-74d2-4557-8f6a-5961db6fad36\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55"
Sep 16 14:27:24.225877 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.225850    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdcr7\" (UniqueName: \"kubernetes.io/projected/608bb3a3-74d2-4557-8f6a-5961db6fad36-kube-api-access-rdcr7\") pod \"payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55\" (UID: \"608bb3a3-74d2-4557-8f6a-5961db6fad36\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55"
Sep 16 14:27:24.298170 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.298146    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55"
Sep 16 14:27:24.310940 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.310915    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5vcm\" (UniqueName: \"kubernetes.io/projected/a7b3132a-66c9-4a5d-b47b-e3cff8a2b644-kube-api-access-q5vcm\") pod \"payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf\" (UID: \"a7b3132a-66c9-4a5d-b47b-e3cff8a2b644\") " pod="openshift-ingress/payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf"
Sep 16 14:27:24.311042 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.310974    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b3132a-66c9-4a5d-b47b-e3cff8a2b644-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf\" (UID: \"a7b3132a-66c9-4a5d-b47b-e3cff8a2b644\") " pod="openshift-ingress/payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf"
Sep 16 14:27:24.311747 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.311715    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b3132a-66c9-4a5d-b47b-e3cff8a2b644-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf\" (UID: \"a7b3132a-66c9-4a5d-b47b-e3cff8a2b644\") " pod="openshift-ingress/payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf"
Sep 16 14:27:24.329104 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.329077    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5vcm\" (UniqueName: \"kubernetes.io/projected/a7b3132a-66c9-4a5d-b47b-e3cff8a2b644-kube-api-access-q5vcm\") pod \"payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf\" (UID: \"a7b3132a-66c9-4a5d-b47b-e3cff8a2b644\") " pod="openshift-ingress/payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf"
Sep 16 14:27:24.339626 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.339517    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7617-57fec3"]
Sep 16 14:27:24.340112 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.340077    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79671c57-57dc-4913-bc5a-acf06a4a5758" containerName="test-curl-7617-f48ce1"
Sep 16 14:27:24.340112 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.340103    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="79671c57-57dc-4913-bc5a-acf06a4a5758" containerName="test-curl-7617-f48ce1"
Sep 16 14:27:24.340254 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.340196    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="79671c57-57dc-4913-bc5a-acf06a4a5758" containerName="test-curl-7617-f48ce1"
Sep 16 14:27:24.341204 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:27:24.341178    2587 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7617-f48ce1" container="test-curl-7617-f48ce1"
Sep 16 14:27:24.344662 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.344645    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7617-57fec3"
Sep 16 14:27:24.353549 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.353151    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7617-57fec3"]
Sep 16 14:27:24.364718 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.363570    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf"
Sep 16 14:27:24.513696 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.513610    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7wzk\" (UniqueName: \"kubernetes.io/projected/722a81b9-eba9-4b21-9d04-907472aec41d-kube-api-access-d7wzk\") pod \"test-curl-7617-57fec3\" (UID: \"722a81b9-eba9-4b21-9d04-907472aec41d\") " pod="opendatahub/test-curl-7617-57fec3"
Sep 16 14:27:24.595551 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.595527    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55"]
Sep 16 14:27:24.596972 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:27:24.596949    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod608bb3a3_74d2_4557_8f6a_5961db6fad36.slice/crio-b3510bc5270da62ec6332e34579e551cdaf16149a90476045ca7816f08ebbaa5 WatchSource:0}: Error finding container b3510bc5270da62ec6332e34579e551cdaf16149a90476045ca7816f08ebbaa5: Status 404 returned error can't find the container with id b3510bc5270da62ec6332e34579e551cdaf16149a90476045ca7816f08ebbaa5
Sep 16 14:27:24.615047 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.615028    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7wzk\" (UniqueName: \"kubernetes.io/projected/722a81b9-eba9-4b21-9d04-907472aec41d-kube-api-access-d7wzk\") pod \"test-curl-7617-57fec3\" (UID: \"722a81b9-eba9-4b21-9d04-907472aec41d\") " pod="opendatahub/test-curl-7617-57fec3"
Sep 16 14:27:24.623734 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.623712    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7wzk\" (UniqueName: \"kubernetes.io/projected/722a81b9-eba9-4b21-9d04-907472aec41d-kube-api-access-d7wzk\") pod \"test-curl-7617-57fec3\" (UID: \"722a81b9-eba9-4b21-9d04-907472aec41d\") " pod="opendatahub/test-curl-7617-57fec3"
Sep 16 14:27:24.658666 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.658644    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf"]
Sep 16 14:27:24.659580 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:27:24.659554    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7b3132a_66c9_4a5d_b47b_e3cff8a2b644.slice/crio-aad5ad75d9c338c38cc5f7379cf8513094e8bea2ed89227e2402cb51d90f8eec WatchSource:0}: Error finding container aad5ad75d9c338c38cc5f7379cf8513094e8bea2ed89227e2402cb51d90f8eec: Status 404 returned error can't find the container with id aad5ad75d9c338c38cc5f7379cf8513094e8bea2ed89227e2402cb51d90f8eec
Sep 16 14:27:24.660209 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.660192    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7617-57fec3"
Sep 16 14:27:24.947139 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:24.947105    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7617-57fec3"]
Sep 16 14:27:24.950801 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:27:24.950759    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod722a81b9_eba9_4b21_9d04_907472aec41d.slice/crio-ad2eb12892746fd21af2cf4d904fd0edfadef6681b2a6ada1dfa1068125a3187 WatchSource:0}: Error finding container ad2eb12892746fd21af2cf4d904fd0edfadef6681b2a6ada1dfa1068125a3187: Status 404 returned error can't find the container with id ad2eb12892746fd21af2cf4d904fd0edfadef6681b2a6ada1dfa1068125a3187
Sep 16 14:27:25.119211 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.119179    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6"]
Sep 16 14:27:25.122818 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.122782    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6"
Sep 16 14:27:25.125667 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.125644    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-5d484d\""
Sep 16 14:27:25.125997 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.125974    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-5d484d-dockercfg-ng8q9\""
Sep 16 14:27:25.139071 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.139039    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6"]
Sep 16 14:27:25.170891 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.170861    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz"]
Sep 16 14:27:25.174217 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.174193    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz"
Sep 16 14:27:25.188162 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.188142    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz"]
Sep 16 14:27:25.321598 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.321512    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swgrv\" (UniqueName: \"kubernetes.io/projected/28382e17-12f4-4f98-bb9b-53c579f10dfb-kube-api-access-swgrv\") pod \"payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6\" (UID: \"28382e17-12f4-4f98-bb9b-53c579f10dfb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6"
Sep 16 14:27:25.321598 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.321555    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz\" (UID: \"1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz"
Sep 16 14:27:25.321774 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.321693    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs759\" (UniqueName: \"kubernetes.io/projected/1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c-kube-api-access-hs759\") pod \"payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz\" (UID: \"1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz"
Sep 16 14:27:25.321774 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.321764    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28382e17-12f4-4f98-bb9b-53c579f10dfb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6\" (UID: \"28382e17-12f4-4f98-bb9b-53c579f10dfb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6"
Sep 16 14:27:25.348088 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.348052    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7617-57fec3" event={"ID":"722a81b9-eba9-4b21-9d04-907472aec41d","Type":"ContainerStarted","Data":"8b7bea789a0efae0983953bf13ccd83342076056ba8ebdc3f6888da636715eba"}
Sep 16 14:27:25.348088 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.348093    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7617-57fec3" event={"ID":"722a81b9-eba9-4b21-9d04-907472aec41d","Type":"ContainerStarted","Data":"ad2eb12892746fd21af2cf4d904fd0edfadef6681b2a6ada1dfa1068125a3187"}
Sep 16 14:27:25.349700 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.349673    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf" event={"ID":"a7b3132a-66c9-4a5d-b47b-e3cff8a2b644","Type":"ContainerStarted","Data":"4c143226815d150a10f4b78ea17114099ea906a51a325ca7113a69d7f0722f16"}
Sep 16 14:27:25.349813 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.349731    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf"
Sep 16 14:27:25.349813 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.349758    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf" event={"ID":"a7b3132a-66c9-4a5d-b47b-e3cff8a2b644","Type":"ContainerStarted","Data":"aad5ad75d9c338c38cc5f7379cf8513094e8bea2ed89227e2402cb51d90f8eec"}
Sep 16 14:27:25.351422 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.351380    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55" event={"ID":"608bb3a3-74d2-4557-8f6a-5961db6fad36","Type":"ContainerStarted","Data":"89c05196e08c2ae58aaef3d4378c2b9e696bbec201bdb229994c3c09dc653176"}
Sep 16 14:27:25.351582 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.351443    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55" event={"ID":"608bb3a3-74d2-4557-8f6a-5961db6fad36","Type":"ContainerStarted","Data":"b3510bc5270da62ec6332e34579e551cdaf16149a90476045ca7816f08ebbaa5"}
Sep 16 14:27:25.351582 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.351441    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7617-f48ce1" podUID="79671c57-57dc-4913-bc5a-acf06a4a5758" containerName="test-curl-7617-f48ce1" containerID="cri-o://195381616c651d0457f9c399beca7c26cea996f44d084d9f5829882aed523486" gracePeriod=2
Sep 16 14:27:25.365345 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.365289    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7617-57fec3" podStartSLOduration=1.365270616 podStartE2EDuration="1.365270616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:27:25.362662064 +0000 UTC m=+793.966287099" watchObservedRunningTime="2026-09-16 14:27:25.365270616 +0000 UTC m=+793.968895649"
Sep 16 14:27:25.381118 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:27:25.381082    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7617-f48ce1\" is forbidden: User \"system:node:ip-10-0-132-217.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-217.ec2.internal' and this object" podUID="79671c57-57dc-4913-bc5a-acf06a4a5758" pod="opendatahub/test-curl-7617-f48ce1"
Sep 16 14:27:25.381312 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.381277    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55" podStartSLOduration=2.381263812 podStartE2EDuration="2.381263812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:27:25.379044622 +0000 UTC m=+793.982669654" watchObservedRunningTime="2026-09-16 14:27:25.381263812 +0000 UTC m=+793.984888843"
Sep 16 14:27:25.401405 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.401352    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf" podStartSLOduration=1.401338742 podStartE2EDuration="1.401338742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:27:25.400261532 +0000 UTC m=+794.003886564" watchObservedRunningTime="2026-09-16 14:27:25.401338742 +0000 UTC m=+794.004963773"
Sep 16 14:27:25.422235 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.422199    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz\" (UID: \"1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz"
Sep 16 14:27:25.422358 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.422263    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs759\" (UniqueName: \"kubernetes.io/projected/1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c-kube-api-access-hs759\") pod \"payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz\" (UID: \"1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz"
Sep 16 14:27:25.422358 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.422322    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28382e17-12f4-4f98-bb9b-53c579f10dfb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6\" (UID: \"28382e17-12f4-4f98-bb9b-53c579f10dfb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6"
Sep 16 14:27:25.422527 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.422422    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swgrv\" (UniqueName: \"kubernetes.io/projected/28382e17-12f4-4f98-bb9b-53c579f10dfb-kube-api-access-swgrv\") pod \"payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6\" (UID: \"28382e17-12f4-4f98-bb9b-53c579f10dfb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6"
Sep 16 14:27:25.423122 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.423092    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz\" (UID: \"1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz"
Sep 16 14:27:25.423229 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.423161    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28382e17-12f4-4f98-bb9b-53c579f10dfb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6\" (UID: \"28382e17-12f4-4f98-bb9b-53c579f10dfb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6"
Sep 16 14:27:25.431912 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.431886    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swgrv\" (UniqueName: \"kubernetes.io/projected/28382e17-12f4-4f98-bb9b-53c579f10dfb-kube-api-access-swgrv\") pod \"payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6\" (UID: \"28382e17-12f4-4f98-bb9b-53c579f10dfb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6"
Sep 16 14:27:25.432243 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.432222    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs759\" (UniqueName: \"kubernetes.io/projected/1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c-kube-api-access-hs759\") pod \"payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz\" (UID: \"1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz"
Sep 16 14:27:25.434976 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.434955    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6"
Sep 16 14:27:25.484365 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.484231    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz"
Sep 16 14:27:25.717899 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.717875    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6"]
Sep 16 14:27:25.720292 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:27:25.720267    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28382e17_12f4_4f98_bb9b_53c579f10dfb.slice/crio-88f8dfff57f994bcecb95487dfcd4cd9854a74bffcffba650741ed24b10114a1 WatchSource:0}: Error finding container 88f8dfff57f994bcecb95487dfcd4cd9854a74bffcffba650741ed24b10114a1: Status 404 returned error can't find the container with id 88f8dfff57f994bcecb95487dfcd4cd9854a74bffcffba650741ed24b10114a1
Sep 16 14:27:25.779232 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:25.779207    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz"]
Sep 16 14:27:25.782167 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:27:25.782139    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad5aeae_ed4e_45be_8e9b_0cae2f37e77c.slice/crio-21d393ecbf48406c1062b178058b12c6f9a247aa4c22a406574123a6e5f64354 WatchSource:0}: Error finding container 21d393ecbf48406c1062b178058b12c6f9a247aa4c22a406574123a6e5f64354: Status 404 returned error can't find the container with id 21d393ecbf48406c1062b178058b12c6f9a247aa4c22a406574123a6e5f64354
Sep 16 14:27:26.020427 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:26.020335    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7617-57fec3"]
Sep 16 14:27:26.024146 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:26.024117    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7617-57fec3"]
Sep 16 14:27:26.246197 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:26.246167    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf"]
Sep 16 14:27:26.246572 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:26.246557    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="722a81b9-eba9-4b21-9d04-907472aec41d" containerName="test-curl-7617-57fec3"
Sep 16 14:27:26.246572 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:26.246573    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="722a81b9-eba9-4b21-9d04-907472aec41d" containerName="test-curl-7617-57fec3"
Sep 16 14:27:26.246729 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:26.246642    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="722a81b9-eba9-4b21-9d04-907472aec41d" containerName="test-curl-7617-57fec3"
Sep 16 14:27:26.249636 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:26.249616    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf"
Sep 16 14:27:26.252686 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:26.252667    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-924916-dockercfg-68zh5\""
Sep 16 14:27:26.252808 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:26.252767    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-924916\""
Sep 16 14:27:26.262551 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:26.262526    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf"]
Sep 16 14:27:26.332834 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:26.332751    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a004d1e5-6df4-4e40-996d-849c2e3e043f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf\" (UID: \"a004d1e5-6df4-4e40-996d-849c2e3e043f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf"
Sep 16 14:27:26.332834 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:26.332815    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xmqn\" (UniqueName: \"kubernetes.io/projected/a004d1e5-6df4-4e40-996d-849c2e3e043f-kube-api-access-5xmqn\") pod \"payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf\" (UID: \"a004d1e5-6df4-4e40-996d-849c2e3e043f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf"
Sep 16 14:27:26.358203 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:26.358153    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6" event={"ID":"28382e17-12f4-4f98-bb9b-53c579f10dfb","Type":"ContainerStarted","Data":"37171d84b8b7f641cbeec9cfabf5e08862d6726bba64258d99cdaa132d3198f8"}
Sep 16 14:27:26.358668 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:26.358267    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6" event={"ID":"28382e17-12f4-4f98-bb9b-53c579f10dfb","Type":"ContainerStarted","Data":"88f8dfff57f994bcecb95487dfcd4cd9854a74bffcffba650741ed24b10114a1"}
Sep 16 14:27:26.358668 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:26.358358    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6"
Sep 16 14:27:26.360167 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:26.360091    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz" event={"ID":"1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c","Type":"ContainerStarted","Data":"05e6df28836a7473dc818bd449ee4a2925529fd0bb733f23ab6d3cd0dd3cf35d"}
Sep 16 14:27:26.360167 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:26.360139    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz" event={"ID":"1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c","Type":"ContainerStarted","Data":"21d393ecbf48406c1062b178058b12c6f9a247aa4c22a406574123a6e5f64354"}
Sep 16 14:27:26.360347 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:27:26.360211    2587 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7617-57fec3" container="test-curl-7617-57fec3"
Sep 16 14:27:26.360561 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:26.360541    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55"
Sep 16 14:27:26.378287 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:26.378239    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6" podStartSLOduration=1.378226789 podStartE2EDuration="1.378226789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:27:26.376465698 +0000 UTC m=+794.980090731" watchObservedRunningTime="2026-09-16 14:27:26.378226789 +0000 UTC m=+794.981851849"
Sep 16 14:27:26.397293 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:26.397250    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz" podStartSLOduration=1.397237884 podStartE2EDuration="1.397237884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:27:26.395585531 +0000 UTC m=+794.999210563" watchObservedRunningTime="2026-09-16 14:27:26.397237884 +0000 UTC m=+795.000862915"
Sep 16 14:27:26.433338 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:26.433300    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a004d1e5-6df4-4e40-996d-849c2e3e043f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf\" (UID: \"a004d1e5-6df4-4e40-996d-849c2e3e043f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf"
Sep 16 14:27:26.433501 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:26.433429    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xmqn\" (UniqueName: \"kubernetes.io/projected/a004d1e5-6df4-4e40-996d-849c2e3e043f-kube-api-access-5xmqn\") pod \"payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf\" (UID: \"a004d1e5-6df4-4e40-996d-849c2e3e043f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf"
Sep 16 14:27:26.434325 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:26.434298    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a004d1e5-6df4-4e40-996d-849c2e3e043f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf\" (UID: \"a004d1e5-6df4-4e40-996d-849c2e3e043f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf"
Sep 16 14:27:26.443140 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:26.443113    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xmqn\" (UniqueName: \"kubernetes.io/projected/a004d1e5-6df4-4e40-996d-849c2e3e043f-kube-api-access-5xmqn\") pod \"payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf\" (UID: \"a004d1e5-6df4-4e40-996d-849c2e3e043f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf"
Sep 16 14:27:26.560804 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:26.560773    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf"
Sep 16 14:27:26.837497 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:26.837211    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf"]
Sep 16 14:27:26.840721 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:27:26.840679    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda004d1e5_6df4_4e40_996d_849c2e3e043f.slice/crio-225beb2b3c35c20ec2aef9e565a4a264125ba9ae7aa9211f7ce8ea58f7d55d9d WatchSource:0}: Error finding container 225beb2b3c35c20ec2aef9e565a4a264125ba9ae7aa9211f7ce8ea58f7d55d9d: Status 404 returned error can't find the container with id 225beb2b3c35c20ec2aef9e565a4a264125ba9ae7aa9211f7ce8ea58f7d55d9d
Sep 16 14:27:27.368547 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:27.368509    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf" event={"ID":"a004d1e5-6df4-4e40-996d-849c2e3e043f","Type":"ContainerStarted","Data":"4ca63b58ddcfb266bc3e593e0b82e01df937bb9d7bee35830d674b2aa2a36268"}
Sep 16 14:27:27.368917 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:27.368578    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf" event={"ID":"a004d1e5-6df4-4e40-996d-849c2e3e043f","Type":"ContainerStarted","Data":"225beb2b3c35c20ec2aef9e565a4a264125ba9ae7aa9211f7ce8ea58f7d55d9d"}
Sep 16 14:27:27.368917 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:27.368568    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7617-57fec3" podUID="722a81b9-eba9-4b21-9d04-907472aec41d" containerName="test-curl-7617-57fec3" containerID="cri-o://8b7bea789a0efae0983953bf13ccd83342076056ba8ebdc3f6888da636715eba" gracePeriod=2
Sep 16 14:27:27.369233 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:27.369202    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz"
Sep 16 14:27:27.374895 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:27:27.374865    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7617-57fec3\" is forbidden: User \"system:node:ip-10-0-132-217.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-217.ec2.internal' and this object" podUID="722a81b9-eba9-4b21-9d04-907472aec41d" pod="opendatahub/test-curl-7617-57fec3"
Sep 16 14:27:27.404470 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:27.404418    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf" podStartSLOduration=1.404402046 podStartE2EDuration="1.404402046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:27:27.402262787 +0000 UTC m=+796.005887821" watchObservedRunningTime="2026-09-16 14:27:27.404402046 +0000 UTC m=+796.008027071"
Sep 16 14:27:27.404610 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:27:27.404581    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7617-57fec3\" is forbidden: User \"system:node:ip-10-0-132-217.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-217.ec2.internal' and this object" podUID="722a81b9-eba9-4b21-9d04-907472aec41d" pod="opendatahub/test-curl-7617-57fec3"
Sep 16 14:27:27.543466 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:27.543434    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7617-f48ce1"
Sep 16 14:27:27.546914 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:27:27.546885    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7617-57fec3\" is forbidden: User \"system:node:ip-10-0-132-217.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-217.ec2.internal' and this object" podUID="722a81b9-eba9-4b21-9d04-907472aec41d" pod="opendatahub/test-curl-7617-57fec3"
Sep 16 14:27:27.644859 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:27.644774    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpcvk\" (UniqueName: \"kubernetes.io/projected/79671c57-57dc-4913-bc5a-acf06a4a5758-kube-api-access-rpcvk\") pod \"79671c57-57dc-4913-bc5a-acf06a4a5758\" (UID: \"79671c57-57dc-4913-bc5a-acf06a4a5758\") "
Sep 16 14:27:27.647294 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:27.647263    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79671c57-57dc-4913-bc5a-acf06a4a5758-kube-api-access-rpcvk" (OuterVolumeSpecName: "kube-api-access-rpcvk") pod "79671c57-57dc-4913-bc5a-acf06a4a5758" (UID: "79671c57-57dc-4913-bc5a-acf06a4a5758"). InnerVolumeSpecName "kube-api-access-rpcvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:27:27.745949 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:27.745913    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpcvk\" (UniqueName: \"kubernetes.io/projected/79671c57-57dc-4913-bc5a-acf06a4a5758-kube-api-access-rpcvk\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:27:27.905134 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:27.905093    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79671c57-57dc-4913-bc5a-acf06a4a5758" path="/var/lib/kubelet/pods/79671c57-57dc-4913-bc5a-acf06a4a5758/volumes"
Sep 16 14:27:28.373943 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:28.373861    2587 generic.go:356] "Generic (PLEG): container finished" podID="79671c57-57dc-4913-bc5a-acf06a4a5758" containerID="195381616c651d0457f9c399beca7c26cea996f44d084d9f5829882aed523486" exitCode=137
Sep 16 14:27:28.373943 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:28.373904    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7617-f48ce1"
Sep 16 14:27:28.374508 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:28.373978    2587 scope.go:117] "RemoveContainer" containerID="195381616c651d0457f9c399beca7c26cea996f44d084d9f5829882aed523486"
Sep 16 14:27:28.374508 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:28.374150    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf"
Sep 16 14:27:28.382975 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:28.382954    2587 scope.go:117] "RemoveContainer" containerID="195381616c651d0457f9c399beca7c26cea996f44d084d9f5829882aed523486"
Sep 16 14:27:28.383285 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:27:28.383261    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"195381616c651d0457f9c399beca7c26cea996f44d084d9f5829882aed523486\": container with ID starting with 195381616c651d0457f9c399beca7c26cea996f44d084d9f5829882aed523486 not found: ID does not exist" containerID="195381616c651d0457f9c399beca7c26cea996f44d084d9f5829882aed523486"
Sep 16 14:27:28.383365 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:28.383298    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195381616c651d0457f9c399beca7c26cea996f44d084d9f5829882aed523486"} err="failed to get container status \"195381616c651d0457f9c399beca7c26cea996f44d084d9f5829882aed523486\": rpc error: code = NotFound desc = could not find container \"195381616c651d0457f9c399beca7c26cea996f44d084d9f5829882aed523486\": container with ID starting with 195381616c651d0457f9c399beca7c26cea996f44d084d9f5829882aed523486 not found: ID does not exist"
Sep 16 14:27:29.560857 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:29.560835    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7617-57fec3"
Sep 16 14:27:29.664454 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:29.664418    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7wzk\" (UniqueName: \"kubernetes.io/projected/722a81b9-eba9-4b21-9d04-907472aec41d-kube-api-access-d7wzk\") pod \"722a81b9-eba9-4b21-9d04-907472aec41d\" (UID: \"722a81b9-eba9-4b21-9d04-907472aec41d\") "
Sep 16 14:27:29.666996 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:29.666964    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722a81b9-eba9-4b21-9d04-907472aec41d-kube-api-access-d7wzk" (OuterVolumeSpecName: "kube-api-access-d7wzk") pod "722a81b9-eba9-4b21-9d04-907472aec41d" (UID: "722a81b9-eba9-4b21-9d04-907472aec41d"). InnerVolumeSpecName "kube-api-access-d7wzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:27:29.765979 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:29.765940    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7wzk\" (UniqueName: \"kubernetes.io/projected/722a81b9-eba9-4b21-9d04-907472aec41d-kube-api-access-d7wzk\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:27:29.903053 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:29.903018    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="722a81b9-eba9-4b21-9d04-907472aec41d" path="/var/lib/kubelet/pods/722a81b9-eba9-4b21-9d04-907472aec41d/volumes"
Sep 16 14:27:30.387922 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:30.387885    2587 generic.go:356] "Generic (PLEG): container finished" podID="722a81b9-eba9-4b21-9d04-907472aec41d" containerID="8b7bea789a0efae0983953bf13ccd83342076056ba8ebdc3f6888da636715eba" exitCode=137
Sep 16 14:27:30.387922 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:30.387924    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7617-57fec3"
Sep 16 14:27:30.388155 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:30.387975    2587 scope.go:117] "RemoveContainer" containerID="8b7bea789a0efae0983953bf13ccd83342076056ba8ebdc3f6888da636715eba"
Sep 16 14:27:30.396564 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:30.396548    2587 scope.go:117] "RemoveContainer" containerID="8b7bea789a0efae0983953bf13ccd83342076056ba8ebdc3f6888da636715eba"
Sep 16 14:27:30.396798 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:27:30.396780    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b7bea789a0efae0983953bf13ccd83342076056ba8ebdc3f6888da636715eba\": container with ID starting with 8b7bea789a0efae0983953bf13ccd83342076056ba8ebdc3f6888da636715eba not found: ID does not exist" containerID="8b7bea789a0efae0983953bf13ccd83342076056ba8ebdc3f6888da636715eba"
Sep 16 14:27:30.396840 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:30.396811    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b7bea789a0efae0983953bf13ccd83342076056ba8ebdc3f6888da636715eba"} err="failed to get container status \"8b7bea789a0efae0983953bf13ccd83342076056ba8ebdc3f6888da636715eba\": rpc error: code = NotFound desc = could not find container \"8b7bea789a0efae0983953bf13ccd83342076056ba8ebdc3f6888da636715eba\": container with ID starting with 8b7bea789a0efae0983953bf13ccd83342076056ba8ebdc3f6888da636715eba not found: ID does not exist"
Sep 16 14:27:33.403081 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:33.403005    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-84fdff-bf594d749-qtxbk"]
Sep 16 14:27:33.407538 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:33.407522    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-84fdff-bf594d749-qtxbk"
Sep 16 14:27:33.410040 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:33.410013    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-84fdff\""
Sep 16 14:27:33.410526 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:33.410506    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-84fdff-dockercfg-n6bwj\""
Sep 16 14:27:33.416606 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:33.416584    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-84fdff-bf594d749-qtxbk"]
Sep 16 14:27:33.444466 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:33.444433    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9"]
Sep 16 14:27:33.447907 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:33.447887    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9"
Sep 16 14:27:33.458088 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:33.458066    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9"]
Sep 16 14:27:33.498502 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:33.498476    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d432ee6-4244-44ee-9f22-226f1837666d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9\" (UID: \"8d432ee6-4244-44ee-9f22-226f1837666d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9"
Sep 16 14:27:33.498618 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:33.498534    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmdgf\" (UniqueName: \"kubernetes.io/projected/2077db6c-4fbc-45eb-98e2-82a4837e9e7a-kube-api-access-jmdgf\") pod \"payload-processing-e2e-autores-84fdff-bf594d749-qtxbk\" (UID: \"2077db6c-4fbc-45eb-98e2-82a4837e9e7a\") " pod="openshift-ingress/payload-processing-e2e-autores-84fdff-bf594d749-qtxbk"
Sep 16 14:27:33.498618 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:33.498601    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2077db6c-4fbc-45eb-98e2-82a4837e9e7a-plugins-config-volume\") pod \"payload-processing-e2e-autores-84fdff-bf594d749-qtxbk\" (UID: \"2077db6c-4fbc-45eb-98e2-82a4837e9e7a\") " pod="openshift-ingress/payload-processing-e2e-autores-84fdff-bf594d749-qtxbk"
Sep 16 14:27:33.498692 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:33.498651    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfdzm\" (UniqueName: \"kubernetes.io/projected/8d432ee6-4244-44ee-9f22-226f1837666d-kube-api-access-jfdzm\") pod \"payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9\" (UID: \"8d432ee6-4244-44ee-9f22-226f1837666d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9"
Sep 16 14:27:33.599684 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:33.599651    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmdgf\" (UniqueName: \"kubernetes.io/projected/2077db6c-4fbc-45eb-98e2-82a4837e9e7a-kube-api-access-jmdgf\") pod \"payload-processing-e2e-autores-84fdff-bf594d749-qtxbk\" (UID: \"2077db6c-4fbc-45eb-98e2-82a4837e9e7a\") " pod="openshift-ingress/payload-processing-e2e-autores-84fdff-bf594d749-qtxbk"
Sep 16 14:27:33.599847 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:33.599692    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2077db6c-4fbc-45eb-98e2-82a4837e9e7a-plugins-config-volume\") pod \"payload-processing-e2e-autores-84fdff-bf594d749-qtxbk\" (UID: \"2077db6c-4fbc-45eb-98e2-82a4837e9e7a\") " pod="openshift-ingress/payload-processing-e2e-autores-84fdff-bf594d749-qtxbk"
Sep 16 14:27:33.599847 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:33.599722    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfdzm\" (UniqueName: \"kubernetes.io/projected/8d432ee6-4244-44ee-9f22-226f1837666d-kube-api-access-jfdzm\") pod \"payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9\" (UID: \"8d432ee6-4244-44ee-9f22-226f1837666d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9"
Sep 16 14:27:33.599847 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:33.599748    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d432ee6-4244-44ee-9f22-226f1837666d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9\" (UID: \"8d432ee6-4244-44ee-9f22-226f1837666d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9"
Sep 16 14:27:33.600368 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:33.600348    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d432ee6-4244-44ee-9f22-226f1837666d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9\" (UID: \"8d432ee6-4244-44ee-9f22-226f1837666d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9"
Sep 16 14:27:33.600481 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:33.600460    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2077db6c-4fbc-45eb-98e2-82a4837e9e7a-plugins-config-volume\") pod \"payload-processing-e2e-autores-84fdff-bf594d749-qtxbk\" (UID: \"2077db6c-4fbc-45eb-98e2-82a4837e9e7a\") " pod="openshift-ingress/payload-processing-e2e-autores-84fdff-bf594d749-qtxbk"
Sep 16 14:27:33.608607 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:33.608585    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfdzm\" (UniqueName: \"kubernetes.io/projected/8d432ee6-4244-44ee-9f22-226f1837666d-kube-api-access-jfdzm\") pod \"payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9\" (UID: \"8d432ee6-4244-44ee-9f22-226f1837666d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9"
Sep 16 14:27:33.608861 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:33.608843    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmdgf\" (UniqueName: \"kubernetes.io/projected/2077db6c-4fbc-45eb-98e2-82a4837e9e7a-kube-api-access-jmdgf\") pod \"payload-processing-e2e-autores-84fdff-bf594d749-qtxbk\" (UID: \"2077db6c-4fbc-45eb-98e2-82a4837e9e7a\") " pod="openshift-ingress/payload-processing-e2e-autores-84fdff-bf594d749-qtxbk"
Sep 16 14:27:33.718556 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:33.718472    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-84fdff-bf594d749-qtxbk"
Sep 16 14:27:33.758341 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:33.758310    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9"
Sep 16 14:27:34.001926 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.001891    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-84fdff-bf594d749-qtxbk"]
Sep 16 14:27:34.004010 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:27:34.003981    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2077db6c_4fbc_45eb_98e2_82a4837e9e7a.slice/crio-a1550fa49632f7de9f8c44581cd57303b1f96c33953cc01e9fb8707d1c0c87ec WatchSource:0}: Error finding container a1550fa49632f7de9f8c44581cd57303b1f96c33953cc01e9fb8707d1c0c87ec: Status 404 returned error can't find the container with id a1550fa49632f7de9f8c44581cd57303b1f96c33953cc01e9fb8707d1c0c87ec
Sep 16 14:27:34.052190 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.052168    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9"]
Sep 16 14:27:34.056479 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:27:34.056453    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d432ee6_4244_44ee_9f22_226f1837666d.slice/crio-e1e6375ab28cc2842f2bd517aafb31b4017998a7c6889e19c62423256d5ec6c9 WatchSource:0}: Error finding container e1e6375ab28cc2842f2bd517aafb31b4017998a7c6889e19c62423256d5ec6c9: Status 404 returned error can't find the container with id e1e6375ab28cc2842f2bd517aafb31b4017998a7c6889e19c62423256d5ec6c9
Sep 16 14:27:34.410260 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.410222    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-84fdff-bf594d749-qtxbk" event={"ID":"2077db6c-4fbc-45eb-98e2-82a4837e9e7a","Type":"ContainerStarted","Data":"dc46fcb121e4ded0b7e1c18a36192b58c049cc3364dc462e7a5a6d40266eb262"}
Sep 16 14:27:34.410777 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.410286    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-84fdff-bf594d749-qtxbk" event={"ID":"2077db6c-4fbc-45eb-98e2-82a4837e9e7a","Type":"ContainerStarted","Data":"a1550fa49632f7de9f8c44581cd57303b1f96c33953cc01e9fb8707d1c0c87ec"}
Sep 16 14:27:34.410777 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.410326    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-84fdff-bf594d749-qtxbk"
Sep 16 14:27:34.412104 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.412076    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9" event={"ID":"8d432ee6-4244-44ee-9f22-226f1837666d","Type":"ContainerStarted","Data":"3ac13f142fb1ee63ec03b49dd3772016540523a178f9330e43495dbce4e41803"}
Sep 16 14:27:34.412213 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.412144    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9" event={"ID":"8d432ee6-4244-44ee-9f22-226f1837666d","Type":"ContainerStarted","Data":"e1e6375ab28cc2842f2bd517aafb31b4017998a7c6889e19c62423256d5ec6c9"}
Sep 16 14:27:34.412213 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.412173    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9"
Sep 16 14:27:34.445662 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.445615    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-84fdff-bf594d749-qtxbk" podStartSLOduration=1.4455997950000001 podStartE2EDuration="1.445599795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:27:34.444092443 +0000 UTC m=+803.047717472" watchObservedRunningTime="2026-09-16 14:27:34.445599795 +0000 UTC m=+803.049224827"
Sep 16 14:27:34.506175 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.506116    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9" podStartSLOduration=1.506096566 podStartE2EDuration="1.506096566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:27:34.504472427 +0000 UTC m=+803.108097459" watchObservedRunningTime="2026-09-16 14:27:34.506096566 +0000 UTC m=+803.109721599"
Sep 16 14:27:34.540931 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.540890    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq"]
Sep 16 14:27:34.550366 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.550333    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq"
Sep 16 14:27:34.553906 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.553878    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1e4514fd\""
Sep 16 14:27:34.554036 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.553941    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1e4514fd-dockercfg-f98pw\""
Sep 16 14:27:34.577634 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.577608    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq"]
Sep 16 14:27:34.611995 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.611968    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e717ad-74d4-446b-a717-f346a293798d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq\" (UID: \"73e717ad-74d4-446b-a717-f346a293798d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq"
Sep 16 14:27:34.612119 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.612013    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwf7r\" (UniqueName: \"kubernetes.io/projected/73e717ad-74d4-446b-a717-f346a293798d-kube-api-access-kwf7r\") pod \"payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq\" (UID: \"73e717ad-74d4-446b-a717-f346a293798d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq"
Sep 16 14:27:34.634161 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.634134    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5"]
Sep 16 14:27:34.638176 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.638159    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5"
Sep 16 14:27:34.652365 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.652343    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5"]
Sep 16 14:27:34.713420 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.713333    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwf7r\" (UniqueName: \"kubernetes.io/projected/73e717ad-74d4-446b-a717-f346a293798d-kube-api-access-kwf7r\") pod \"payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq\" (UID: \"73e717ad-74d4-446b-a717-f346a293798d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq"
Sep 16 14:27:34.713420 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.713384    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed70c51d-d881-4822-84f7-e909809e04dd-plugins-config-volume\") pod \"payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5\" (UID: \"ed70c51d-d881-4822-84f7-e909809e04dd\") " pod="openshift-ingress/payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5"
Sep 16 14:27:34.713578 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.713465    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v2xr\" (UniqueName: \"kubernetes.io/projected/ed70c51d-d881-4822-84f7-e909809e04dd-kube-api-access-9v2xr\") pod \"payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5\" (UID: \"ed70c51d-d881-4822-84f7-e909809e04dd\") " pod="openshift-ingress/payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5"
Sep 16 14:27:34.713578 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.713490    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e717ad-74d4-446b-a717-f346a293798d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq\" (UID: \"73e717ad-74d4-446b-a717-f346a293798d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq"
Sep 16 14:27:34.714070 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.714054    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e717ad-74d4-446b-a717-f346a293798d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq\" (UID: \"73e717ad-74d4-446b-a717-f346a293798d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq"
Sep 16 14:27:34.724751 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.724728    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwf7r\" (UniqueName: \"kubernetes.io/projected/73e717ad-74d4-446b-a717-f346a293798d-kube-api-access-kwf7r\") pod \"payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq\" (UID: \"73e717ad-74d4-446b-a717-f346a293798d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq"
Sep 16 14:27:34.814688 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.814648    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v2xr\" (UniqueName: \"kubernetes.io/projected/ed70c51d-d881-4822-84f7-e909809e04dd-kube-api-access-9v2xr\") pod \"payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5\" (UID: \"ed70c51d-d881-4822-84f7-e909809e04dd\") " pod="openshift-ingress/payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5"
Sep 16 14:27:34.814847 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.814746    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed70c51d-d881-4822-84f7-e909809e04dd-plugins-config-volume\") pod \"payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5\" (UID: \"ed70c51d-d881-4822-84f7-e909809e04dd\") " pod="openshift-ingress/payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5"
Sep 16 14:27:34.815566 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.815547    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed70c51d-d881-4822-84f7-e909809e04dd-plugins-config-volume\") pod \"payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5\" (UID: \"ed70c51d-d881-4822-84f7-e909809e04dd\") " pod="openshift-ingress/payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5"
Sep 16 14:27:34.823619 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.823581    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v2xr\" (UniqueName: \"kubernetes.io/projected/ed70c51d-d881-4822-84f7-e909809e04dd-kube-api-access-9v2xr\") pod \"payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5\" (UID: \"ed70c51d-d881-4822-84f7-e909809e04dd\") " pod="openshift-ingress/payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5"
Sep 16 14:27:34.861067 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.861031    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq"
Sep 16 14:27:34.948773 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:34.948739    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5"
Sep 16 14:27:35.136572 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:35.136526    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq"]
Sep 16 14:27:35.139573 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:27:35.139538    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e717ad_74d4_446b_a717_f346a293798d.slice/crio-f4da8c33c1a6955261f61f9be48837e6c9e61e69be6d1655daf99bffcaead078 WatchSource:0}: Error finding container f4da8c33c1a6955261f61f9be48837e6c9e61e69be6d1655daf99bffcaead078: Status 404 returned error can't find the container with id f4da8c33c1a6955261f61f9be48837e6c9e61e69be6d1655daf99bffcaead078
Sep 16 14:27:35.235727 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:35.235693    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5"]
Sep 16 14:27:35.237829 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:27:35.237805    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded70c51d_d881_4822_84f7_e909809e04dd.slice/crio-7d174c7cb5fb6636775b843ab26422f23a5c449d3759f4a77253a48f43df2f0c WatchSource:0}: Error finding container 7d174c7cb5fb6636775b843ab26422f23a5c449d3759f4a77253a48f43df2f0c: Status 404 returned error can't find the container with id 7d174c7cb5fb6636775b843ab26422f23a5c449d3759f4a77253a48f43df2f0c
Sep 16 14:27:35.420051 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:35.419544    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5" event={"ID":"ed70c51d-d881-4822-84f7-e909809e04dd","Type":"ContainerStarted","Data":"a2f707934687be40620d6db57d2438a2b624c34ef6a8b7cba45683927d109522"}
Sep 16 14:27:35.420051 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:35.419601    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5" event={"ID":"ed70c51d-d881-4822-84f7-e909809e04dd","Type":"ContainerStarted","Data":"7d174c7cb5fb6636775b843ab26422f23a5c449d3759f4a77253a48f43df2f0c"}
Sep 16 14:27:35.420051 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:35.419723    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5"
Sep 16 14:27:35.421532 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:35.421507    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq" event={"ID":"73e717ad-74d4-446b-a717-f346a293798d","Type":"ContainerStarted","Data":"ed9bb24b6ecfab3b4b1617c08ea4560df57e5434b5fe210c02ffa0d1258d0f7f"}
Sep 16 14:27:35.421669 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:35.421554    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq" event={"ID":"73e717ad-74d4-446b-a717-f346a293798d","Type":"ContainerStarted","Data":"f4da8c33c1a6955261f61f9be48837e6c9e61e69be6d1655daf99bffcaead078"}
Sep 16 14:27:35.421833 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:35.421818    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq"
Sep 16 14:27:35.440279 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:35.440226    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5" podStartSLOduration=1.440209582 podStartE2EDuration="1.440209582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:27:35.437058728 +0000 UTC m=+804.040683764" watchObservedRunningTime="2026-09-16 14:27:35.440209582 +0000 UTC m=+804.043834618"
Sep 16 14:27:35.457013 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:35.456965    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq" podStartSLOduration=1.456948458 podStartE2EDuration="1.456948458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:27:35.453852509 +0000 UTC m=+804.057477544" watchObservedRunningTime="2026-09-16 14:27:35.456948458 +0000 UTC m=+804.060573491"
Sep 16 14:27:36.363038 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:36.363008    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf"
Sep 16 14:27:37.371429 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:37.371379    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55"
Sep 16 14:27:37.372004 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:37.371988    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6"
Sep 16 14:27:38.376505 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:38.376472    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz"
Sep 16 14:27:39.384755 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:39.384726    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf"
Sep 16 14:27:45.423614 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:45.423580    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-84fdff-bf594d749-qtxbk"
Sep 16 14:27:45.423997 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:45.423666    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9"
Sep 16 14:27:46.428269 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:46.428234    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5"
Sep 16 14:27:46.428754 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:46.428303    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq"
Sep 16 14:27:50.040163 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:50.040129    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2"]
Sep 16 14:27:50.047620 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:50.047595    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2"
Sep 16 14:27:50.050417 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:50.050379    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d0b89a\""
Sep 16 14:27:50.050542 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:50.050445    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d0b89a-dockercfg-ctwgt\""
Sep 16 14:27:50.055854 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:50.055831    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2"]
Sep 16 14:27:50.077617 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:50.077589    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk"]
Sep 16 14:27:50.086059 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:50.085524    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk"
Sep 16 14:27:50.093964 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:50.093937    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk"]
Sep 16 14:27:50.159583 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:50.159537    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84948549-f033-4ba9-97f6-71e5284d9391-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk\" (UID: \"84948549-f033-4ba9-97f6-71e5284d9391\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk"
Sep 16 14:27:50.159766 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:50.159588    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v7gb\" (UniqueName: \"kubernetes.io/projected/3a939cd7-244b-4af2-840e-d5704891cd4e-kube-api-access-6v7gb\") pod \"payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2\" (UID: \"3a939cd7-244b-4af2-840e-d5704891cd4e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2"
Sep 16 14:27:50.159766 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:50.159681    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a939cd7-244b-4af2-840e-d5704891cd4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2\" (UID: \"3a939cd7-244b-4af2-840e-d5704891cd4e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2"
Sep 16 14:27:50.159766 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:50.159711    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp5gm\" (UniqueName: \"kubernetes.io/projected/84948549-f033-4ba9-97f6-71e5284d9391-kube-api-access-zp5gm\") pod \"payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk\" (UID: \"84948549-f033-4ba9-97f6-71e5284d9391\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk"
Sep 16 14:27:50.260838 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:50.260799    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84948549-f033-4ba9-97f6-71e5284d9391-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk\" (UID: \"84948549-f033-4ba9-97f6-71e5284d9391\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk"
Sep 16 14:27:50.261032 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:50.260848    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v7gb\" (UniqueName: \"kubernetes.io/projected/3a939cd7-244b-4af2-840e-d5704891cd4e-kube-api-access-6v7gb\") pod \"payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2\" (UID: \"3a939cd7-244b-4af2-840e-d5704891cd4e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2"
Sep 16 14:27:50.261032 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:50.260895    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a939cd7-244b-4af2-840e-d5704891cd4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2\" (UID: \"3a939cd7-244b-4af2-840e-d5704891cd4e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2"
Sep 16 14:27:50.261032 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:50.260919    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp5gm\" (UniqueName: \"kubernetes.io/projected/84948549-f033-4ba9-97f6-71e5284d9391-kube-api-access-zp5gm\") pod \"payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk\" (UID: \"84948549-f033-4ba9-97f6-71e5284d9391\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk"
Sep 16 14:27:50.261593 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:50.261571    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84948549-f033-4ba9-97f6-71e5284d9391-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk\" (UID: \"84948549-f033-4ba9-97f6-71e5284d9391\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk"
Sep 16 14:27:50.261679 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:50.261600    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a939cd7-244b-4af2-840e-d5704891cd4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2\" (UID: \"3a939cd7-244b-4af2-840e-d5704891cd4e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2"
Sep 16 14:27:50.269664 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:50.269638    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v7gb\" (UniqueName: \"kubernetes.io/projected/3a939cd7-244b-4af2-840e-d5704891cd4e-kube-api-access-6v7gb\") pod \"payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2\" (UID: \"3a939cd7-244b-4af2-840e-d5704891cd4e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2"
Sep 16 14:27:50.270031 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:50.270010    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp5gm\" (UniqueName: \"kubernetes.io/projected/84948549-f033-4ba9-97f6-71e5284d9391-kube-api-access-zp5gm\") pod \"payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk\" (UID: \"84948549-f033-4ba9-97f6-71e5284d9391\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk"
Sep 16 14:27:50.362018 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:50.361936    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2"
Sep 16 14:27:50.398442 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:50.398408    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk"
Sep 16 14:27:50.649533 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:50.649508    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2"]
Sep 16 14:27:50.651635 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:27:50.651615    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a939cd7_244b_4af2_840e_d5704891cd4e.slice/crio-2e826b7614d5b815b1aeb4a5677bdce3177ca709208bd54a06cbb18e30dea8b7 WatchSource:0}: Error finding container 2e826b7614d5b815b1aeb4a5677bdce3177ca709208bd54a06cbb18e30dea8b7: Status 404 returned error can't find the container with id 2e826b7614d5b815b1aeb4a5677bdce3177ca709208bd54a06cbb18e30dea8b7
Sep 16 14:27:50.688136 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:50.688112    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk"]
Sep 16 14:27:50.690528 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:27:50.690505    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84948549_f033_4ba9_97f6_71e5284d9391.slice/crio-d33e53b32ed6853482231aac91c2cc2530ce3ef49f5cb47ce4d6de026365d561 WatchSource:0}: Error finding container d33e53b32ed6853482231aac91c2cc2530ce3ef49f5cb47ce4d6de026365d561: Status 404 returned error can't find the container with id d33e53b32ed6853482231aac91c2cc2530ce3ef49f5cb47ce4d6de026365d561
Sep 16 14:27:51.505944 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:51.505906    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk" event={"ID":"84948549-f033-4ba9-97f6-71e5284d9391","Type":"ContainerStarted","Data":"eddf81146969c4521f8a55f494d7cd864c3aa12d734d117bf42cd5b2a939eebe"}
Sep 16 14:27:51.506379 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:51.506010    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk" event={"ID":"84948549-f033-4ba9-97f6-71e5284d9391","Type":"ContainerStarted","Data":"d33e53b32ed6853482231aac91c2cc2530ce3ef49f5cb47ce4d6de026365d561"}
Sep 16 14:27:51.506379 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:51.506043    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk"
Sep 16 14:27:51.507577 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:51.507546    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2" event={"ID":"3a939cd7-244b-4af2-840e-d5704891cd4e","Type":"ContainerStarted","Data":"e9655d2049106e69e33667eeba7afc8850053e6240586cf312fd62b6fa16ccf5"}
Sep 16 14:27:51.507678 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:51.507599    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2" event={"ID":"3a939cd7-244b-4af2-840e-d5704891cd4e","Type":"ContainerStarted","Data":"2e826b7614d5b815b1aeb4a5677bdce3177ca709208bd54a06cbb18e30dea8b7"}
Sep 16 14:27:51.507678 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:51.507622    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2"
Sep 16 14:27:51.526231 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:51.526185    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk" podStartSLOduration=1.526171697 podStartE2EDuration="1.526171697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:27:51.524349953 +0000 UTC m=+820.127974985" watchObservedRunningTime="2026-09-16 14:27:51.526171697 +0000 UTC m=+820.129796729"
Sep 16 14:27:51.543269 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:51.543221    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2" podStartSLOduration=1.543203653 podStartE2EDuration="1.543203653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:27:51.542196668 +0000 UTC m=+820.145821701" watchObservedRunningTime="2026-09-16 14:27:51.543203653 +0000 UTC m=+820.146828686"
Sep 16 14:27:58.075547 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.075509    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5"]
Sep 16 14:27:58.076048 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.075787    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5" podUID="ed70c51d-d881-4822-84f7-e909809e04dd" containerName="payload-processing" containerID="cri-o://a2f707934687be40620d6db57d2438a2b624c34ef6a8b7cba45683927d109522" gracePeriod=30
Sep 16 14:27:58.089228 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.089196    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq"]
Sep 16 14:27:58.089460 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.089428    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq" podUID="73e717ad-74d4-446b-a717-f346a293798d" containerName="payload-pre-processing" containerID="cri-o://ed9bb24b6ecfab3b4b1617c08ea4560df57e5434b5fe210c02ffa0d1258d0f7f" gracePeriod=30
Sep 16 14:27:58.348714 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.348687    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5"
Sep 16 14:27:58.374352 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.374324    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq"
Sep 16 14:27:58.436097 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.436063    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed70c51d-d881-4822-84f7-e909809e04dd-plugins-config-volume\") pod \"ed70c51d-d881-4822-84f7-e909809e04dd\" (UID: \"ed70c51d-d881-4822-84f7-e909809e04dd\") "
Sep 16 14:27:58.436230 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.436172    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9v2xr\" (UniqueName: \"kubernetes.io/projected/ed70c51d-d881-4822-84f7-e909809e04dd-kube-api-access-9v2xr\") pod \"ed70c51d-d881-4822-84f7-e909809e04dd\" (UID: \"ed70c51d-d881-4822-84f7-e909809e04dd\") "
Sep 16 14:27:58.436521 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.436496    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed70c51d-d881-4822-84f7-e909809e04dd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ed70c51d-d881-4822-84f7-e909809e04dd" (UID: "ed70c51d-d881-4822-84f7-e909809e04dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:27:58.438384 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.438360    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed70c51d-d881-4822-84f7-e909809e04dd-kube-api-access-9v2xr" (OuterVolumeSpecName: "kube-api-access-9v2xr") pod "ed70c51d-d881-4822-84f7-e909809e04dd" (UID: "ed70c51d-d881-4822-84f7-e909809e04dd"). InnerVolumeSpecName "kube-api-access-9v2xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:27:58.537288 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.537260    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwf7r\" (UniqueName: \"kubernetes.io/projected/73e717ad-74d4-446b-a717-f346a293798d-kube-api-access-kwf7r\") pod \"73e717ad-74d4-446b-a717-f346a293798d\" (UID: \"73e717ad-74d4-446b-a717-f346a293798d\") "
Sep 16 14:27:58.537459 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.537337    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e717ad-74d4-446b-a717-f346a293798d-plugins-config-volume\") pod \"73e717ad-74d4-446b-a717-f346a293798d\" (UID: \"73e717ad-74d4-446b-a717-f346a293798d\") "
Sep 16 14:27:58.537586 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.537573    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed70c51d-d881-4822-84f7-e909809e04dd-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:27:58.537627 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.537589    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v2xr\" (UniqueName: \"kubernetes.io/projected/ed70c51d-d881-4822-84f7-e909809e04dd-kube-api-access-9v2xr\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:27:58.537784 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.537758    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e717ad-74d4-446b-a717-f346a293798d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "73e717ad-74d4-446b-a717-f346a293798d" (UID: "73e717ad-74d4-446b-a717-f346a293798d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:27:58.539482 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.539460    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e717ad-74d4-446b-a717-f346a293798d-kube-api-access-kwf7r" (OuterVolumeSpecName: "kube-api-access-kwf7r") pod "73e717ad-74d4-446b-a717-f346a293798d" (UID: "73e717ad-74d4-446b-a717-f346a293798d"). InnerVolumeSpecName "kube-api-access-kwf7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:27:58.550132 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.550098    2587 generic.go:356] "Generic (PLEG): container finished" podID="ed70c51d-d881-4822-84f7-e909809e04dd" containerID="a2f707934687be40620d6db57d2438a2b624c34ef6a8b7cba45683927d109522" exitCode=0
Sep 16 14:27:58.550266 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.550158    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5"
Sep 16 14:27:58.550266 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.550183    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5" event={"ID":"ed70c51d-d881-4822-84f7-e909809e04dd","Type":"ContainerDied","Data":"a2f707934687be40620d6db57d2438a2b624c34ef6a8b7cba45683927d109522"}
Sep 16 14:27:58.550384 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.550259    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5" event={"ID":"ed70c51d-d881-4822-84f7-e909809e04dd","Type":"ContainerDied","Data":"7d174c7cb5fb6636775b843ab26422f23a5c449d3759f4a77253a48f43df2f0c"}
Sep 16 14:27:58.550384 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.550319    2587 scope.go:117] "RemoveContainer" containerID="a2f707934687be40620d6db57d2438a2b624c34ef6a8b7cba45683927d109522"
Sep 16 14:27:58.551707 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.551683    2587 generic.go:356] "Generic (PLEG): container finished" podID="73e717ad-74d4-446b-a717-f346a293798d" containerID="ed9bb24b6ecfab3b4b1617c08ea4560df57e5434b5fe210c02ffa0d1258d0f7f" exitCode=0
Sep 16 14:27:58.551789 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.551735    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq" event={"ID":"73e717ad-74d4-446b-a717-f346a293798d","Type":"ContainerDied","Data":"ed9bb24b6ecfab3b4b1617c08ea4560df57e5434b5fe210c02ffa0d1258d0f7f"}
Sep 16 14:27:58.551789 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.551774    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq" event={"ID":"73e717ad-74d4-446b-a717-f346a293798d","Type":"ContainerDied","Data":"f4da8c33c1a6955261f61f9be48837e6c9e61e69be6d1655daf99bffcaead078"}
Sep 16 14:27:58.551901 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.551836    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq"
Sep 16 14:27:58.559829 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.559811    2587 scope.go:117] "RemoveContainer" containerID="a2f707934687be40620d6db57d2438a2b624c34ef6a8b7cba45683927d109522"
Sep 16 14:27:58.560117 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:27:58.560096    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2f707934687be40620d6db57d2438a2b624c34ef6a8b7cba45683927d109522\": container with ID starting with a2f707934687be40620d6db57d2438a2b624c34ef6a8b7cba45683927d109522 not found: ID does not exist" containerID="a2f707934687be40620d6db57d2438a2b624c34ef6a8b7cba45683927d109522"
Sep 16 14:27:58.560206 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.560125    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2f707934687be40620d6db57d2438a2b624c34ef6a8b7cba45683927d109522"} err="failed to get container status \"a2f707934687be40620d6db57d2438a2b624c34ef6a8b7cba45683927d109522\": rpc error: code = NotFound desc = could not find container \"a2f707934687be40620d6db57d2438a2b624c34ef6a8b7cba45683927d109522\": container with ID starting with a2f707934687be40620d6db57d2438a2b624c34ef6a8b7cba45683927d109522 not found: ID does not exist"
Sep 16 14:27:58.560206 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.560143    2587 scope.go:117] "RemoveContainer" containerID="ed9bb24b6ecfab3b4b1617c08ea4560df57e5434b5fe210c02ffa0d1258d0f7f"
Sep 16 14:27:58.568289 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.568274    2587 scope.go:117] "RemoveContainer" containerID="ed9bb24b6ecfab3b4b1617c08ea4560df57e5434b5fe210c02ffa0d1258d0f7f"
Sep 16 14:27:58.568581 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:27:58.568563    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed9bb24b6ecfab3b4b1617c08ea4560df57e5434b5fe210c02ffa0d1258d0f7f\": container with ID starting with ed9bb24b6ecfab3b4b1617c08ea4560df57e5434b5fe210c02ffa0d1258d0f7f not found: ID does not exist" containerID="ed9bb24b6ecfab3b4b1617c08ea4560df57e5434b5fe210c02ffa0d1258d0f7f"
Sep 16 14:27:58.568626 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.568588    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed9bb24b6ecfab3b4b1617c08ea4560df57e5434b5fe210c02ffa0d1258d0f7f"} err="failed to get container status \"ed9bb24b6ecfab3b4b1617c08ea4560df57e5434b5fe210c02ffa0d1258d0f7f\": rpc error: code = NotFound desc = could not find container \"ed9bb24b6ecfab3b4b1617c08ea4560df57e5434b5fe210c02ffa0d1258d0f7f\": container with ID starting with ed9bb24b6ecfab3b4b1617c08ea4560df57e5434b5fe210c02ffa0d1258d0f7f not found: ID does not exist"
Sep 16 14:27:58.577165 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.577145    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq"]
Sep 16 14:27:58.579073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.579014    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1e4514fd-fc9dbb4b-cswwq"]
Sep 16 14:27:58.587997 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.587978    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5"]
Sep 16 14:27:58.593686 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.593664    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1e4514fd-95859dbd7-q7pj5"]
Sep 16 14:27:58.638251 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.638232    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwf7r\" (UniqueName: \"kubernetes.io/projected/73e717ad-74d4-446b-a717-f346a293798d-kube-api-access-kwf7r\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:27:58.638344 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:58.638254    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e717ad-74d4-446b-a717-f346a293798d-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:27:59.903325 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:59.903293    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73e717ad-74d4-446b-a717-f346a293798d" path="/var/lib/kubelet/pods/73e717ad-74d4-446b-a717-f346a293798d/volumes"
Sep 16 14:27:59.903745 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:27:59.903729    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed70c51d-d881-4822-84f7-e909809e04dd" path="/var/lib/kubelet/pods/ed70c51d-d881-4822-84f7-e909809e04dd/volumes"
Sep 16 14:28:02.515571 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:02.515533    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2"
Sep 16 14:28:02.515915 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:02.515703    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk"
Sep 16 14:28:04.423859 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:04.423830    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv"]
Sep 16 14:28:04.424226 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:04.424212    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73e717ad-74d4-446b-a717-f346a293798d" containerName="payload-pre-processing"
Sep 16 14:28:04.424275 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:04.424228    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="73e717ad-74d4-446b-a717-f346a293798d" containerName="payload-pre-processing"
Sep 16 14:28:04.424275 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:04.424240    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed70c51d-d881-4822-84f7-e909809e04dd" containerName="payload-processing"
Sep 16 14:28:04.424275 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:04.424245    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed70c51d-d881-4822-84f7-e909809e04dd" containerName="payload-processing"
Sep 16 14:28:04.424374 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:04.424312    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed70c51d-d881-4822-84f7-e909809e04dd" containerName="payload-processing"
Sep 16 14:28:04.424374 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:04.424324    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="73e717ad-74d4-446b-a717-f346a293798d" containerName="payload-pre-processing"
Sep 16 14:28:04.428798 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:04.428782    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv"
Sep 16 14:28:04.431576 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:04.431551    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-7cfdb2ad-dockercfg-sjww4\""
Sep 16 14:28:04.431748 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:04.431730    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-7cfdb2ad\""
Sep 16 14:28:04.438217 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:04.438194    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv"]
Sep 16 14:28:04.456430 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:04.456379    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9"]
Sep 16 14:28:04.459983 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:04.459966    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9"
Sep 16 14:28:04.476836 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:04.476805    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9"]
Sep 16 14:28:04.592975 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:04.592946    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0abf5701-a913-4fbb-8ca2-e1d841ed81eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv\" (UID: \"0abf5701-a913-4fbb-8ca2-e1d841ed81eb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv"
Sep 16 14:28:04.593142 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:04.592984    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cknd\" (UniqueName: \"kubernetes.io/projected/d5b50637-bbd2-4288-84b1-c2c73731d241-kube-api-access-2cknd\") pod \"payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9\" (UID: \"d5b50637-bbd2-4288-84b1-c2c73731d241\") " pod="openshift-ingress/payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9"
Sep 16 14:28:04.593142 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:04.593008    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr9nc\" (UniqueName: \"kubernetes.io/projected/0abf5701-a913-4fbb-8ca2-e1d841ed81eb-kube-api-access-xr9nc\") pod \"payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv\" (UID: \"0abf5701-a913-4fbb-8ca2-e1d841ed81eb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv"
Sep 16 14:28:04.593142 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:04.593032    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b50637-bbd2-4288-84b1-c2c73731d241-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9\" (UID: \"d5b50637-bbd2-4288-84b1-c2c73731d241\") " pod="openshift-ingress/payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9"
Sep 16 14:28:04.694384 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:04.694291    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0abf5701-a913-4fbb-8ca2-e1d841ed81eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv\" (UID: \"0abf5701-a913-4fbb-8ca2-e1d841ed81eb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv"
Sep 16 14:28:04.694384 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:04.694338    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cknd\" (UniqueName: \"kubernetes.io/projected/d5b50637-bbd2-4288-84b1-c2c73731d241-kube-api-access-2cknd\") pod \"payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9\" (UID: \"d5b50637-bbd2-4288-84b1-c2c73731d241\") " pod="openshift-ingress/payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9"
Sep 16 14:28:04.694384 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:04.694369    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr9nc\" (UniqueName: \"kubernetes.io/projected/0abf5701-a913-4fbb-8ca2-e1d841ed81eb-kube-api-access-xr9nc\") pod \"payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv\" (UID: \"0abf5701-a913-4fbb-8ca2-e1d841ed81eb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv"
Sep 16 14:28:04.694647 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:04.694440    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b50637-bbd2-4288-84b1-c2c73731d241-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9\" (UID: \"d5b50637-bbd2-4288-84b1-c2c73731d241\") " pod="openshift-ingress/payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9"
Sep 16 14:28:04.695005 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:04.694986    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0abf5701-a913-4fbb-8ca2-e1d841ed81eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv\" (UID: \"0abf5701-a913-4fbb-8ca2-e1d841ed81eb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv"
Sep 16 14:28:04.695223 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:04.695205    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b50637-bbd2-4288-84b1-c2c73731d241-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9\" (UID: \"d5b50637-bbd2-4288-84b1-c2c73731d241\") " pod="openshift-ingress/payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9"
Sep 16 14:28:04.703190 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:04.703158    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr9nc\" (UniqueName: \"kubernetes.io/projected/0abf5701-a913-4fbb-8ca2-e1d841ed81eb-kube-api-access-xr9nc\") pod \"payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv\" (UID: \"0abf5701-a913-4fbb-8ca2-e1d841ed81eb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv"
Sep 16 14:28:04.703312 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:04.703158    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cknd\" (UniqueName: \"kubernetes.io/projected/d5b50637-bbd2-4288-84b1-c2c73731d241-kube-api-access-2cknd\") pod \"payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9\" (UID: \"d5b50637-bbd2-4288-84b1-c2c73731d241\") " pod="openshift-ingress/payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9"
Sep 16 14:28:04.740170 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:04.740141    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv"
Sep 16 14:28:04.769729 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:04.769705    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9"
Sep 16 14:28:05.023288 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:05.023262    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv"]
Sep 16 14:28:05.025213 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:28:05.025186    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0abf5701_a913_4fbb_8ca2_e1d841ed81eb.slice/crio-fda8b7b9cb3f8f558090ff669ba857dfaf7d854b505d4eeebb1a8bd816fad847 WatchSource:0}: Error finding container fda8b7b9cb3f8f558090ff669ba857dfaf7d854b505d4eeebb1a8bd816fad847: Status 404 returned error can't find the container with id fda8b7b9cb3f8f558090ff669ba857dfaf7d854b505d4eeebb1a8bd816fad847
Sep 16 14:28:05.067566 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:05.067544    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9"]
Sep 16 14:28:05.069511 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:28:05.069483    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b50637_bbd2_4288_84b1_c2c73731d241.slice/crio-990c9c1815005d2307a4748fe7aa1c38e5ef3db3cffa80916778e11590064f2d WatchSource:0}: Error finding container 990c9c1815005d2307a4748fe7aa1c38e5ef3db3cffa80916778e11590064f2d: Status 404 returned error can't find the container with id 990c9c1815005d2307a4748fe7aa1c38e5ef3db3cffa80916778e11590064f2d
Sep 16 14:28:05.596205 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:05.596167    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9" event={"ID":"d5b50637-bbd2-4288-84b1-c2c73731d241","Type":"ContainerStarted","Data":"694b5786dd48689b7c1e280ef6634be45efb1c5d4c6b3a1a2d8c000f0eaf91d5"}
Sep 16 14:28:05.596698 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:05.596232    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9" event={"ID":"d5b50637-bbd2-4288-84b1-c2c73731d241","Type":"ContainerStarted","Data":"990c9c1815005d2307a4748fe7aa1c38e5ef3db3cffa80916778e11590064f2d"}
Sep 16 14:28:05.596698 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:05.596262    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9"
Sep 16 14:28:05.598275 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:05.598248    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv" event={"ID":"0abf5701-a913-4fbb-8ca2-e1d841ed81eb","Type":"ContainerStarted","Data":"c7ea1b197056a4ccaebe4bb64b3a5856c74a6a34f779725a611a2f782e2195e1"}
Sep 16 14:28:05.598405 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:05.598303    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv" event={"ID":"0abf5701-a913-4fbb-8ca2-e1d841ed81eb","Type":"ContainerStarted","Data":"fda8b7b9cb3f8f558090ff669ba857dfaf7d854b505d4eeebb1a8bd816fad847"}
Sep 16 14:28:05.598405 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:05.598332    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv"
Sep 16 14:28:05.618531 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:05.618475    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9" podStartSLOduration=1.6184556570000002 podStartE2EDuration="1.618455657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:28:05.615073194 +0000 UTC m=+834.218698227" watchObservedRunningTime="2026-09-16 14:28:05.618455657 +0000 UTC m=+834.222080687"
Sep 16 14:28:05.636680 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:05.636474    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv" podStartSLOduration=1.636458996 podStartE2EDuration="1.636458996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:28:05.636105503 +0000 UTC m=+834.239730536" watchObservedRunningTime="2026-09-16 14:28:05.636458996 +0000 UTC m=+834.240084030"
Sep 16 14:28:16.606405 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:16.606357    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9"
Sep 16 14:28:16.606911 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:16.606456    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv"
Sep 16 14:28:34.237167 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.237132    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9"]
Sep 16 14:28:34.237623 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.237412    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9" podUID="d5b50637-bbd2-4288-84b1-c2c73731d241" containerName="payload-processing" containerID="cri-o://694b5786dd48689b7c1e280ef6634be45efb1c5d4c6b3a1a2d8c000f0eaf91d5" gracePeriod=30
Sep 16 14:28:34.246286 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.246249    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv"]
Sep 16 14:28:34.246559 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.246534    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv" podUID="0abf5701-a913-4fbb-8ca2-e1d841ed81eb" containerName="payload-pre-processing" containerID="cri-o://c7ea1b197056a4ccaebe4bb64b3a5856c74a6a34f779725a611a2f782e2195e1" gracePeriod=30
Sep 16 14:28:34.535485 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.535451    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9"
Sep 16 14:28:34.538204 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.538180    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv"
Sep 16 14:28:34.667241 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.667204    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b50637-bbd2-4288-84b1-c2c73731d241-plugins-config-volume\") pod \"d5b50637-bbd2-4288-84b1-c2c73731d241\" (UID: \"d5b50637-bbd2-4288-84b1-c2c73731d241\") "
Sep 16 14:28:34.667480 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.667258    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cknd\" (UniqueName: \"kubernetes.io/projected/d5b50637-bbd2-4288-84b1-c2c73731d241-kube-api-access-2cknd\") pod \"d5b50637-bbd2-4288-84b1-c2c73731d241\" (UID: \"d5b50637-bbd2-4288-84b1-c2c73731d241\") "
Sep 16 14:28:34.667480 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.667299    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0abf5701-a913-4fbb-8ca2-e1d841ed81eb-plugins-config-volume\") pod \"0abf5701-a913-4fbb-8ca2-e1d841ed81eb\" (UID: \"0abf5701-a913-4fbb-8ca2-e1d841ed81eb\") "
Sep 16 14:28:34.667480 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.667343    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xr9nc\" (UniqueName: \"kubernetes.io/projected/0abf5701-a913-4fbb-8ca2-e1d841ed81eb-kube-api-access-xr9nc\") pod \"0abf5701-a913-4fbb-8ca2-e1d841ed81eb\" (UID: \"0abf5701-a913-4fbb-8ca2-e1d841ed81eb\") "
Sep 16 14:28:34.667672 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.667588    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b50637-bbd2-4288-84b1-c2c73731d241-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d5b50637-bbd2-4288-84b1-c2c73731d241" (UID: "d5b50637-bbd2-4288-84b1-c2c73731d241"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:28:34.667848 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.667823    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0abf5701-a913-4fbb-8ca2-e1d841ed81eb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0abf5701-a913-4fbb-8ca2-e1d841ed81eb" (UID: "0abf5701-a913-4fbb-8ca2-e1d841ed81eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:28:34.669688 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.669658    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b50637-bbd2-4288-84b1-c2c73731d241-kube-api-access-2cknd" (OuterVolumeSpecName: "kube-api-access-2cknd") pod "d5b50637-bbd2-4288-84b1-c2c73731d241" (UID: "d5b50637-bbd2-4288-84b1-c2c73731d241"). InnerVolumeSpecName "kube-api-access-2cknd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:28:34.669819 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.669695    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0abf5701-a913-4fbb-8ca2-e1d841ed81eb-kube-api-access-xr9nc" (OuterVolumeSpecName: "kube-api-access-xr9nc") pod "0abf5701-a913-4fbb-8ca2-e1d841ed81eb" (UID: "0abf5701-a913-4fbb-8ca2-e1d841ed81eb"). InnerVolumeSpecName "kube-api-access-xr9nc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:28:34.754948 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.754905    2587 generic.go:356] "Generic (PLEG): container finished" podID="d5b50637-bbd2-4288-84b1-c2c73731d241" containerID="694b5786dd48689b7c1e280ef6634be45efb1c5d4c6b3a1a2d8c000f0eaf91d5" exitCode=0
Sep 16 14:28:34.754948 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.754947    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9"
Sep 16 14:28:34.755200 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.754944    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9" event={"ID":"d5b50637-bbd2-4288-84b1-c2c73731d241","Type":"ContainerDied","Data":"694b5786dd48689b7c1e280ef6634be45efb1c5d4c6b3a1a2d8c000f0eaf91d5"}
Sep 16 14:28:34.755200 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.755072    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9" event={"ID":"d5b50637-bbd2-4288-84b1-c2c73731d241","Type":"ContainerDied","Data":"990c9c1815005d2307a4748fe7aa1c38e5ef3db3cffa80916778e11590064f2d"}
Sep 16 14:28:34.755200 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.755102    2587 scope.go:117] "RemoveContainer" containerID="694b5786dd48689b7c1e280ef6634be45efb1c5d4c6b3a1a2d8c000f0eaf91d5"
Sep 16 14:28:34.756523 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.756502    2587 generic.go:356] "Generic (PLEG): container finished" podID="0abf5701-a913-4fbb-8ca2-e1d841ed81eb" containerID="c7ea1b197056a4ccaebe4bb64b3a5856c74a6a34f779725a611a2f782e2195e1" exitCode=0
Sep 16 14:28:34.756694 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.756530    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv"
Sep 16 14:28:34.756694 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.756571    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv" event={"ID":"0abf5701-a913-4fbb-8ca2-e1d841ed81eb","Type":"ContainerDied","Data":"c7ea1b197056a4ccaebe4bb64b3a5856c74a6a34f779725a611a2f782e2195e1"}
Sep 16 14:28:34.756694 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.756610    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv" event={"ID":"0abf5701-a913-4fbb-8ca2-e1d841ed81eb","Type":"ContainerDied","Data":"fda8b7b9cb3f8f558090ff669ba857dfaf7d854b505d4eeebb1a8bd816fad847"}
Sep 16 14:28:34.764992 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.764952    2587 scope.go:117] "RemoveContainer" containerID="694b5786dd48689b7c1e280ef6634be45efb1c5d4c6b3a1a2d8c000f0eaf91d5"
Sep 16 14:28:34.765334 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:28:34.765304    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"694b5786dd48689b7c1e280ef6634be45efb1c5d4c6b3a1a2d8c000f0eaf91d5\": container with ID starting with 694b5786dd48689b7c1e280ef6634be45efb1c5d4c6b3a1a2d8c000f0eaf91d5 not found: ID does not exist" containerID="694b5786dd48689b7c1e280ef6634be45efb1c5d4c6b3a1a2d8c000f0eaf91d5"
Sep 16 14:28:34.765441 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.765339    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"694b5786dd48689b7c1e280ef6634be45efb1c5d4c6b3a1a2d8c000f0eaf91d5"} err="failed to get container status \"694b5786dd48689b7c1e280ef6634be45efb1c5d4c6b3a1a2d8c000f0eaf91d5\": rpc error: code = NotFound desc = could not find container \"694b5786dd48689b7c1e280ef6634be45efb1c5d4c6b3a1a2d8c000f0eaf91d5\": container with ID starting with 694b5786dd48689b7c1e280ef6634be45efb1c5d4c6b3a1a2d8c000f0eaf91d5 not found: ID does not exist"
Sep 16 14:28:34.765441 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.765357    2587 scope.go:117] "RemoveContainer" containerID="c7ea1b197056a4ccaebe4bb64b3a5856c74a6a34f779725a611a2f782e2195e1"
Sep 16 14:28:34.768776 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.768757    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0abf5701-a913-4fbb-8ca2-e1d841ed81eb-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:28:34.768852 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.768778    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr9nc\" (UniqueName: \"kubernetes.io/projected/0abf5701-a913-4fbb-8ca2-e1d841ed81eb-kube-api-access-xr9nc\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:28:34.768852 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.768788    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b50637-bbd2-4288-84b1-c2c73731d241-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:28:34.768852 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.768798    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cknd\" (UniqueName: \"kubernetes.io/projected/d5b50637-bbd2-4288-84b1-c2c73731d241-kube-api-access-2cknd\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:28:34.773577 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.773561    2587 scope.go:117] "RemoveContainer" containerID="c7ea1b197056a4ccaebe4bb64b3a5856c74a6a34f779725a611a2f782e2195e1"
Sep 16 14:28:34.773822 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:28:34.773805    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7ea1b197056a4ccaebe4bb64b3a5856c74a6a34f779725a611a2f782e2195e1\": container with ID starting with c7ea1b197056a4ccaebe4bb64b3a5856c74a6a34f779725a611a2f782e2195e1 not found: ID does not exist" containerID="c7ea1b197056a4ccaebe4bb64b3a5856c74a6a34f779725a611a2f782e2195e1"
Sep 16 14:28:34.773877 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.773828    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7ea1b197056a4ccaebe4bb64b3a5856c74a6a34f779725a611a2f782e2195e1"} err="failed to get container status \"c7ea1b197056a4ccaebe4bb64b3a5856c74a6a34f779725a611a2f782e2195e1\": rpc error: code = NotFound desc = could not find container \"c7ea1b197056a4ccaebe4bb64b3a5856c74a6a34f779725a611a2f782e2195e1\": container with ID starting with c7ea1b197056a4ccaebe4bb64b3a5856c74a6a34f779725a611a2f782e2195e1 not found: ID does not exist"
Sep 16 14:28:34.799893 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.799860    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9"]
Sep 16 14:28:34.814034 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.814004    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7cfdb2ad-76b59fcc98-jhtt9"]
Sep 16 14:28:34.839142 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.839103    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv"]
Sep 16 14:28:34.841138 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:34.841118    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7cfdb2ad-5dbd7f8566-lr2zv"]
Sep 16 14:28:35.904026 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:35.903984    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0abf5701-a913-4fbb-8ca2-e1d841ed81eb" path="/var/lib/kubelet/pods/0abf5701-a913-4fbb-8ca2-e1d841ed81eb/volumes"
Sep 16 14:28:35.904555 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:35.904535    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5b50637-bbd2-4288-84b1-c2c73731d241" path="/var/lib/kubelet/pods/d5b50637-bbd2-4288-84b1-c2c73731d241/volumes"
Sep 16 14:28:42.092092 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.092057    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5317692d-5f79588665-rdztg"]
Sep 16 14:28:42.092507 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.092480    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0abf5701-a913-4fbb-8ca2-e1d841ed81eb" containerName="payload-pre-processing"
Sep 16 14:28:42.092507 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.092493    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="0abf5701-a913-4fbb-8ca2-e1d841ed81eb" containerName="payload-pre-processing"
Sep 16 14:28:42.092507 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.092503    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5b50637-bbd2-4288-84b1-c2c73731d241" containerName="payload-processing"
Sep 16 14:28:42.092507 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.092509    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5b50637-bbd2-4288-84b1-c2c73731d241" containerName="payload-processing"
Sep 16 14:28:42.092644 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.092566    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="0abf5701-a913-4fbb-8ca2-e1d841ed81eb" containerName="payload-pre-processing"
Sep 16 14:28:42.092644 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.092576    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5b50637-bbd2-4288-84b1-c2c73731d241" containerName="payload-processing"
Sep 16 14:28:42.098265 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.098234    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5317692d-5f79588665-rdztg"
Sep 16 14:28:42.102347 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.102323    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5317692d\""
Sep 16 14:28:42.122202 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.122167    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5317692d-5f79588665-rdztg"]
Sep 16 14:28:42.147329 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.147291    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4"]
Sep 16 14:28:42.151071 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.151049    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4"
Sep 16 14:28:42.154782 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.154759    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5317692d-dockercfg-bf4xg\""
Sep 16 14:28:42.165466 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.165432    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4"]
Sep 16 14:28:42.244165 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.244126    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3a8e0cb-1fe6-4cd3-a8db-810e14447447-plugins-config-volume\") pod \"payload-processing-e2e-ait-5317692d-5f79588665-rdztg\" (UID: \"e3a8e0cb-1fe6-4cd3-a8db-810e14447447\") " pod="openshift-ingress/payload-processing-e2e-ait-5317692d-5f79588665-rdztg"
Sep 16 14:28:42.244466 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.244264    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwbpq\" (UniqueName: \"kubernetes.io/projected/e3a8e0cb-1fe6-4cd3-a8db-810e14447447-kube-api-access-xwbpq\") pod \"payload-processing-e2e-ait-5317692d-5f79588665-rdztg\" (UID: \"e3a8e0cb-1fe6-4cd3-a8db-810e14447447\") " pod="openshift-ingress/payload-processing-e2e-ait-5317692d-5f79588665-rdztg"
Sep 16 14:28:42.244466 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.244284    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb25e4e-cf32-46b1-9a32-fc0be171d2f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4\" (UID: \"3cb25e4e-cf32-46b1-9a32-fc0be171d2f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4"
Sep 16 14:28:42.244466 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.244362    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htnmp\" (UniqueName: \"kubernetes.io/projected/3cb25e4e-cf32-46b1-9a32-fc0be171d2f4-kube-api-access-htnmp\") pod \"payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4\" (UID: \"3cb25e4e-cf32-46b1-9a32-fc0be171d2f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4"
Sep 16 14:28:42.345781 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.345666    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwbpq\" (UniqueName: \"kubernetes.io/projected/e3a8e0cb-1fe6-4cd3-a8db-810e14447447-kube-api-access-xwbpq\") pod \"payload-processing-e2e-ait-5317692d-5f79588665-rdztg\" (UID: \"e3a8e0cb-1fe6-4cd3-a8db-810e14447447\") " pod="openshift-ingress/payload-processing-e2e-ait-5317692d-5f79588665-rdztg"
Sep 16 14:28:42.345781 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.345714    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb25e4e-cf32-46b1-9a32-fc0be171d2f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4\" (UID: \"3cb25e4e-cf32-46b1-9a32-fc0be171d2f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4"
Sep 16 14:28:42.345781 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.345740    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htnmp\" (UniqueName: \"kubernetes.io/projected/3cb25e4e-cf32-46b1-9a32-fc0be171d2f4-kube-api-access-htnmp\") pod \"payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4\" (UID: \"3cb25e4e-cf32-46b1-9a32-fc0be171d2f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4"
Sep 16 14:28:42.345781 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.345760    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3a8e0cb-1fe6-4cd3-a8db-810e14447447-plugins-config-volume\") pod \"payload-processing-e2e-ait-5317692d-5f79588665-rdztg\" (UID: \"e3a8e0cb-1fe6-4cd3-a8db-810e14447447\") " pod="openshift-ingress/payload-processing-e2e-ait-5317692d-5f79588665-rdztg"
Sep 16 14:28:42.346519 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.346500    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3a8e0cb-1fe6-4cd3-a8db-810e14447447-plugins-config-volume\") pod \"payload-processing-e2e-ait-5317692d-5f79588665-rdztg\" (UID: \"e3a8e0cb-1fe6-4cd3-a8db-810e14447447\") " pod="openshift-ingress/payload-processing-e2e-ait-5317692d-5f79588665-rdztg"
Sep 16 14:28:42.346595 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.346552    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb25e4e-cf32-46b1-9a32-fc0be171d2f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4\" (UID: \"3cb25e4e-cf32-46b1-9a32-fc0be171d2f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4"
Sep 16 14:28:42.356518 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.356488    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwbpq\" (UniqueName: \"kubernetes.io/projected/e3a8e0cb-1fe6-4cd3-a8db-810e14447447-kube-api-access-xwbpq\") pod \"payload-processing-e2e-ait-5317692d-5f79588665-rdztg\" (UID: \"e3a8e0cb-1fe6-4cd3-a8db-810e14447447\") " pod="openshift-ingress/payload-processing-e2e-ait-5317692d-5f79588665-rdztg"
Sep 16 14:28:42.356801 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.356782    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htnmp\" (UniqueName: \"kubernetes.io/projected/3cb25e4e-cf32-46b1-9a32-fc0be171d2f4-kube-api-access-htnmp\") pod \"payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4\" (UID: \"3cb25e4e-cf32-46b1-9a32-fc0be171d2f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4"
Sep 16 14:28:42.410664 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.410625    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5317692d-5f79588665-rdztg"
Sep 16 14:28:42.462891 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.462514    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4"
Sep 16 14:28:42.703254 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.703219    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5317692d-5f79588665-rdztg"]
Sep 16 14:28:42.706744 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:28:42.706711    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a8e0cb_1fe6_4cd3_a8db_810e14447447.slice/crio-a836b270de41c81454eda70658e1a53adb95d86ad528a17c47cd5465e48e973d WatchSource:0}: Error finding container a836b270de41c81454eda70658e1a53adb95d86ad528a17c47cd5465e48e973d: Status 404 returned error can't find the container with id a836b270de41c81454eda70658e1a53adb95d86ad528a17c47cd5465e48e973d
Sep 16 14:28:42.769454 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.769428    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4"]
Sep 16 14:28:42.772011 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:28:42.771976    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb25e4e_cf32_46b1_9a32_fc0be171d2f4.slice/crio-a34f028b8b854c76decd972e596bce1690c647e500b6f26f4e47dbfebcfec66b WatchSource:0}: Error finding container a34f028b8b854c76decd972e596bce1690c647e500b6f26f4e47dbfebcfec66b: Status 404 returned error can't find the container with id a34f028b8b854c76decd972e596bce1690c647e500b6f26f4e47dbfebcfec66b
Sep 16 14:28:42.801957 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.801922    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4" event={"ID":"3cb25e4e-cf32-46b1-9a32-fc0be171d2f4","Type":"ContainerStarted","Data":"a34f028b8b854c76decd972e596bce1690c647e500b6f26f4e47dbfebcfec66b"}
Sep 16 14:28:42.803616 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.803585    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5317692d-5f79588665-rdztg" event={"ID":"e3a8e0cb-1fe6-4cd3-a8db-810e14447447","Type":"ContainerStarted","Data":"d251d0c54ba01d3c5f503186833c792995bc2a6a5796613318abb6950ac17512"}
Sep 16 14:28:42.803745 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.803639    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5317692d-5f79588665-rdztg" event={"ID":"e3a8e0cb-1fe6-4cd3-a8db-810e14447447","Type":"ContainerStarted","Data":"a836b270de41c81454eda70658e1a53adb95d86ad528a17c47cd5465e48e973d"}
Sep 16 14:28:42.803745 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.803678    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5317692d-5f79588665-rdztg"
Sep 16 14:28:42.828813 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:42.828764    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5317692d-5f79588665-rdztg" podStartSLOduration=0.828745327 podStartE2EDuration="828.745327ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:28:42.827939667 +0000 UTC m=+871.431564701" watchObservedRunningTime="2026-09-16 14:28:42.828745327 +0000 UTC m=+871.432370419"
Sep 16 14:28:43.509633 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:43.509588    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"]
Sep 16 14:28:43.513633 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:43.513605    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"
Sep 16 14:28:43.517511 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:43.517484    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-84fdff\"/\"openshift-service-ca.crt\""
Sep 16 14:28:43.518819 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:43.518790    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-84fdff\"/\"default-dockercfg-dh7xv\""
Sep 16 14:28:43.518819 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:43.518810    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-84fdff\"/\"autores-model-84fdff-kserve-self-signed-certs\""
Sep 16 14:28:43.519061 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:43.518833    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-84fdff\"/\"kube-root-ca.crt\""
Sep 16 14:28:43.528273 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:43.528234    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"]
Sep 16 14:28:43.658065 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:43.658018    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cfcefbe-9baa-4157-93f6-4474dbefd43c-model-cache\") pod \"autores-model-84fdff-kserve-5b5869dff-7ws2c\" (UID: \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\") " pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"
Sep 16 14:28:43.658272 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:43.658112    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cfcefbe-9baa-4157-93f6-4474dbefd43c-dshm\") pod \"autores-model-84fdff-kserve-5b5869dff-7ws2c\" (UID: \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\") " pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"
Sep 16 14:28:43.658272 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:43.658141    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cfcefbe-9baa-4157-93f6-4474dbefd43c-tmp-dir\") pod \"autores-model-84fdff-kserve-5b5869dff-7ws2c\" (UID: \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\") " pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"
Sep 16 14:28:43.658272 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:43.658169    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qltd6\" (UniqueName: \"kubernetes.io/projected/3cfcefbe-9baa-4157-93f6-4474dbefd43c-kube-api-access-qltd6\") pod \"autores-model-84fdff-kserve-5b5869dff-7ws2c\" (UID: \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\") " pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"
Sep 16 14:28:43.658272 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:43.658245    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cfcefbe-9baa-4157-93f6-4474dbefd43c-home\") pod \"autores-model-84fdff-kserve-5b5869dff-7ws2c\" (UID: \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\") " pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"
Sep 16 14:28:43.658499 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:43.658332    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cfcefbe-9baa-4157-93f6-4474dbefd43c-tls-certs\") pod \"autores-model-84fdff-kserve-5b5869dff-7ws2c\" (UID: \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\") " pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"
Sep 16 14:28:43.759199 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:43.759163    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cfcefbe-9baa-4157-93f6-4474dbefd43c-model-cache\") pod \"autores-model-84fdff-kserve-5b5869dff-7ws2c\" (UID: \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\") " pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"
Sep 16 14:28:43.759438 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:43.759226    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cfcefbe-9baa-4157-93f6-4474dbefd43c-dshm\") pod \"autores-model-84fdff-kserve-5b5869dff-7ws2c\" (UID: \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\") " pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"
Sep 16 14:28:43.759438 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:43.759250    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cfcefbe-9baa-4157-93f6-4474dbefd43c-tmp-dir\") pod \"autores-model-84fdff-kserve-5b5869dff-7ws2c\" (UID: \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\") " pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"
Sep 16 14:28:43.759438 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:43.759274    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qltd6\" (UniqueName: \"kubernetes.io/projected/3cfcefbe-9baa-4157-93f6-4474dbefd43c-kube-api-access-qltd6\") pod \"autores-model-84fdff-kserve-5b5869dff-7ws2c\" (UID: \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\") " pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"
Sep 16 14:28:43.759438 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:43.759304    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cfcefbe-9baa-4157-93f6-4474dbefd43c-home\") pod \"autores-model-84fdff-kserve-5b5869dff-7ws2c\" (UID: \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\") " pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"
Sep 16 14:28:43.759438 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:43.759374    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cfcefbe-9baa-4157-93f6-4474dbefd43c-tls-certs\") pod \"autores-model-84fdff-kserve-5b5869dff-7ws2c\" (UID: \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\") " pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"
Sep 16 14:28:43.759779 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:43.759711    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cfcefbe-9baa-4157-93f6-4474dbefd43c-tmp-dir\") pod \"autores-model-84fdff-kserve-5b5869dff-7ws2c\" (UID: \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\") " pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"
Sep 16 14:28:43.759779 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:43.759758    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cfcefbe-9baa-4157-93f6-4474dbefd43c-model-cache\") pod \"autores-model-84fdff-kserve-5b5869dff-7ws2c\" (UID: \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\") " pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"
Sep 16 14:28:43.760125 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:43.759852    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cfcefbe-9baa-4157-93f6-4474dbefd43c-home\") pod \"autores-model-84fdff-kserve-5b5869dff-7ws2c\" (UID: \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\") " pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"
Sep 16 14:28:43.761712 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:43.761691    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cfcefbe-9baa-4157-93f6-4474dbefd43c-dshm\") pod \"autores-model-84fdff-kserve-5b5869dff-7ws2c\" (UID: \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\") " pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"
Sep 16 14:28:43.762002 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:43.761986    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cfcefbe-9baa-4157-93f6-4474dbefd43c-tls-certs\") pod \"autores-model-84fdff-kserve-5b5869dff-7ws2c\" (UID: \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\") " pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"
Sep 16 14:28:43.768887 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:43.768854    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qltd6\" (UniqueName: \"kubernetes.io/projected/3cfcefbe-9baa-4157-93f6-4474dbefd43c-kube-api-access-qltd6\") pod \"autores-model-84fdff-kserve-5b5869dff-7ws2c\" (UID: \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\") " pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"
Sep 16 14:28:43.811534 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:43.811488    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4" event={"ID":"3cb25e4e-cf32-46b1-9a32-fc0be171d2f4","Type":"ContainerStarted","Data":"79a1aa0124249b2545027c2f07c118371514a126ceca3908cb180399cc1d2360"}
Sep 16 14:28:43.824860 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:43.824822    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"
Sep 16 14:28:43.838961 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:43.838897    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4" podStartSLOduration=1.838873735 podStartE2EDuration="1.838873735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:28:43.835553269 +0000 UTC m=+872.439178313" watchObservedRunningTime="2026-09-16 14:28:43.838873735 +0000 UTC m=+872.442498771"
Sep 16 14:28:44.111236 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:44.111207    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"]
Sep 16 14:28:44.111940 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:28:44.111915    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3cfcefbe_9baa_4157_93f6_4474dbefd43c.slice/crio-0cbd59e13c5dd983d48f961a983488149c026cd16cb619de1c3b488dc459133e WatchSource:0}: Error finding container 0cbd59e13c5dd983d48f961a983488149c026cd16cb619de1c3b488dc459133e: Status 404 returned error can't find the container with id 0cbd59e13c5dd983d48f961a983488149c026cd16cb619de1c3b488dc459133e
Sep 16 14:28:44.818444 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:44.818384    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c" event={"ID":"3cfcefbe-9baa-4157-93f6-4474dbefd43c","Type":"ContainerStarted","Data":"5b64ebca6219b2e13b96f45449a0a91f1594531a1d79a383ee9411bba8553414"}
Sep 16 14:28:44.818444 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:44.818450    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c" event={"ID":"3cfcefbe-9baa-4157-93f6-4474dbefd43c","Type":"ContainerStarted","Data":"0cbd59e13c5dd983d48f961a983488149c026cd16cb619de1c3b488dc459133e"}
Sep 16 14:28:44.818871 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:44.818616    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4"
Sep 16 14:28:44.854735 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:44.854674    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c" podStartSLOduration=1.854654985 podStartE2EDuration="1.854654985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:28:44.852308537 +0000 UTC m=+873.455933571" watchObservedRunningTime="2026-09-16 14:28:44.854654985 +0000 UTC m=+873.458280079"
Sep 16 14:28:53.814374 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:53.814337    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5317692d-5f79588665-rdztg"
Sep 16 14:28:53.825643 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:53.825614    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"
Sep 16 14:28:53.825780 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:53.825763    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"
Sep 16 14:28:53.838190 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:53.838163    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"
Sep 16 14:28:53.883747 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:53.883715    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"
Sep 16 14:28:55.826968 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:28:55.826938    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4"
Sep 16 14:29:08.724949 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:08.724908    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5317692d-mbjbh"]
Sep 16 14:29:08.728612 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:08.728585    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5317692d-mbjbh"
Sep 16 14:29:08.731323 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:08.731298    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:29:08.731509 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:08.731419    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4jf\""
Sep 16 14:29:08.738163 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:08.738130    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5317692d-mbjbh"]
Sep 16 14:29:08.783778 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:08.783739    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70cc7a48-e1a7-4826-9b80-e89d2c7ac08a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5317692d-mbjbh\" (UID: \"70cc7a48-e1a7-4826-9b80-e89d2c7ac08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5317692d-mbjbh"
Sep 16 14:29:08.884855 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:08.884811    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70cc7a48-e1a7-4826-9b80-e89d2c7ac08a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5317692d-mbjbh\" (UID: \"70cc7a48-e1a7-4826-9b80-e89d2c7ac08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5317692d-mbjbh"
Sep 16 14:29:08.885498 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:08.885479    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70cc7a48-e1a7-4826-9b80-e89d2c7ac08a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5317692d-mbjbh\" (UID: \"70cc7a48-e1a7-4826-9b80-e89d2c7ac08a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5317692d-mbjbh"
Sep 16 14:29:09.041253 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:09.041161    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5317692d-mbjbh"
Sep 16 14:29:09.321633 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:09.321587    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5317692d-mbjbh"]
Sep 16 14:29:09.323073 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:29:09.323047    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70cc7a48_e1a7_4826_9b80_e89d2c7ac08a.slice/crio-0b585864d3b137936e69890759da5b3d8eabae4f9a0539d868d3a7799adb000d WatchSource:0}: Error finding container 0b585864d3b137936e69890759da5b3d8eabae4f9a0539d868d3a7799adb000d: Status 404 returned error can't find the container with id 0b585864d3b137936e69890759da5b3d8eabae4f9a0539d868d3a7799adb000d
Sep 16 14:29:09.956016 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:09.955974    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5317692d-mbjbh" event={"ID":"70cc7a48-e1a7-4826-9b80-e89d2c7ac08a","Type":"ContainerStarted","Data":"0b585864d3b137936e69890759da5b3d8eabae4f9a0539d868d3a7799adb000d"}
Sep 16 14:29:11.969489 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:11.969441    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5317692d-mbjbh" event={"ID":"70cc7a48-e1a7-4826-9b80-e89d2c7ac08a","Type":"ContainerStarted","Data":"4611c4f2c63fcdc46e8d649b1e763c29f44e93171e73f95ffee0d33193ccdd3b"}
Sep 16 14:29:11.998407 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:11.998334    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5317692d-mbjbh" podStartSLOduration=1.700391884 podStartE2EDuration="3.998312106s" totalImagesPullingTime="2.297920222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:29:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:29:11.997489597 +0000 UTC m=+900.601114629" watchObservedRunningTime="2026-09-16 14:29:11.998312106 +0000 UTC m=+900.601937141"
Sep 16 14:29:12.977125 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:12.977083    2587 generic.go:356] "Generic (PLEG): container finished" podID="70cc7a48-e1a7-4826-9b80-e89d2c7ac08a" containerID="4611c4f2c63fcdc46e8d649b1e763c29f44e93171e73f95ffee0d33193ccdd3b" exitCode=0
Sep 16 14:29:12.979467 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:12.977128    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5317692d-mbjbh" event={"ID":"70cc7a48-e1a7-4826-9b80-e89d2c7ac08a","Type":"ContainerDied","Data":"4611c4f2c63fcdc46e8d649b1e763c29f44e93171e73f95ffee0d33193ccdd3b"}
Sep 16 14:29:14.126803 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:14.126780    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5317692d-mbjbh"
Sep 16 14:29:14.238989 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:14.238902    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70cc7a48-e1a7-4826-9b80-e89d2c7ac08a-maas-api-service-ca\") pod \"70cc7a48-e1a7-4826-9b80-e89d2c7ac08a\" (UID: \"70cc7a48-e1a7-4826-9b80-e89d2c7ac08a\") "
Sep 16 14:29:14.239294 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:14.239267    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70cc7a48-e1a7-4826-9b80-e89d2c7ac08a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "70cc7a48-e1a7-4826-9b80-e89d2c7ac08a" (UID: "70cc7a48-e1a7-4826-9b80-e89d2c7ac08a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:29:14.339933 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:14.339899    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70cc7a48-e1a7-4826-9b80-e89d2c7ac08a-maas-api-service-ca\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:29:14.987854 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:14.987817    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5317692d-mbjbh" event={"ID":"70cc7a48-e1a7-4826-9b80-e89d2c7ac08a","Type":"ContainerDied","Data":"0b585864d3b137936e69890759da5b3d8eabae4f9a0539d868d3a7799adb000d"}
Sep 16 14:29:14.988026 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:14.987869    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b585864d3b137936e69890759da5b3d8eabae4f9a0539d868d3a7799adb000d"
Sep 16 14:29:14.988026 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:14.987830    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5317692d-mbjbh"
Sep 16 14:29:17.477875 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:17.477838    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84fdff-dvc5g"]
Sep 16 14:29:17.478272 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:17.478259    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70cc7a48-e1a7-4826-9b80-e89d2c7ac08a" containerName="revoke-keys"
Sep 16 14:29:17.478328 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:17.478273    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="70cc7a48-e1a7-4826-9b80-e89d2c7ac08a" containerName="revoke-keys"
Sep 16 14:29:17.478364 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:17.478354    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="70cc7a48-e1a7-4826-9b80-e89d2c7ac08a" containerName="revoke-keys"
Sep 16 14:29:17.481639 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:17.481624    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84fdff-dvc5g"
Sep 16 14:29:17.485069 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:17.485044    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:29:17.485202 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:17.485044    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4jf\""
Sep 16 14:29:17.503773 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:17.503739    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84fdff-dvc5g"]
Sep 16 14:29:17.569142 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:17.569103    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/695de043-1f9e-4b15-a7af-8c7fa0f56e25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-84fdff-dvc5g\" (UID: \"695de043-1f9e-4b15-a7af-8c7fa0f56e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84fdff-dvc5g"
Sep 16 14:29:17.670554 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:17.670520    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/695de043-1f9e-4b15-a7af-8c7fa0f56e25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-84fdff-dvc5g\" (UID: \"695de043-1f9e-4b15-a7af-8c7fa0f56e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84fdff-dvc5g"
Sep 16 14:29:17.671191 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:17.671161    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/695de043-1f9e-4b15-a7af-8c7fa0f56e25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-84fdff-dvc5g\" (UID: \"695de043-1f9e-4b15-a7af-8c7fa0f56e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84fdff-dvc5g"
Sep 16 14:29:17.791621 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:17.791522    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84fdff-dvc5g"
Sep 16 14:29:18.082771 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:18.082683    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84fdff-dvc5g"]
Sep 16 14:29:18.090342 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:29:18.090310    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695de043_1f9e_4b15_a7af_8c7fa0f56e25.slice/crio-18809bbf14485e94948875618ccaf530657bae432fa8aa5e9d8c7aaa3ced7ff9 WatchSource:0}: Error finding container 18809bbf14485e94948875618ccaf530657bae432fa8aa5e9d8c7aaa3ced7ff9: Status 404 returned error can't find the container with id 18809bbf14485e94948875618ccaf530657bae432fa8aa5e9d8c7aaa3ced7ff9
Sep 16 14:29:19.011439 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:19.011406    2587 generic.go:356] "Generic (PLEG): container finished" podID="695de043-1f9e-4b15-a7af-8c7fa0f56e25" containerID="76c9b2587d03b96653af49aa716d5499c076c0864e77eb5f0003bf635bce745b" exitCode=0
Sep 16 14:29:19.011811 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:19.011486    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84fdff-dvc5g" event={"ID":"695de043-1f9e-4b15-a7af-8c7fa0f56e25","Type":"ContainerDied","Data":"76c9b2587d03b96653af49aa716d5499c076c0864e77eb5f0003bf635bce745b"}
Sep 16 14:29:19.011811 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:19.011541    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84fdff-dvc5g" event={"ID":"695de043-1f9e-4b15-a7af-8c7fa0f56e25","Type":"ContainerStarted","Data":"18809bbf14485e94948875618ccaf530657bae432fa8aa5e9d8c7aaa3ced7ff9"}
Sep 16 14:29:20.159158 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:20.159131    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84fdff-dvc5g"
Sep 16 14:29:20.298746 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:20.298653    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/695de043-1f9e-4b15-a7af-8c7fa0f56e25-maas-api-service-ca\") pod \"695de043-1f9e-4b15-a7af-8c7fa0f56e25\" (UID: \"695de043-1f9e-4b15-a7af-8c7fa0f56e25\") "
Sep 16 14:29:20.299014 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:20.298992    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695de043-1f9e-4b15-a7af-8c7fa0f56e25-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "695de043-1f9e-4b15-a7af-8c7fa0f56e25" (UID: "695de043-1f9e-4b15-a7af-8c7fa0f56e25"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:29:20.399879 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:20.399840    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/695de043-1f9e-4b15-a7af-8c7fa0f56e25-maas-api-service-ca\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:29:21.023632 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:21.023601    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84fdff-dvc5g"
Sep 16 14:29:21.023632 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:21.023627    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84fdff-dvc5g" event={"ID":"695de043-1f9e-4b15-a7af-8c7fa0f56e25","Type":"ContainerDied","Data":"18809bbf14485e94948875618ccaf530657bae432fa8aa5e9d8c7aaa3ced7ff9"}
Sep 16 14:29:21.023846 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:21.023673    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18809bbf14485e94948875618ccaf530657bae432fa8aa5e9d8c7aaa3ced7ff9"
Sep 16 14:29:22.029358 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:22.029320    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5317692d-5f79588665-rdztg"]
Sep 16 14:29:22.030174 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:22.030115    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5317692d-5f79588665-rdztg" podUID="e3a8e0cb-1fe6-4cd3-a8db-810e14447447" containerName="payload-processing" containerID="cri-o://d251d0c54ba01d3c5f503186833c792995bc2a6a5796613318abb6950ac17512" gracePeriod=30
Sep 16 14:29:22.043210 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:22.043181    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4"]
Sep 16 14:29:22.043665 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:22.043632    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4" podUID="3cb25e4e-cf32-46b1-9a32-fc0be171d2f4" containerName="payload-pre-processing" containerID="cri-o://79a1aa0124249b2545027c2f07c118371514a126ceca3908cb180399cc1d2360" gracePeriod=30
Sep 16 14:29:22.304918 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:22.304891    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5317692d-5f79588665-rdztg"
Sep 16 14:29:22.335254 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:22.335229    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4"
Sep 16 14:29:22.419730 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:22.419692    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3a8e0cb-1fe6-4cd3-a8db-810e14447447-plugins-config-volume\") pod \"e3a8e0cb-1fe6-4cd3-a8db-810e14447447\" (UID: \"e3a8e0cb-1fe6-4cd3-a8db-810e14447447\") "
Sep 16 14:29:22.419925 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:22.419751    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb25e4e-cf32-46b1-9a32-fc0be171d2f4-plugins-config-volume\") pod \"3cb25e4e-cf32-46b1-9a32-fc0be171d2f4\" (UID: \"3cb25e4e-cf32-46b1-9a32-fc0be171d2f4\") "
Sep 16 14:29:22.419925 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:22.419791    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwbpq\" (UniqueName: \"kubernetes.io/projected/e3a8e0cb-1fe6-4cd3-a8db-810e14447447-kube-api-access-xwbpq\") pod \"e3a8e0cb-1fe6-4cd3-a8db-810e14447447\" (UID: \"e3a8e0cb-1fe6-4cd3-a8db-810e14447447\") "
Sep 16 14:29:22.419925 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:22.419879    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htnmp\" (UniqueName: \"kubernetes.io/projected/3cb25e4e-cf32-46b1-9a32-fc0be171d2f4-kube-api-access-htnmp\") pod \"3cb25e4e-cf32-46b1-9a32-fc0be171d2f4\" (UID: \"3cb25e4e-cf32-46b1-9a32-fc0be171d2f4\") "
Sep 16 14:29:22.420311 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:22.420188    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3a8e0cb-1fe6-4cd3-a8db-810e14447447-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e3a8e0cb-1fe6-4cd3-a8db-810e14447447" (UID: "e3a8e0cb-1fe6-4cd3-a8db-810e14447447"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:29:22.420311 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:22.420272    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb25e4e-cf32-46b1-9a32-fc0be171d2f4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3cb25e4e-cf32-46b1-9a32-fc0be171d2f4" (UID: "3cb25e4e-cf32-46b1-9a32-fc0be171d2f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:29:22.422574 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:22.422547    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb25e4e-cf32-46b1-9a32-fc0be171d2f4-kube-api-access-htnmp" (OuterVolumeSpecName: "kube-api-access-htnmp") pod "3cb25e4e-cf32-46b1-9a32-fc0be171d2f4" (UID: "3cb25e4e-cf32-46b1-9a32-fc0be171d2f4"). InnerVolumeSpecName "kube-api-access-htnmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:29:22.422682 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:22.422593    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3a8e0cb-1fe6-4cd3-a8db-810e14447447-kube-api-access-xwbpq" (OuterVolumeSpecName: "kube-api-access-xwbpq") pod "e3a8e0cb-1fe6-4cd3-a8db-810e14447447" (UID: "e3a8e0cb-1fe6-4cd3-a8db-810e14447447"). InnerVolumeSpecName "kube-api-access-xwbpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:29:22.517033 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:22.516997    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5317692d-mbjbh"]
Sep 16 14:29:22.521207 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:22.521144    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwbpq\" (UniqueName: \"kubernetes.io/projected/e3a8e0cb-1fe6-4cd3-a8db-810e14447447-kube-api-access-xwbpq\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:29:22.521207 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:22.521166    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htnmp\" (UniqueName: \"kubernetes.io/projected/3cb25e4e-cf32-46b1-9a32-fc0be171d2f4-kube-api-access-htnmp\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:29:22.521207 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:22.521176    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3a8e0cb-1fe6-4cd3-a8db-810e14447447-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:29:22.521207 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:22.521186    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb25e4e-cf32-46b1-9a32-fc0be171d2f4-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:29:22.524918 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:22.524887    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5317692d-mbjbh"]
Sep 16 14:29:23.037967 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:23.037931    2587 generic.go:356] "Generic (PLEG): container finished" podID="3cb25e4e-cf32-46b1-9a32-fc0be171d2f4" containerID="79a1aa0124249b2545027c2f07c118371514a126ceca3908cb180399cc1d2360" exitCode=0
Sep 16 14:29:23.038456 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:23.037979    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4"
Sep 16 14:29:23.038456 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:23.038017    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4" event={"ID":"3cb25e4e-cf32-46b1-9a32-fc0be171d2f4","Type":"ContainerDied","Data":"79a1aa0124249b2545027c2f07c118371514a126ceca3908cb180399cc1d2360"}
Sep 16 14:29:23.038456 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:23.038085    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4" event={"ID":"3cb25e4e-cf32-46b1-9a32-fc0be171d2f4","Type":"ContainerDied","Data":"a34f028b8b854c76decd972e596bce1690c647e500b6f26f4e47dbfebcfec66b"}
Sep 16 14:29:23.038456 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:23.038112    2587 scope.go:117] "RemoveContainer" containerID="79a1aa0124249b2545027c2f07c118371514a126ceca3908cb180399cc1d2360"
Sep 16 14:29:23.039663 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:23.039641    2587 generic.go:356] "Generic (PLEG): container finished" podID="e3a8e0cb-1fe6-4cd3-a8db-810e14447447" containerID="d251d0c54ba01d3c5f503186833c792995bc2a6a5796613318abb6950ac17512" exitCode=0
Sep 16 14:29:23.039751 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:23.039689    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5317692d-5f79588665-rdztg" event={"ID":"e3a8e0cb-1fe6-4cd3-a8db-810e14447447","Type":"ContainerDied","Data":"d251d0c54ba01d3c5f503186833c792995bc2a6a5796613318abb6950ac17512"}
Sep 16 14:29:23.039751 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:23.039695    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5317692d-5f79588665-rdztg"
Sep 16 14:29:23.039831 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:23.039728    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5317692d-5f79588665-rdztg" event={"ID":"e3a8e0cb-1fe6-4cd3-a8db-810e14447447","Type":"ContainerDied","Data":"a836b270de41c81454eda70658e1a53adb95d86ad528a17c47cd5465e48e973d"}
Sep 16 14:29:23.056869 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:23.056845    2587 scope.go:117] "RemoveContainer" containerID="79a1aa0124249b2545027c2f07c118371514a126ceca3908cb180399cc1d2360"
Sep 16 14:29:23.057210 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:29:23.057183    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79a1aa0124249b2545027c2f07c118371514a126ceca3908cb180399cc1d2360\": container with ID starting with 79a1aa0124249b2545027c2f07c118371514a126ceca3908cb180399cc1d2360 not found: ID does not exist" containerID="79a1aa0124249b2545027c2f07c118371514a126ceca3908cb180399cc1d2360"
Sep 16 14:29:23.057304 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:23.057218    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79a1aa0124249b2545027c2f07c118371514a126ceca3908cb180399cc1d2360"} err="failed to get container status \"79a1aa0124249b2545027c2f07c118371514a126ceca3908cb180399cc1d2360\": rpc error: code = NotFound desc = could not find container \"79a1aa0124249b2545027c2f07c118371514a126ceca3908cb180399cc1d2360\": container with ID starting with 79a1aa0124249b2545027c2f07c118371514a126ceca3908cb180399cc1d2360 not found: ID does not exist"
Sep 16 14:29:23.057304 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:23.057237    2587 scope.go:117] "RemoveContainer" containerID="d251d0c54ba01d3c5f503186833c792995bc2a6a5796613318abb6950ac17512"
Sep 16 14:29:23.066188 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:23.066167    2587 scope.go:117] "RemoveContainer" containerID="d251d0c54ba01d3c5f503186833c792995bc2a6a5796613318abb6950ac17512"
Sep 16 14:29:23.066536 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:29:23.066516    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d251d0c54ba01d3c5f503186833c792995bc2a6a5796613318abb6950ac17512\": container with ID starting with d251d0c54ba01d3c5f503186833c792995bc2a6a5796613318abb6950ac17512 not found: ID does not exist" containerID="d251d0c54ba01d3c5f503186833c792995bc2a6a5796613318abb6950ac17512"
Sep 16 14:29:23.066588 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:23.066544    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d251d0c54ba01d3c5f503186833c792995bc2a6a5796613318abb6950ac17512"} err="failed to get container status \"d251d0c54ba01d3c5f503186833c792995bc2a6a5796613318abb6950ac17512\": rpc error: code = NotFound desc = could not find container \"d251d0c54ba01d3c5f503186833c792995bc2a6a5796613318abb6950ac17512\": container with ID starting with d251d0c54ba01d3c5f503186833c792995bc2a6a5796613318abb6950ac17512 not found: ID does not exist"
Sep 16 14:29:23.084231 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:23.084193    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5317692d-5f79588665-rdztg"]
Sep 16 14:29:23.091117 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:23.091085    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5317692d-5f79588665-rdztg"]
Sep 16 14:29:23.110210 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:23.110175    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4"]
Sep 16 14:29:23.122103 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:23.122057    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5317692d-59df4fcd5d-78wn4"]
Sep 16 14:29:23.903477 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:23.903443    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cb25e4e-cf32-46b1-9a32-fc0be171d2f4" path="/var/lib/kubelet/pods/3cb25e4e-cf32-46b1-9a32-fc0be171d2f4/volumes"
Sep 16 14:29:23.903803 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:23.903790    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70cc7a48-e1a7-4826-9b80-e89d2c7ac08a" path="/var/lib/kubelet/pods/70cc7a48-e1a7-4826-9b80-e89d2c7ac08a/volumes"
Sep 16 14:29:23.904108 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:23.904097    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3a8e0cb-1fe6-4cd3-a8db-810e14447447" path="/var/lib/kubelet/pods/e3a8e0cb-1fe6-4cd3-a8db-810e14447447/volumes"
Sep 16 14:29:29.602793 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.602759    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4"]
Sep 16 14:29:29.603199 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.603185    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3a8e0cb-1fe6-4cd3-a8db-810e14447447" containerName="payload-processing"
Sep 16 14:29:29.603199 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.603200    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3a8e0cb-1fe6-4cd3-a8db-810e14447447" containerName="payload-processing"
Sep 16 14:29:29.603275 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.603216    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cb25e4e-cf32-46b1-9a32-fc0be171d2f4" containerName="payload-pre-processing"
Sep 16 14:29:29.603275 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.603222    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cb25e4e-cf32-46b1-9a32-fc0be171d2f4" containerName="payload-pre-processing"
Sep 16 14:29:29.603275 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.603228    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="695de043-1f9e-4b15-a7af-8c7fa0f56e25" containerName="revoke-keys"
Sep 16 14:29:29.603275 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.603233    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="695de043-1f9e-4b15-a7af-8c7fa0f56e25" containerName="revoke-keys"
Sep 16 14:29:29.603433 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.603298    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3a8e0cb-1fe6-4cd3-a8db-810e14447447" containerName="payload-processing"
Sep 16 14:29:29.603433 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.603308    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cb25e4e-cf32-46b1-9a32-fc0be171d2f4" containerName="payload-pre-processing"
Sep 16 14:29:29.603433 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.603314    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="695de043-1f9e-4b15-a7af-8c7fa0f56e25" containerName="revoke-keys"
Sep 16 14:29:29.607785 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.607766    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4"
Sep 16 14:29:29.610696 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.610663    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-dfce3e96-dockercfg-sn9jl\""
Sep 16 14:29:29.610696 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.610674    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-dfce3e96\""
Sep 16 14:29:29.619115 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.619090    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4"]
Sep 16 14:29:29.687115 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.687082    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz"]
Sep 16 14:29:29.690256 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.690230    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7fqr\" (UniqueName: \"kubernetes.io/projected/2877e137-6e33-4a4a-9d3a-fcfd87b46174-kube-api-access-b7fqr\") pod \"payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4\" (UID: \"2877e137-6e33-4a4a-9d3a-fcfd87b46174\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4"
Sep 16 14:29:29.690415 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.690319    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2877e137-6e33-4a4a-9d3a-fcfd87b46174-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4\" (UID: \"2877e137-6e33-4a4a-9d3a-fcfd87b46174\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4"
Sep 16 14:29:29.690682 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.690664    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz"
Sep 16 14:29:29.704283 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.704256    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz"]
Sep 16 14:29:29.791745 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.791711    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f456875b-dd77-4e09-9937-fc413c8cdb00-plugins-config-volume\") pod \"payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz\" (UID: \"f456875b-dd77-4e09-9937-fc413c8cdb00\") " pod="openshift-ingress/payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz"
Sep 16 14:29:29.791905 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.791762    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxnlf\" (UniqueName: \"kubernetes.io/projected/f456875b-dd77-4e09-9937-fc413c8cdb00-kube-api-access-qxnlf\") pod \"payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz\" (UID: \"f456875b-dd77-4e09-9937-fc413c8cdb00\") " pod="openshift-ingress/payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz"
Sep 16 14:29:29.791905 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.791817    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7fqr\" (UniqueName: \"kubernetes.io/projected/2877e137-6e33-4a4a-9d3a-fcfd87b46174-kube-api-access-b7fqr\") pod \"payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4\" (UID: \"2877e137-6e33-4a4a-9d3a-fcfd87b46174\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4"
Sep 16 14:29:29.791905 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.791886    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2877e137-6e33-4a4a-9d3a-fcfd87b46174-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4\" (UID: \"2877e137-6e33-4a4a-9d3a-fcfd87b46174\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4"
Sep 16 14:29:29.792550 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.792533    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2877e137-6e33-4a4a-9d3a-fcfd87b46174-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4\" (UID: \"2877e137-6e33-4a4a-9d3a-fcfd87b46174\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4"
Sep 16 14:29:29.802275 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.802248    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7fqr\" (UniqueName: \"kubernetes.io/projected/2877e137-6e33-4a4a-9d3a-fcfd87b46174-kube-api-access-b7fqr\") pod \"payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4\" (UID: \"2877e137-6e33-4a4a-9d3a-fcfd87b46174\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4"
Sep 16 14:29:29.857838 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.855502    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-84fdff-bf594d749-qtxbk"]
Sep 16 14:29:29.858933 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.858697    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9"]
Sep 16 14:29:29.858933 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.858946    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-84fdff-bf594d749-qtxbk" podUID="2077db6c-4fbc-45eb-98e2-82a4837e9e7a" containerName="payload-processing" containerID="cri-o://dc46fcb121e4ded0b7e1c18a36192b58c049cc3364dc462e7a5a6d40266eb262" gracePeriod=30
Sep 16 14:29:29.860507 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.859233    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9" podUID="8d432ee6-4244-44ee-9f22-226f1837666d" containerName="payload-pre-processing" containerID="cri-o://3ac13f142fb1ee63ec03b49dd3772016540523a178f9330e43495dbce4e41803" gracePeriod=30
Sep 16 14:29:29.892450 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.892419    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f456875b-dd77-4e09-9937-fc413c8cdb00-plugins-config-volume\") pod \"payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz\" (UID: \"f456875b-dd77-4e09-9937-fc413c8cdb00\") " pod="openshift-ingress/payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz"
Sep 16 14:29:29.892606 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.892535    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxnlf\" (UniqueName: \"kubernetes.io/projected/f456875b-dd77-4e09-9937-fc413c8cdb00-kube-api-access-qxnlf\") pod \"payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz\" (UID: \"f456875b-dd77-4e09-9937-fc413c8cdb00\") " pod="openshift-ingress/payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz"
Sep 16 14:29:29.893356 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.893320    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f456875b-dd77-4e09-9937-fc413c8cdb00-plugins-config-volume\") pod \"payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz\" (UID: \"f456875b-dd77-4e09-9937-fc413c8cdb00\") " pod="openshift-ingress/payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz"
Sep 16 14:29:29.904097 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.904064    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxnlf\" (UniqueName: \"kubernetes.io/projected/f456875b-dd77-4e09-9937-fc413c8cdb00-kube-api-access-qxnlf\") pod \"payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz\" (UID: \"f456875b-dd77-4e09-9937-fc413c8cdb00\") " pod="openshift-ingress/payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz"
Sep 16 14:29:29.918456 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:29.918422    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4"
Sep 16 14:29:30.001197 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:30.001159    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz"
Sep 16 14:29:30.069594 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:30.067612    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84fdff-dvc5g"]
Sep 16 14:29:30.071672 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:30.071630    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84fdff-dvc5g"]
Sep 16 14:29:30.083383 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:30.083360    2587 generic.go:356] "Generic (PLEG): container finished" podID="2077db6c-4fbc-45eb-98e2-82a4837e9e7a" containerID="dc46fcb121e4ded0b7e1c18a36192b58c049cc3364dc462e7a5a6d40266eb262" exitCode=0
Sep 16 14:29:30.083514 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:30.083423    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-84fdff-bf594d749-qtxbk" event={"ID":"2077db6c-4fbc-45eb-98e2-82a4837e9e7a","Type":"ContainerDied","Data":"dc46fcb121e4ded0b7e1c18a36192b58c049cc3364dc462e7a5a6d40266eb262"}
Sep 16 14:29:30.085519 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:30.085494    2587 generic.go:356] "Generic (PLEG): container finished" podID="8d432ee6-4244-44ee-9f22-226f1837666d" containerID="3ac13f142fb1ee63ec03b49dd3772016540523a178f9330e43495dbce4e41803" exitCode=0
Sep 16 14:29:30.085806 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:30.085770    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9" event={"ID":"8d432ee6-4244-44ee-9f22-226f1837666d","Type":"ContainerDied","Data":"3ac13f142fb1ee63ec03b49dd3772016540523a178f9330e43495dbce4e41803"}
Sep 16 14:29:30.173636 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:30.173608    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-84fdff-bf594d749-qtxbk"
Sep 16 14:29:30.175984 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:30.175961    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9"
Sep 16 14:29:30.202699 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:30.202673    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4"]
Sep 16 14:29:30.204727 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:29:30.204704    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2877e137_6e33_4a4a_9d3a_fcfd87b46174.slice/crio-7e8c726b9900a26d17526eaa9a9cd36a44a705067af1134236c027d8c60109d4 WatchSource:0}: Error finding container 7e8c726b9900a26d17526eaa9a9cd36a44a705067af1134236c027d8c60109d4: Status 404 returned error can't find the container with id 7e8c726b9900a26d17526eaa9a9cd36a44a705067af1134236c027d8c60109d4
Sep 16 14:29:30.296654 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:30.296626    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2077db6c-4fbc-45eb-98e2-82a4837e9e7a-plugins-config-volume\") pod \"2077db6c-4fbc-45eb-98e2-82a4837e9e7a\" (UID: \"2077db6c-4fbc-45eb-98e2-82a4837e9e7a\") "
Sep 16 14:29:30.296807 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:30.296751    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfdzm\" (UniqueName: \"kubernetes.io/projected/8d432ee6-4244-44ee-9f22-226f1837666d-kube-api-access-jfdzm\") pod \"8d432ee6-4244-44ee-9f22-226f1837666d\" (UID: \"8d432ee6-4244-44ee-9f22-226f1837666d\") "
Sep 16 14:29:30.296807 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:30.296783    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmdgf\" (UniqueName: \"kubernetes.io/projected/2077db6c-4fbc-45eb-98e2-82a4837e9e7a-kube-api-access-jmdgf\") pod \"2077db6c-4fbc-45eb-98e2-82a4837e9e7a\" (UID: \"2077db6c-4fbc-45eb-98e2-82a4837e9e7a\") "
Sep 16 14:29:30.296887 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:30.296812    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d432ee6-4244-44ee-9f22-226f1837666d-plugins-config-volume\") pod \"8d432ee6-4244-44ee-9f22-226f1837666d\" (UID: \"8d432ee6-4244-44ee-9f22-226f1837666d\") "
Sep 16 14:29:30.297167 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:30.297041    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2077db6c-4fbc-45eb-98e2-82a4837e9e7a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2077db6c-4fbc-45eb-98e2-82a4837e9e7a" (UID: "2077db6c-4fbc-45eb-98e2-82a4837e9e7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:29:30.297278 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:30.297217    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d432ee6-4244-44ee-9f22-226f1837666d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8d432ee6-4244-44ee-9f22-226f1837666d" (UID: "8d432ee6-4244-44ee-9f22-226f1837666d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:29:30.298990 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:30.298960    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2077db6c-4fbc-45eb-98e2-82a4837e9e7a-kube-api-access-jmdgf" (OuterVolumeSpecName: "kube-api-access-jmdgf") pod "2077db6c-4fbc-45eb-98e2-82a4837e9e7a" (UID: "2077db6c-4fbc-45eb-98e2-82a4837e9e7a"). InnerVolumeSpecName "kube-api-access-jmdgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:29:30.299319 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:30.299305    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d432ee6-4244-44ee-9f22-226f1837666d-kube-api-access-jfdzm" (OuterVolumeSpecName: "kube-api-access-jfdzm") pod "8d432ee6-4244-44ee-9f22-226f1837666d" (UID: "8d432ee6-4244-44ee-9f22-226f1837666d"). InnerVolumeSpecName "kube-api-access-jfdzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:29:30.315048 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:30.315027    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz"]
Sep 16 14:29:30.316760 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:29:30.316735    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf456875b_dd77_4e09_9937_fc413c8cdb00.slice/crio-a0901806ef7b5d7709223b3a0981ec94811c56389c9df94ac1ef5d9e3a9b66ed WatchSource:0}: Error finding container a0901806ef7b5d7709223b3a0981ec94811c56389c9df94ac1ef5d9e3a9b66ed: Status 404 returned error can't find the container with id a0901806ef7b5d7709223b3a0981ec94811c56389c9df94ac1ef5d9e3a9b66ed
Sep 16 14:29:30.398276 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:30.398242    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d432ee6-4244-44ee-9f22-226f1837666d-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:29:30.398429 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:30.398278    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2077db6c-4fbc-45eb-98e2-82a4837e9e7a-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:29:30.398429 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:30.398305    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfdzm\" (UniqueName: \"kubernetes.io/projected/8d432ee6-4244-44ee-9f22-226f1837666d-kube-api-access-jfdzm\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:29:30.398429 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:30.398319    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmdgf\" (UniqueName: \"kubernetes.io/projected/2077db6c-4fbc-45eb-98e2-82a4837e9e7a-kube-api-access-jmdgf\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:29:31.094862 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:31.094756    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-84fdff-bf594d749-qtxbk" event={"ID":"2077db6c-4fbc-45eb-98e2-82a4837e9e7a","Type":"ContainerDied","Data":"a1550fa49632f7de9f8c44581cd57303b1f96c33953cc01e9fb8707d1c0c87ec"}
Sep 16 14:29:31.094862 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:31.094784    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-84fdff-bf594d749-qtxbk"
Sep 16 14:29:31.094862 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:31.094838    2587 scope.go:117] "RemoveContainer" containerID="dc46fcb121e4ded0b7e1c18a36192b58c049cc3364dc462e7a5a6d40266eb262"
Sep 16 14:29:31.096537 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:31.096518    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9"
Sep 16 14:29:31.096641 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:31.096547    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9" event={"ID":"8d432ee6-4244-44ee-9f22-226f1837666d","Type":"ContainerDied","Data":"e1e6375ab28cc2842f2bd517aafb31b4017998a7c6889e19c62423256d5ec6c9"}
Sep 16 14:29:31.099351 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:31.098839    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4" event={"ID":"2877e137-6e33-4a4a-9d3a-fcfd87b46174","Type":"ContainerStarted","Data":"5a210e232db79607dad9aded2bd1f13c2d22336cf2499e7b01798c14eff8d7a4"}
Sep 16 14:29:31.099351 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:31.098888    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4" event={"ID":"2877e137-6e33-4a4a-9d3a-fcfd87b46174","Type":"ContainerStarted","Data":"7e8c726b9900a26d17526eaa9a9cd36a44a705067af1134236c027d8c60109d4"}
Sep 16 14:29:31.099351 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:31.099231    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4"
Sep 16 14:29:31.101717 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:31.101594    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz" event={"ID":"f456875b-dd77-4e09-9937-fc413c8cdb00","Type":"ContainerStarted","Data":"209f16e70eb6fede89031b15e268946d8e233d62036eccc60eae8e52f25659f2"}
Sep 16 14:29:31.101717 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:31.101639    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz" event={"ID":"f456875b-dd77-4e09-9937-fc413c8cdb00","Type":"ContainerStarted","Data":"a0901806ef7b5d7709223b3a0981ec94811c56389c9df94ac1ef5d9e3a9b66ed"}
Sep 16 14:29:31.101717 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:31.101683    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz"
Sep 16 14:29:31.107245 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:31.107189    2587 scope.go:117] "RemoveContainer" containerID="3ac13f142fb1ee63ec03b49dd3772016540523a178f9330e43495dbce4e41803"
Sep 16 14:29:31.121277 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:31.121227    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4" podStartSLOduration=2.121207814 podStartE2EDuration="2.121207814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:29:31.119218683 +0000 UTC m=+919.722843716" watchObservedRunningTime="2026-09-16 14:29:31.121207814 +0000 UTC m=+919.724832845"
Sep 16 14:29:31.146219 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:31.146179    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9"]
Sep 16 14:29:31.149219 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:31.149190    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-84fdff-b5c44f5bb-nhpv9"]
Sep 16 14:29:31.160767 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:31.160735    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-84fdff-bf594d749-qtxbk"]
Sep 16 14:29:31.167685 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:31.167655    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-84fdff-bf594d749-qtxbk"]
Sep 16 14:29:31.189747 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:31.189686    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz" podStartSLOduration=2.189667694 podStartE2EDuration="2.189667694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:29:31.188506316 +0000 UTC m=+919.792131348" watchObservedRunningTime="2026-09-16 14:29:31.189667694 +0000 UTC m=+919.793292726"
Sep 16 14:29:31.903321 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:31.903285    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2077db6c-4fbc-45eb-98e2-82a4837e9e7a" path="/var/lib/kubelet/pods/2077db6c-4fbc-45eb-98e2-82a4837e9e7a/volumes"
Sep 16 14:29:31.903670 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:31.903655    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="695de043-1f9e-4b15-a7af-8c7fa0f56e25" path="/var/lib/kubelet/pods/695de043-1f9e-4b15-a7af-8c7fa0f56e25/volumes"
Sep 16 14:29:31.903965 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:31.903952    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d432ee6-4244-44ee-9f22-226f1837666d" path="/var/lib/kubelet/pods/8d432ee6-4244-44ee-9f22-226f1837666d/volumes"
Sep 16 14:29:35.432542 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:35.432504    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"]
Sep 16 14:29:42.112603 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:42.112555    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz"
Sep 16 14:29:42.113020 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:42.112631    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4"
Sep 16 14:29:50.452808 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:50.452757    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c" podUID="3cfcefbe-9baa-4157-93f6-4474dbefd43c" containerName="main" containerID="cri-o://5b64ebca6219b2e13b96f45449a0a91f1594531a1d79a383ee9411bba8553414" gracePeriod=45
Sep 16 14:29:50.703291 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:50.703224    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"
Sep 16 14:29:50.789046 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:50.789007    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cfcefbe-9baa-4157-93f6-4474dbefd43c-tmp-dir\") pod \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\" (UID: \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\") "
Sep 16 14:29:50.789046 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:50.789047    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qltd6\" (UniqueName: \"kubernetes.io/projected/3cfcefbe-9baa-4157-93f6-4474dbefd43c-kube-api-access-qltd6\") pod \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\" (UID: \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\") "
Sep 16 14:29:50.789310 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:50.789067    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cfcefbe-9baa-4157-93f6-4474dbefd43c-home\") pod \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\" (UID: \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\") "
Sep 16 14:29:50.789310 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:50.789180    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cfcefbe-9baa-4157-93f6-4474dbefd43c-model-cache\") pod \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\" (UID: \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\") "
Sep 16 14:29:50.789310 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:50.789215    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cfcefbe-9baa-4157-93f6-4474dbefd43c-tls-certs\") pod \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\" (UID: \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\") "
Sep 16 14:29:50.789310 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:50.789268    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cfcefbe-9baa-4157-93f6-4474dbefd43c-dshm\") pod \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\" (UID: \"3cfcefbe-9baa-4157-93f6-4474dbefd43c\") "
Sep 16 14:29:50.789310 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:50.789283    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cfcefbe-9baa-4157-93f6-4474dbefd43c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3cfcefbe-9baa-4157-93f6-4474dbefd43c" (UID: "3cfcefbe-9baa-4157-93f6-4474dbefd43c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:29:50.789646 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:50.789430    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cfcefbe-9baa-4157-93f6-4474dbefd43c-home" (OuterVolumeSpecName: "home") pod "3cfcefbe-9baa-4157-93f6-4474dbefd43c" (UID: "3cfcefbe-9baa-4157-93f6-4474dbefd43c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:29:50.789646 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:50.789478    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cfcefbe-9baa-4157-93f6-4474dbefd43c-model-cache" (OuterVolumeSpecName: "model-cache") pod "3cfcefbe-9baa-4157-93f6-4474dbefd43c" (UID: "3cfcefbe-9baa-4157-93f6-4474dbefd43c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:29:50.789646 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:50.789567    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cfcefbe-9baa-4157-93f6-4474dbefd43c-tmp-dir\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:29:50.789646 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:50.789587    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cfcefbe-9baa-4157-93f6-4474dbefd43c-home\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:29:50.789646 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:50.789601    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cfcefbe-9baa-4157-93f6-4474dbefd43c-model-cache\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:29:50.791756 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:50.791707    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cfcefbe-9baa-4157-93f6-4474dbefd43c-kube-api-access-qltd6" (OuterVolumeSpecName: "kube-api-access-qltd6") pod "3cfcefbe-9baa-4157-93f6-4474dbefd43c" (UID: "3cfcefbe-9baa-4157-93f6-4474dbefd43c"). InnerVolumeSpecName "kube-api-access-qltd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:29:50.792038 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:50.792019    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cfcefbe-9baa-4157-93f6-4474dbefd43c-dshm" (OuterVolumeSpecName: "dshm") pod "3cfcefbe-9baa-4157-93f6-4474dbefd43c" (UID: "3cfcefbe-9baa-4157-93f6-4474dbefd43c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:29:50.792289 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:50.792273    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cfcefbe-9baa-4157-93f6-4474dbefd43c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3cfcefbe-9baa-4157-93f6-4474dbefd43c" (UID: "3cfcefbe-9baa-4157-93f6-4474dbefd43c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:29:50.890164 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:50.890122    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cfcefbe-9baa-4157-93f6-4474dbefd43c-tls-certs\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:29:50.890164 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:50.890161    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cfcefbe-9baa-4157-93f6-4474dbefd43c-dshm\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:29:50.890446 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:50.890176    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qltd6\" (UniqueName: \"kubernetes.io/projected/3cfcefbe-9baa-4157-93f6-4474dbefd43c-kube-api-access-qltd6\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:29:51.207460 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:51.207426    2587 generic.go:356] "Generic (PLEG): container finished" podID="3cfcefbe-9baa-4157-93f6-4474dbefd43c" containerID="5b64ebca6219b2e13b96f45449a0a91f1594531a1d79a383ee9411bba8553414" exitCode=0
Sep 16 14:29:51.207640 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:51.207507    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"
Sep 16 14:29:51.207640 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:51.207517    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c" event={"ID":"3cfcefbe-9baa-4157-93f6-4474dbefd43c","Type":"ContainerDied","Data":"5b64ebca6219b2e13b96f45449a0a91f1594531a1d79a383ee9411bba8553414"}
Sep 16 14:29:51.207640 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:51.207567    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c" event={"ID":"3cfcefbe-9baa-4157-93f6-4474dbefd43c","Type":"ContainerDied","Data":"0cbd59e13c5dd983d48f961a983488149c026cd16cb619de1c3b488dc459133e"}
Sep 16 14:29:51.207640 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:51.207589    2587 scope.go:117] "RemoveContainer" containerID="5b64ebca6219b2e13b96f45449a0a91f1594531a1d79a383ee9411bba8553414"
Sep 16 14:29:51.217377 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:51.217361    2587 scope.go:117] "RemoveContainer" containerID="5b64ebca6219b2e13b96f45449a0a91f1594531a1d79a383ee9411bba8553414"
Sep 16 14:29:51.217692 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:29:51.217672    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b64ebca6219b2e13b96f45449a0a91f1594531a1d79a383ee9411bba8553414\": container with ID starting with 5b64ebca6219b2e13b96f45449a0a91f1594531a1d79a383ee9411bba8553414 not found: ID does not exist" containerID="5b64ebca6219b2e13b96f45449a0a91f1594531a1d79a383ee9411bba8553414"
Sep 16 14:29:51.217748 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:51.217702    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b64ebca6219b2e13b96f45449a0a91f1594531a1d79a383ee9411bba8553414"} err="failed to get container status \"5b64ebca6219b2e13b96f45449a0a91f1594531a1d79a383ee9411bba8553414\": rpc error: code = NotFound desc = could not find container \"5b64ebca6219b2e13b96f45449a0a91f1594531a1d79a383ee9411bba8553414\": container with ID starting with 5b64ebca6219b2e13b96f45449a0a91f1594531a1d79a383ee9411bba8553414 not found: ID does not exist"
Sep 16 14:29:51.235177 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:51.235144    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"]
Sep 16 14:29:51.242421 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:51.238964    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-84fdff/autores-model-84fdff-kserve-5b5869dff-7ws2c"]
Sep 16 14:29:51.740077 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:51.740027    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfce3e96-tlcz4"]
Sep 16 14:29:51.740693 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:51.740672    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d432ee6-4244-44ee-9f22-226f1837666d" containerName="payload-pre-processing"
Sep 16 14:29:51.740788 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:51.740696    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d432ee6-4244-44ee-9f22-226f1837666d" containerName="payload-pre-processing"
Sep 16 14:29:51.740788 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:51.740712    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2077db6c-4fbc-45eb-98e2-82a4837e9e7a" containerName="payload-processing"
Sep 16 14:29:51.740788 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:51.740722    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="2077db6c-4fbc-45eb-98e2-82a4837e9e7a" containerName="payload-processing"
Sep 16 14:29:51.740788 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:51.740735    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cfcefbe-9baa-4157-93f6-4474dbefd43c" containerName="main"
Sep 16 14:29:51.740788 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:51.740744    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cfcefbe-9baa-4157-93f6-4474dbefd43c" containerName="main"
Sep 16 14:29:51.741032 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:51.740854    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="2077db6c-4fbc-45eb-98e2-82a4837e9e7a" containerName="payload-processing"
Sep 16 14:29:51.741032 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:51.740867    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d432ee6-4244-44ee-9f22-226f1837666d" containerName="payload-pre-processing"
Sep 16 14:29:51.741032 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:51.740876    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cfcefbe-9baa-4157-93f6-4474dbefd43c" containerName="main"
Sep 16 14:29:51.745341 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:51.745315    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfce3e96-tlcz4"
Sep 16 14:29:51.748492 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:51.748457    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4jf\""
Sep 16 14:29:51.748637 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:51.748521    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:29:51.761169 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:51.761139    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfce3e96-tlcz4"]
Sep 16 14:29:51.801084 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:51.801048    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5cdb62-7bd9-4e59-a46b-7ea066190216-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-dfce3e96-tlcz4\" (UID: \"8e5cdb62-7bd9-4e59-a46b-7ea066190216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfce3e96-tlcz4"
Sep 16 14:29:51.902492 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:51.902454    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5cdb62-7bd9-4e59-a46b-7ea066190216-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-dfce3e96-tlcz4\" (UID: \"8e5cdb62-7bd9-4e59-a46b-7ea066190216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfce3e96-tlcz4"
Sep 16 14:29:51.903108 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:51.903088    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5cdb62-7bd9-4e59-a46b-7ea066190216-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-dfce3e96-tlcz4\" (UID: \"8e5cdb62-7bd9-4e59-a46b-7ea066190216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfce3e96-tlcz4"
Sep 16 14:29:51.903569 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:51.903545    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cfcefbe-9baa-4157-93f6-4474dbefd43c" path="/var/lib/kubelet/pods/3cfcefbe-9baa-4157-93f6-4474dbefd43c/volumes"
Sep 16 14:29:52.055548 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:52.055458    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfce3e96-tlcz4"
Sep 16 14:29:52.346202 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:52.346073    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfce3e96-tlcz4"]
Sep 16 14:29:52.348844 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:29:52.348817    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5cdb62_7bd9_4e59_a46b_7ea066190216.slice/crio-bdeb46d023201a2bbdc80fc3c401fe651142d8e26d6ae9ce1ad155b0f59716c9 WatchSource:0}: Error finding container bdeb46d023201a2bbdc80fc3c401fe651142d8e26d6ae9ce1ad155b0f59716c9: Status 404 returned error can't find the container with id bdeb46d023201a2bbdc80fc3c401fe651142d8e26d6ae9ce1ad155b0f59716c9
Sep 16 14:29:53.223137 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:53.223108    2587 generic.go:356] "Generic (PLEG): container finished" podID="8e5cdb62-7bd9-4e59-a46b-7ea066190216" containerID="f96779c179520859ed64ec8b52810fa6e82a81d652e03bf0f4b0362442149e27" exitCode=0
Sep 16 14:29:53.223555 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:53.223148    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfce3e96-tlcz4" event={"ID":"8e5cdb62-7bd9-4e59-a46b-7ea066190216","Type":"ContainerDied","Data":"f96779c179520859ed64ec8b52810fa6e82a81d652e03bf0f4b0362442149e27"}
Sep 16 14:29:53.223555 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:53.223184    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfce3e96-tlcz4" event={"ID":"8e5cdb62-7bd9-4e59-a46b-7ea066190216","Type":"ContainerStarted","Data":"bdeb46d023201a2bbdc80fc3c401fe651142d8e26d6ae9ce1ad155b0f59716c9"}
Sep 16 14:29:54.366888 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:54.366857    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfce3e96-tlcz4"
Sep 16 14:29:54.427863 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:54.427826    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5cdb62-7bd9-4e59-a46b-7ea066190216-maas-api-service-ca\") pod \"8e5cdb62-7bd9-4e59-a46b-7ea066190216\" (UID: \"8e5cdb62-7bd9-4e59-a46b-7ea066190216\") "
Sep 16 14:29:54.428184 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:54.428159    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e5cdb62-7bd9-4e59-a46b-7ea066190216-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8e5cdb62-7bd9-4e59-a46b-7ea066190216" (UID: "8e5cdb62-7bd9-4e59-a46b-7ea066190216"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:29:54.529241 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:54.529145    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5cdb62-7bd9-4e59-a46b-7ea066190216-maas-api-service-ca\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:29:55.236143 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:55.236114    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfce3e96-tlcz4"
Sep 16 14:29:55.236311 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:55.236154    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfce3e96-tlcz4" event={"ID":"8e5cdb62-7bd9-4e59-a46b-7ea066190216","Type":"ContainerDied","Data":"bdeb46d023201a2bbdc80fc3c401fe651142d8e26d6ae9ce1ad155b0f59716c9"}
Sep 16 14:29:55.236311 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:55.236205    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdeb46d023201a2bbdc80fc3c401fe651142d8e26d6ae9ce1ad155b0f59716c9"
Sep 16 14:29:58.833384 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:58.833349    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz"]
Sep 16 14:29:58.833887 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:58.833633    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz" podUID="f456875b-dd77-4e09-9937-fc413c8cdb00" containerName="payload-processing" containerID="cri-o://209f16e70eb6fede89031b15e268946d8e233d62036eccc60eae8e52f25659f2" gracePeriod=30
Sep 16 14:29:58.843823 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:58.843759    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4"]
Sep 16 14:29:58.844289 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:58.844255    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4" podUID="2877e137-6e33-4a4a-9d3a-fcfd87b46174" containerName="payload-pre-processing" containerID="cri-o://5a210e232db79607dad9aded2bd1f13c2d22336cf2499e7b01798c14eff8d7a4" gracePeriod=30
Sep 16 14:29:59.007926 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.007857    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfce3e96-tlcz4"]
Sep 16 14:29:59.010825 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.010757    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-dfce3e96-tlcz4"]
Sep 16 14:29:59.141803 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.141776    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz"
Sep 16 14:29:59.145599 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.145572    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4"
Sep 16 14:29:59.172657 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.172629    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxnlf\" (UniqueName: \"kubernetes.io/projected/f456875b-dd77-4e09-9937-fc413c8cdb00-kube-api-access-qxnlf\") pod \"f456875b-dd77-4e09-9937-fc413c8cdb00\" (UID: \"f456875b-dd77-4e09-9937-fc413c8cdb00\") "
Sep 16 14:29:59.172793 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.172674    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7fqr\" (UniqueName: \"kubernetes.io/projected/2877e137-6e33-4a4a-9d3a-fcfd87b46174-kube-api-access-b7fqr\") pod \"2877e137-6e33-4a4a-9d3a-fcfd87b46174\" (UID: \"2877e137-6e33-4a4a-9d3a-fcfd87b46174\") "
Sep 16 14:29:59.172793 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.172696    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2877e137-6e33-4a4a-9d3a-fcfd87b46174-plugins-config-volume\") pod \"2877e137-6e33-4a4a-9d3a-fcfd87b46174\" (UID: \"2877e137-6e33-4a4a-9d3a-fcfd87b46174\") "
Sep 16 14:29:59.172876 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.172818    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f456875b-dd77-4e09-9937-fc413c8cdb00-plugins-config-volume\") pod \"f456875b-dd77-4e09-9937-fc413c8cdb00\" (UID: \"f456875b-dd77-4e09-9937-fc413c8cdb00\") "
Sep 16 14:29:59.173070 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.173042    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2877e137-6e33-4a4a-9d3a-fcfd87b46174-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2877e137-6e33-4a4a-9d3a-fcfd87b46174" (UID: "2877e137-6e33-4a4a-9d3a-fcfd87b46174"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:29:59.173195 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.173175    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f456875b-dd77-4e09-9937-fc413c8cdb00-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f456875b-dd77-4e09-9937-fc413c8cdb00" (UID: "f456875b-dd77-4e09-9937-fc413c8cdb00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:29:59.175668 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.175638    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2877e137-6e33-4a4a-9d3a-fcfd87b46174-kube-api-access-b7fqr" (OuterVolumeSpecName: "kube-api-access-b7fqr") pod "2877e137-6e33-4a4a-9d3a-fcfd87b46174" (UID: "2877e137-6e33-4a4a-9d3a-fcfd87b46174"). InnerVolumeSpecName "kube-api-access-b7fqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:29:59.175763 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.175679    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f456875b-dd77-4e09-9937-fc413c8cdb00-kube-api-access-qxnlf" (OuterVolumeSpecName: "kube-api-access-qxnlf") pod "f456875b-dd77-4e09-9937-fc413c8cdb00" (UID: "f456875b-dd77-4e09-9937-fc413c8cdb00"). InnerVolumeSpecName "kube-api-access-qxnlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:29:59.258043 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.258003    2587 generic.go:356] "Generic (PLEG): container finished" podID="2877e137-6e33-4a4a-9d3a-fcfd87b46174" containerID="5a210e232db79607dad9aded2bd1f13c2d22336cf2499e7b01798c14eff8d7a4" exitCode=0
Sep 16 14:29:59.258212 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.258059    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4"
Sep 16 14:29:59.258212 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.258094    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4" event={"ID":"2877e137-6e33-4a4a-9d3a-fcfd87b46174","Type":"ContainerDied","Data":"5a210e232db79607dad9aded2bd1f13c2d22336cf2499e7b01798c14eff8d7a4"}
Sep 16 14:29:59.258212 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.258156    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4" event={"ID":"2877e137-6e33-4a4a-9d3a-fcfd87b46174","Type":"ContainerDied","Data":"7e8c726b9900a26d17526eaa9a9cd36a44a705067af1134236c027d8c60109d4"}
Sep 16 14:29:59.258212 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.258188    2587 scope.go:117] "RemoveContainer" containerID="5a210e232db79607dad9aded2bd1f13c2d22336cf2499e7b01798c14eff8d7a4"
Sep 16 14:29:59.259677 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.259649    2587 generic.go:356] "Generic (PLEG): container finished" podID="f456875b-dd77-4e09-9937-fc413c8cdb00" containerID="209f16e70eb6fede89031b15e268946d8e233d62036eccc60eae8e52f25659f2" exitCode=0
Sep 16 14:29:59.259810 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.259697    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz"
Sep 16 14:29:59.259810 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.259744    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz" event={"ID":"f456875b-dd77-4e09-9937-fc413c8cdb00","Type":"ContainerDied","Data":"209f16e70eb6fede89031b15e268946d8e233d62036eccc60eae8e52f25659f2"}
Sep 16 14:29:59.259810 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.259797    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz" event={"ID":"f456875b-dd77-4e09-9937-fc413c8cdb00","Type":"ContainerDied","Data":"a0901806ef7b5d7709223b3a0981ec94811c56389c9df94ac1ef5d9e3a9b66ed"}
Sep 16 14:29:59.267946 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.267930    2587 scope.go:117] "RemoveContainer" containerID="5a210e232db79607dad9aded2bd1f13c2d22336cf2499e7b01798c14eff8d7a4"
Sep 16 14:29:59.268333 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:29:59.268268    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a210e232db79607dad9aded2bd1f13c2d22336cf2499e7b01798c14eff8d7a4\": container with ID starting with 5a210e232db79607dad9aded2bd1f13c2d22336cf2499e7b01798c14eff8d7a4 not found: ID does not exist" containerID="5a210e232db79607dad9aded2bd1f13c2d22336cf2499e7b01798c14eff8d7a4"
Sep 16 14:29:59.268333 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.268296    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a210e232db79607dad9aded2bd1f13c2d22336cf2499e7b01798c14eff8d7a4"} err="failed to get container status \"5a210e232db79607dad9aded2bd1f13c2d22336cf2499e7b01798c14eff8d7a4\": rpc error: code = NotFound desc = could not find container \"5a210e232db79607dad9aded2bd1f13c2d22336cf2499e7b01798c14eff8d7a4\": container with ID starting with 5a210e232db79607dad9aded2bd1f13c2d22336cf2499e7b01798c14eff8d7a4 not found: ID does not exist"
Sep 16 14:29:59.268333 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.268311    2587 scope.go:117] "RemoveContainer" containerID="209f16e70eb6fede89031b15e268946d8e233d62036eccc60eae8e52f25659f2"
Sep 16 14:29:59.274290 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.274270    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f456875b-dd77-4e09-9937-fc413c8cdb00-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:29:59.274382 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.274292    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxnlf\" (UniqueName: \"kubernetes.io/projected/f456875b-dd77-4e09-9937-fc413c8cdb00-kube-api-access-qxnlf\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:29:59.274382 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.274312    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7fqr\" (UniqueName: \"kubernetes.io/projected/2877e137-6e33-4a4a-9d3a-fcfd87b46174-kube-api-access-b7fqr\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:29:59.274382 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.274321    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2877e137-6e33-4a4a-9d3a-fcfd87b46174-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:29:59.277119 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.277104    2587 scope.go:117] "RemoveContainer" containerID="209f16e70eb6fede89031b15e268946d8e233d62036eccc60eae8e52f25659f2"
Sep 16 14:29:59.277438 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:29:59.277418    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"209f16e70eb6fede89031b15e268946d8e233d62036eccc60eae8e52f25659f2\": container with ID starting with 209f16e70eb6fede89031b15e268946d8e233d62036eccc60eae8e52f25659f2 not found: ID does not exist" containerID="209f16e70eb6fede89031b15e268946d8e233d62036eccc60eae8e52f25659f2"
Sep 16 14:29:59.277495 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.277445    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"209f16e70eb6fede89031b15e268946d8e233d62036eccc60eae8e52f25659f2"} err="failed to get container status \"209f16e70eb6fede89031b15e268946d8e233d62036eccc60eae8e52f25659f2\": rpc error: code = NotFound desc = could not find container \"209f16e70eb6fede89031b15e268946d8e233d62036eccc60eae8e52f25659f2\": container with ID starting with 209f16e70eb6fede89031b15e268946d8e233d62036eccc60eae8e52f25659f2 not found: ID does not exist"
Sep 16 14:29:59.290093 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.290072    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4"]
Sep 16 14:29:59.294364 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.294337    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-dfce3e96-64c9f58b86-9btd4"]
Sep 16 14:29:59.310073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.310049    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz"]
Sep 16 14:29:59.312973 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.312948    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-dfce3e96-767dcb9d57-zjmfz"]
Sep 16 14:29:59.904763 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.904716    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2877e137-6e33-4a4a-9d3a-fcfd87b46174" path="/var/lib/kubelet/pods/2877e137-6e33-4a4a-9d3a-fcfd87b46174/volumes"
Sep 16 14:29:59.905263 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.905240    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e5cdb62-7bd9-4e59-a46b-7ea066190216" path="/var/lib/kubelet/pods/8e5cdb62-7bd9-4e59-a46b-7ea066190216/volumes"
Sep 16 14:29:59.905800 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:29:59.905780    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f456875b-dd77-4e09-9937-fc413c8cdb00" path="/var/lib/kubelet/pods/f456875b-dd77-4e09-9937-fc413c8cdb00/volumes"
Sep 16 14:30:00.172471 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:00.172439    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-5d484d-29826150-6xckn"]
Sep 16 14:30:00.172838 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:00.172825    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2877e137-6e33-4a4a-9d3a-fcfd87b46174" containerName="payload-pre-processing"
Sep 16 14:30:00.172903 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:00.172840    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="2877e137-6e33-4a4a-9d3a-fcfd87b46174" containerName="payload-pre-processing"
Sep 16 14:30:00.172903 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:00.172863    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e5cdb62-7bd9-4e59-a46b-7ea066190216" containerName="revoke-keys"
Sep 16 14:30:00.172903 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:00.172868    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e5cdb62-7bd9-4e59-a46b-7ea066190216" containerName="revoke-keys"
Sep 16 14:30:00.172903 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:00.172878    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f456875b-dd77-4e09-9937-fc413c8cdb00" containerName="payload-processing"
Sep 16 14:30:00.172903 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:00.172884    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="f456875b-dd77-4e09-9937-fc413c8cdb00" containerName="payload-processing"
Sep 16 14:30:00.173054 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:00.172962    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="f456875b-dd77-4e09-9937-fc413c8cdb00" containerName="payload-processing"
Sep 16 14:30:00.173054 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:00.172969    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="2877e137-6e33-4a4a-9d3a-fcfd87b46174" containerName="payload-pre-processing"
Sep 16 14:30:00.173054 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:00.172976    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e5cdb62-7bd9-4e59-a46b-7ea066190216" containerName="revoke-keys"
Sep 16 14:30:00.177449 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:00.177426    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-5d484d-29826150-6xckn"
Sep 16 14:30:00.185140 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:00.185118    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0b89a-29826150-vz9g8"]
Sep 16 14:30:00.188688 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:00.188671    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4jf\""
Sep 16 14:30:00.188910 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:00.188896    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0b89a-29826150-vz9g8"
Sep 16 14:30:00.197689 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:00.197674    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0b89a-29826150-vz9g8"
Sep 16 14:30:00.198517 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:00.198500    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-5d484d-29826150-6xckn"
Sep 16 14:30:00.222099 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:00.218775    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-5d484d-29826150-6xckn"]
Sep 16 14:30:00.224177 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:00.223008    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-924916-29826150-gdtmq"]
Sep 16 14:30:00.231121 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:00.229542    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-924916-29826150-gdtmq"
Sep 16 14:30:00.231121 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:00.229959    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0b89a-29826150-vz9g8"]
Sep 16 14:30:00.236125 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:00.233474    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-924916-29826150-gdtmq"]
Sep 16 14:30:00.256765 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:00.256730    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-924916-29826150-gdtmq"
Sep 16 14:30:00.504037 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:00.503998    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0b89a-29826150-vz9g8"]
Sep 16 14:30:00.505417 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:30:00.505374    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a6caff_e0df_4d1e_9bab_56052ba7ddfe.slice/crio-b3d48aab90f4950882669f91dac3e37de7273b3f417eaa56ffa61e8c3d765d12 WatchSource:0}: Error finding container b3d48aab90f4950882669f91dac3e37de7273b3f417eaa56ffa61e8c3d765d12: Status 404 returned error can't find the container with id b3d48aab90f4950882669f91dac3e37de7273b3f417eaa56ffa61e8c3d765d12
Sep 16 14:30:00.519686 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:00.519660    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-5d484d-29826150-6xckn"]
Sep 16 14:30:00.521497 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:30:00.521464    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4759566e_b40f_421b_b998_e5bbc1738a1c.slice/crio-d3049a6333af9796e4e5bde47e3faa70060517890f125d2f5f390d9b0d77b005 WatchSource:0}: Error finding container d3049a6333af9796e4e5bde47e3faa70060517890f125d2f5f390d9b0d77b005: Status 404 returned error can't find the container with id d3049a6333af9796e4e5bde47e3faa70060517890f125d2f5f390d9b0d77b005
Sep 16 14:30:00.561161 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:00.561133    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-924916-29826150-gdtmq"]
Sep 16 14:30:00.562683 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:30:00.562653    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a96c4bb_f7b3_4f0a_8870_a5a3cc813c34.slice/crio-47be9bfd75f70fa5dd9ffcb53ea0f6b6e82e377525927681c758ca44763ff15d WatchSource:0}: Error finding container 47be9bfd75f70fa5dd9ffcb53ea0f6b6e82e377525927681c758ca44763ff15d: Status 404 returned error can't find the container with id 47be9bfd75f70fa5dd9ffcb53ea0f6b6e82e377525927681c758ca44763ff15d
Sep 16 14:30:01.295370 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:01.295330    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-924916-29826150-gdtmq" event={"ID":"9a96c4bb-f7b3-4f0a-8870-a5a3cc813c34","Type":"ContainerStarted","Data":"5b879fca25bff2b978931a8297bea0f44ea89d882f6570fe4cd0a0938842c519"}
Sep 16 14:30:01.295876 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:01.295423    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-924916-29826150-gdtmq" event={"ID":"9a96c4bb-f7b3-4f0a-8870-a5a3cc813c34","Type":"ContainerStarted","Data":"47be9bfd75f70fa5dd9ffcb53ea0f6b6e82e377525927681c758ca44763ff15d"}
Sep 16 14:30:01.296965 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:01.296933    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-5d484d-29826150-6xckn" event={"ID":"4759566e-b40f-421b-b998-e5bbc1738a1c","Type":"ContainerStarted","Data":"f09eb55be577e2c916402f6fd7f50505169f332ecfa63e05d660795d408eee2d"}
Sep 16 14:30:01.297082 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:01.296976    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-5d484d-29826150-6xckn" event={"ID":"4759566e-b40f-421b-b998-e5bbc1738a1c","Type":"ContainerStarted","Data":"d3049a6333af9796e4e5bde47e3faa70060517890f125d2f5f390d9b0d77b005"}
Sep 16 14:30:01.298543 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:01.298516    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0b89a-29826150-vz9g8" event={"ID":"82a6caff-e0df-4d1e-9bab-56052ba7ddfe","Type":"ContainerStarted","Data":"b6ffe6e36775ca2d8333b4a0b371bbdcf980137333889ce7defacd385c8ef2b3"}
Sep 16 14:30:01.298648 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:01.298563    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0b89a-29826150-vz9g8" event={"ID":"82a6caff-e0df-4d1e-9bab-56052ba7ddfe","Type":"ContainerStarted","Data":"b3d48aab90f4950882669f91dac3e37de7273b3f417eaa56ffa61e8c3d765d12"}
Sep 16 14:30:01.318253 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:01.318105    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-924916-29826150-gdtmq" podStartSLOduration=1.318084791 podStartE2EDuration="1.318084791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:30:01.317547706 +0000 UTC m=+949.921172738" watchObservedRunningTime="2026-09-16 14:30:01.318084791 +0000 UTC m=+949.921709825"
Sep 16 14:30:01.337700 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:01.337649    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-5d484d-29826150-6xckn" podStartSLOduration=1.337633088 podStartE2EDuration="1.337633088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:30:01.335810605 +0000 UTC m=+949.939435638" watchObservedRunningTime="2026-09-16 14:30:01.337633088 +0000 UTC m=+949.941258125"
Sep 16 14:30:01.351443 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:01.351356    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0b89a-29826150-vz9g8" podStartSLOduration=1.3513406479999999 podStartE2EDuration="1.351340648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:30:01.351169247 +0000 UTC m=+949.954794279" watchObservedRunningTime="2026-09-16 14:30:01.351340648 +0000 UTC m=+949.954965680"
Sep 16 14:30:02.305492 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:02.305456    2587 generic.go:356] "Generic (PLEG): container finished" podID="82a6caff-e0df-4d1e-9bab-56052ba7ddfe" containerID="b6ffe6e36775ca2d8333b4a0b371bbdcf980137333889ce7defacd385c8ef2b3" exitCode=0
Sep 16 14:30:02.305927 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:02.305541    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0b89a-29826150-vz9g8" event={"ID":"82a6caff-e0df-4d1e-9bab-56052ba7ddfe","Type":"ContainerDied","Data":"b6ffe6e36775ca2d8333b4a0b371bbdcf980137333889ce7defacd385c8ef2b3"}
Sep 16 14:30:02.307261 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:02.307238    2587 generic.go:356] "Generic (PLEG): container finished" podID="9a96c4bb-f7b3-4f0a-8870-a5a3cc813c34" containerID="5b879fca25bff2b978931a8297bea0f44ea89d882f6570fe4cd0a0938842c519" exitCode=0
Sep 16 14:30:02.307367 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:02.307309    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-924916-29826150-gdtmq" event={"ID":"9a96c4bb-f7b3-4f0a-8870-a5a3cc813c34","Type":"ContainerDied","Data":"5b879fca25bff2b978931a8297bea0f44ea89d882f6570fe4cd0a0938842c519"}
Sep 16 14:30:02.308969 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:02.308938    2587 generic.go:356] "Generic (PLEG): container finished" podID="4759566e-b40f-421b-b998-e5bbc1738a1c" containerID="f09eb55be577e2c916402f6fd7f50505169f332ecfa63e05d660795d408eee2d" exitCode=0
Sep 16 14:30:02.309060 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:02.308969    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-5d484d-29826150-6xckn" event={"ID":"4759566e-b40f-421b-b998-e5bbc1738a1c","Type":"ContainerDied","Data":"f09eb55be577e2c916402f6fd7f50505169f332ecfa63e05d660795d408eee2d"}
Sep 16 14:30:03.458587 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:03.458556    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-924916-29826150-gdtmq"
Sep 16 14:30:03.516038 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:03.516012    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0b89a-29826150-vz9g8"
Sep 16 14:30:03.519228 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:03.519207    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-5d484d-29826150-6xckn"
Sep 16 14:30:04.320841 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:04.320756    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-5d484d-29826150-6xckn"
Sep 16 14:30:04.320841 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:04.320798    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-5d484d-29826150-6xckn" event={"ID":"4759566e-b40f-421b-b998-e5bbc1738a1c","Type":"ContainerDied","Data":"d3049a6333af9796e4e5bde47e3faa70060517890f125d2f5f390d9b0d77b005"}
Sep 16 14:30:04.321073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:04.320852    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3049a6333af9796e4e5bde47e3faa70060517890f125d2f5f390d9b0d77b005"
Sep 16 14:30:04.322224 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:04.322204    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0b89a-29826150-vz9g8"
Sep 16 14:30:04.322382 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:04.322252    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0b89a-29826150-vz9g8" event={"ID":"82a6caff-e0df-4d1e-9bab-56052ba7ddfe","Type":"ContainerDied","Data":"b3d48aab90f4950882669f91dac3e37de7273b3f417eaa56ffa61e8c3d765d12"}
Sep 16 14:30:04.322382 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:04.322289    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3d48aab90f4950882669f91dac3e37de7273b3f417eaa56ffa61e8c3d765d12"
Sep 16 14:30:04.323813 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:04.323798    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-924916-29826150-gdtmq"
Sep 16 14:30:04.323813 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:04.323796    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-924916-29826150-gdtmq" event={"ID":"9a96c4bb-f7b3-4f0a-8870-a5a3cc813c34","Type":"ContainerDied","Data":"47be9bfd75f70fa5dd9ffcb53ea0f6b6e82e377525927681c758ca44763ff15d"}
Sep 16 14:30:04.323924 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:04.323841    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47be9bfd75f70fa5dd9ffcb53ea0f6b6e82e377525927681c758ca44763ff15d"
Sep 16 14:30:20.919988 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:20.919952    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4"]
Sep 16 14:30:20.920447 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:20.920418    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a96c4bb-f7b3-4f0a-8870-a5a3cc813c34" containerName="cleanup"
Sep 16 14:30:20.920447 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:20.920437    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a96c4bb-f7b3-4f0a-8870-a5a3cc813c34" containerName="cleanup"
Sep 16 14:30:20.920447 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:20.920448    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4759566e-b40f-421b-b998-e5bbc1738a1c" containerName="cleanup"
Sep 16 14:30:20.920447 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:20.920454    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="4759566e-b40f-421b-b998-e5bbc1738a1c" containerName="cleanup"
Sep 16 14:30:20.920742 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:20.920468    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82a6caff-e0df-4d1e-9bab-56052ba7ddfe" containerName="cleanup"
Sep 16 14:30:20.920742 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:20.920474    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="82a6caff-e0df-4d1e-9bab-56052ba7ddfe" containerName="cleanup"
Sep 16 14:30:20.920742 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:20.920540    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a96c4bb-f7b3-4f0a-8870-a5a3cc813c34" containerName="cleanup"
Sep 16 14:30:20.920742 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:20.920546    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="4759566e-b40f-421b-b998-e5bbc1738a1c" containerName="cleanup"
Sep 16 14:30:20.920742 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:20.920555    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="82a6caff-e0df-4d1e-9bab-56052ba7ddfe" containerName="cleanup"
Sep 16 14:30:20.924889 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:20.924869    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4"
Sep 16 14:30:20.927676 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:20.927656    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-7ce904-dockercfg-qzx2g\""
Sep 16 14:30:20.927808 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:20.927656    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-7ce904\""
Sep 16 14:30:20.939329 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:20.939292    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4"]
Sep 16 14:30:20.959004 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:20.958971    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4"]
Sep 16 14:30:20.962554 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:20.962532    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4"
Sep 16 14:30:20.970907 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:20.970883    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4"]
Sep 16 14:30:21.085582 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:21.085541    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96a7b4f-a182-4ef1-aa53-adb433fb7457-plugins-config-volume\") pod \"payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4\" (UID: \"f96a7b4f-a182-4ef1-aa53-adb433fb7457\") " pod="openshift-ingress/payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4"
Sep 16 14:30:21.085743 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:21.085585    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7r4g\" (UniqueName: \"kubernetes.io/projected/e71bd69e-4c4c-433a-85e2-c7b7d3465e43-kube-api-access-j7r4g\") pod \"payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4\" (UID: \"e71bd69e-4c4c-433a-85e2-c7b7d3465e43\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4"
Sep 16 14:30:21.085743 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:21.085608    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25rnj\" (UniqueName: \"kubernetes.io/projected/f96a7b4f-a182-4ef1-aa53-adb433fb7457-kube-api-access-25rnj\") pod \"payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4\" (UID: \"f96a7b4f-a182-4ef1-aa53-adb433fb7457\") " pod="openshift-ingress/payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4"
Sep 16 14:30:21.085743 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:21.085719    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e71bd69e-4c4c-433a-85e2-c7b7d3465e43-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4\" (UID: \"e71bd69e-4c4c-433a-85e2-c7b7d3465e43\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4"
Sep 16 14:30:21.186893 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:21.186799    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e71bd69e-4c4c-433a-85e2-c7b7d3465e43-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4\" (UID: \"e71bd69e-4c4c-433a-85e2-c7b7d3465e43\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4"
Sep 16 14:30:21.187051 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:21.186903    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96a7b4f-a182-4ef1-aa53-adb433fb7457-plugins-config-volume\") pod \"payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4\" (UID: \"f96a7b4f-a182-4ef1-aa53-adb433fb7457\") " pod="openshift-ingress/payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4"
Sep 16 14:30:21.187051 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:21.186922    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7r4g\" (UniqueName: \"kubernetes.io/projected/e71bd69e-4c4c-433a-85e2-c7b7d3465e43-kube-api-access-j7r4g\") pod \"payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4\" (UID: \"e71bd69e-4c4c-433a-85e2-c7b7d3465e43\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4"
Sep 16 14:30:21.187051 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:21.186941    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25rnj\" (UniqueName: \"kubernetes.io/projected/f96a7b4f-a182-4ef1-aa53-adb433fb7457-kube-api-access-25rnj\") pod \"payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4\" (UID: \"f96a7b4f-a182-4ef1-aa53-adb433fb7457\") " pod="openshift-ingress/payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4"
Sep 16 14:30:21.187754 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:21.187719    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96a7b4f-a182-4ef1-aa53-adb433fb7457-plugins-config-volume\") pod \"payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4\" (UID: \"f96a7b4f-a182-4ef1-aa53-adb433fb7457\") " pod="openshift-ingress/payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4"
Sep 16 14:30:21.187886 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:21.187772    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e71bd69e-4c4c-433a-85e2-c7b7d3465e43-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4\" (UID: \"e71bd69e-4c4c-433a-85e2-c7b7d3465e43\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4"
Sep 16 14:30:21.195465 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:21.195417    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7r4g\" (UniqueName: \"kubernetes.io/projected/e71bd69e-4c4c-433a-85e2-c7b7d3465e43-kube-api-access-j7r4g\") pod \"payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4\" (UID: \"e71bd69e-4c4c-433a-85e2-c7b7d3465e43\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4"
Sep 16 14:30:21.195620 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:21.195479    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25rnj\" (UniqueName: \"kubernetes.io/projected/f96a7b4f-a182-4ef1-aa53-adb433fb7457-kube-api-access-25rnj\") pod \"payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4\" (UID: \"f96a7b4f-a182-4ef1-aa53-adb433fb7457\") " pod="openshift-ingress/payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4"
Sep 16 14:30:21.235812 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:21.235782    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4"
Sep 16 14:30:21.274192 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:21.274156    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4"
Sep 16 14:30:21.524930 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:21.524902    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4"]
Sep 16 14:30:21.526735 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:30:21.526701    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71bd69e_4c4c_433a_85e2_c7b7d3465e43.slice/crio-f9cf84e808721a5fd1fb7baf05a5df641414207864f334ec92eb9d67e216cde4 WatchSource:0}: Error finding container f9cf84e808721a5fd1fb7baf05a5df641414207864f334ec92eb9d67e216cde4: Status 404 returned error can't find the container with id f9cf84e808721a5fd1fb7baf05a5df641414207864f334ec92eb9d67e216cde4
Sep 16 14:30:21.561512 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:21.561450    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4"]
Sep 16 14:30:21.564833 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:30:21.564789    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96a7b4f_a182_4ef1_aa53_adb433fb7457.slice/crio-e340902a1a315d48de747bcbcee563f8975dec3c878ea8b00d1bc36a8d9f8453 WatchSource:0}: Error finding container e340902a1a315d48de747bcbcee563f8975dec3c878ea8b00d1bc36a8d9f8453: Status 404 returned error can't find the container with id e340902a1a315d48de747bcbcee563f8975dec3c878ea8b00d1bc36a8d9f8453
Sep 16 14:30:22.423329 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:22.423294    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4" event={"ID":"f96a7b4f-a182-4ef1-aa53-adb433fb7457","Type":"ContainerStarted","Data":"6d1ff189ccac1e960b5f48ee2c8cd124faa18b6c6573f3ec0c08eedb16579e6c"}
Sep 16 14:30:22.423863 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:22.423359    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4" event={"ID":"f96a7b4f-a182-4ef1-aa53-adb433fb7457","Type":"ContainerStarted","Data":"e340902a1a315d48de747bcbcee563f8975dec3c878ea8b00d1bc36a8d9f8453"}
Sep 16 14:30:22.423863 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:22.423505    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4"
Sep 16 14:30:22.425166 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:22.425142    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4" event={"ID":"e71bd69e-4c4c-433a-85e2-c7b7d3465e43","Type":"ContainerStarted","Data":"0e552e71c76993840aa2274c1900f5fb72b439c5f81baf795c6062e6311c2b29"}
Sep 16 14:30:22.425313 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:22.425190    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4" event={"ID":"e71bd69e-4c4c-433a-85e2-c7b7d3465e43","Type":"ContainerStarted","Data":"f9cf84e808721a5fd1fb7baf05a5df641414207864f334ec92eb9d67e216cde4"}
Sep 16 14:30:22.425313 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:22.425238    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4"
Sep 16 14:30:22.444735 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:22.444676    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4" podStartSLOduration=2.444658499 podStartE2EDuration="2.444658499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:30:22.442894677 +0000 UTC m=+971.046519705" watchObservedRunningTime="2026-09-16 14:30:22.444658499 +0000 UTC m=+971.048283544"
Sep 16 14:30:22.461171 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:22.461112    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4" podStartSLOduration=2.46109506 podStartE2EDuration="2.46109506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:30:22.46019738 +0000 UTC m=+971.063822413" watchObservedRunningTime="2026-09-16 14:30:22.46109506 +0000 UTC m=+971.064720092"
Sep 16 14:30:31.594983 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:31.594943    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"]
Sep 16 14:30:31.598374 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:31.598356    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"
Sep 16 14:30:31.601872 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:31.601844    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-5d484d\"/\"kube-root-ca.crt\""
Sep 16 14:30:31.602012 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:31.601856    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5d484d\"/\"default-dockercfg-vdgm5\""
Sep 16 14:30:31.602012 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:31.601856    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-5d484d\"/\"openshift-service-ca.crt\""
Sep 16 14:30:31.602012 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:31.601861    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5d484d\"/\"premium-w4-5d484d-kserve-self-signed-certs\""
Sep 16 14:30:31.609493 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:31.609471    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"]
Sep 16 14:30:31.684695 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:31.684653    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq8tq\" (UniqueName: \"kubernetes.io/projected/6edfb182-3de4-4361-ab1a-2c70fb152fc7-kube-api-access-rq8tq\") pod \"premium-w4-5d484d-kserve-765d7d889f-xpsn4\" (UID: \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\") " pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"
Sep 16 14:30:31.684695 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:31.684707    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edfb182-3de4-4361-ab1a-2c70fb152fc7-tmp-dir\") pod \"premium-w4-5d484d-kserve-765d7d889f-xpsn4\" (UID: \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\") " pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"
Sep 16 14:30:31.684921 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:31.684775    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6edfb182-3de4-4361-ab1a-2c70fb152fc7-home\") pod \"premium-w4-5d484d-kserve-765d7d889f-xpsn4\" (UID: \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\") " pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"
Sep 16 14:30:31.684921 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:31.684812    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6edfb182-3de4-4361-ab1a-2c70fb152fc7-dshm\") pod \"premium-w4-5d484d-kserve-765d7d889f-xpsn4\" (UID: \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\") " pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"
Sep 16 14:30:31.684921 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:31.684840    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6edfb182-3de4-4361-ab1a-2c70fb152fc7-tls-certs\") pod \"premium-w4-5d484d-kserve-765d7d889f-xpsn4\" (UID: \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\") " pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"
Sep 16 14:30:31.684921 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:31.684884    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6edfb182-3de4-4361-ab1a-2c70fb152fc7-model-cache\") pod \"premium-w4-5d484d-kserve-765d7d889f-xpsn4\" (UID: \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\") " pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"
Sep 16 14:30:31.786154 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:31.786110    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6edfb182-3de4-4361-ab1a-2c70fb152fc7-model-cache\") pod \"premium-w4-5d484d-kserve-765d7d889f-xpsn4\" (UID: \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\") " pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"
Sep 16 14:30:31.786362 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:31.786182    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq8tq\" (UniqueName: \"kubernetes.io/projected/6edfb182-3de4-4361-ab1a-2c70fb152fc7-kube-api-access-rq8tq\") pod \"premium-w4-5d484d-kserve-765d7d889f-xpsn4\" (UID: \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\") " pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"
Sep 16 14:30:31.786362 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:31.786218    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edfb182-3de4-4361-ab1a-2c70fb152fc7-tmp-dir\") pod \"premium-w4-5d484d-kserve-765d7d889f-xpsn4\" (UID: \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\") " pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"
Sep 16 14:30:31.786362 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:31.786245    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6edfb182-3de4-4361-ab1a-2c70fb152fc7-home\") pod \"premium-w4-5d484d-kserve-765d7d889f-xpsn4\" (UID: \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\") " pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"
Sep 16 14:30:31.786362 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:31.786262    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6edfb182-3de4-4361-ab1a-2c70fb152fc7-dshm\") pod \"premium-w4-5d484d-kserve-765d7d889f-xpsn4\" (UID: \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\") " pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"
Sep 16 14:30:31.786362 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:31.786297    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6edfb182-3de4-4361-ab1a-2c70fb152fc7-tls-certs\") pod \"premium-w4-5d484d-kserve-765d7d889f-xpsn4\" (UID: \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\") " pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"
Sep 16 14:30:31.786680 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:31.786588    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6edfb182-3de4-4361-ab1a-2c70fb152fc7-model-cache\") pod \"premium-w4-5d484d-kserve-765d7d889f-xpsn4\" (UID: \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\") " pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"
Sep 16 14:30:31.786680 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:31.786627    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edfb182-3de4-4361-ab1a-2c70fb152fc7-tmp-dir\") pod \"premium-w4-5d484d-kserve-765d7d889f-xpsn4\" (UID: \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\") " pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"
Sep 16 14:30:31.786920 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:31.786899    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6edfb182-3de4-4361-ab1a-2c70fb152fc7-home\") pod \"premium-w4-5d484d-kserve-765d7d889f-xpsn4\" (UID: \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\") " pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"
Sep 16 14:30:31.788759 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:31.788737    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6edfb182-3de4-4361-ab1a-2c70fb152fc7-dshm\") pod \"premium-w4-5d484d-kserve-765d7d889f-xpsn4\" (UID: \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\") " pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"
Sep 16 14:30:31.789025 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:31.789007    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6edfb182-3de4-4361-ab1a-2c70fb152fc7-tls-certs\") pod \"premium-w4-5d484d-kserve-765d7d889f-xpsn4\" (UID: \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\") " pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"
Sep 16 14:30:31.795124 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:31.795103    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq8tq\" (UniqueName: \"kubernetes.io/projected/6edfb182-3de4-4361-ab1a-2c70fb152fc7-kube-api-access-rq8tq\") pod \"premium-w4-5d484d-kserve-765d7d889f-xpsn4\" (UID: \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\") " pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"
Sep 16 14:30:31.908723 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:31.908684    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"
Sep 16 14:30:32.184179 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:32.184154    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"]
Sep 16 14:30:32.185876 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:30:32.185838    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6edfb182_3de4_4361_ab1a_2c70fb152fc7.slice/crio-098b12b172796a6cf6877304751e12c7bdbd30e6d874923e572c31d25df8d3bc WatchSource:0}: Error finding container 098b12b172796a6cf6877304751e12c7bdbd30e6d874923e572c31d25df8d3bc: Status 404 returned error can't find the container with id 098b12b172796a6cf6877304751e12c7bdbd30e6d874923e572c31d25df8d3bc
Sep 16 14:30:32.480781 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:32.480684    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4" event={"ID":"6edfb182-3de4-4361-ab1a-2c70fb152fc7","Type":"ContainerStarted","Data":"4394d64ef49dd238ae6e7f6444af77c9b794a6bd9788b05be9f383b66474500e"}
Sep 16 14:30:32.480781 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:32.480730    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4" event={"ID":"6edfb182-3de4-4361-ab1a-2c70fb152fc7","Type":"ContainerStarted","Data":"098b12b172796a6cf6877304751e12c7bdbd30e6d874923e572c31d25df8d3bc"}
Sep 16 14:30:32.499188 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:32.499132    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4" podStartSLOduration=1.499114027 podStartE2EDuration="1.499114027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:30:32.498468001 +0000 UTC m=+981.102093032" watchObservedRunningTime="2026-09-16 14:30:32.499114027 +0000 UTC m=+981.102739092"
Sep 16 14:30:33.433040 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:33.433007    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4"
Sep 16 14:30:33.433446 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:33.433104    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4"
Sep 16 14:30:41.909236 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:41.909192    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"
Sep 16 14:30:41.909654 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:41.909247    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"
Sep 16 14:30:41.926817 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:41.926782    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"
Sep 16 14:30:42.547659 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:30:42.547631    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"
Sep 16 14:31:20.451558 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:20.451525    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"]
Sep 16 14:31:20.456805 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:20.456767    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"
Sep 16 14:31:20.460316 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:20.460287    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-924916\"/\"kube-root-ca.crt\""
Sep 16 14:31:20.461299 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:20.461271    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-924916\"/\"openshift-service-ca.crt\""
Sep 16 14:31:20.461567 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:20.461549    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-924916\"/\"default-dockercfg-xgs7f\""
Sep 16 14:31:20.462125 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:20.462109    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-924916\"/\"premium-w0-924916-kserve-self-signed-certs\""
Sep 16 14:31:20.467530 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:20.467506    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"]
Sep 16 14:31:20.556468 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:20.556426    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f90242-bf5f-46fc-8acc-28dfc45ce02f-home\") pod \"premium-w0-924916-kserve-58645fcd45-wc58w\" (UID: \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\") " pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"
Sep 16 14:31:20.556659 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:20.556501    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f90242-bf5f-46fc-8acc-28dfc45ce02f-tls-certs\") pod \"premium-w0-924916-kserve-58645fcd45-wc58w\" (UID: \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\") " pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"
Sep 16 14:31:20.556659 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:20.556548    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f90242-bf5f-46fc-8acc-28dfc45ce02f-dshm\") pod \"premium-w0-924916-kserve-58645fcd45-wc58w\" (UID: \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\") " pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"
Sep 16 14:31:20.556659 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:20.556565    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f90242-bf5f-46fc-8acc-28dfc45ce02f-model-cache\") pod \"premium-w0-924916-kserve-58645fcd45-wc58w\" (UID: \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\") " pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"
Sep 16 14:31:20.556659 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:20.556612    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgtbb\" (UniqueName: \"kubernetes.io/projected/27f90242-bf5f-46fc-8acc-28dfc45ce02f-kube-api-access-xgtbb\") pod \"premium-w0-924916-kserve-58645fcd45-wc58w\" (UID: \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\") " pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"
Sep 16 14:31:20.556659 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:20.556651    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f90242-bf5f-46fc-8acc-28dfc45ce02f-tmp-dir\") pod \"premium-w0-924916-kserve-58645fcd45-wc58w\" (UID: \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\") " pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"
Sep 16 14:31:20.657571 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:20.657529    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f90242-bf5f-46fc-8acc-28dfc45ce02f-dshm\") pod \"premium-w0-924916-kserve-58645fcd45-wc58w\" (UID: \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\") " pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"
Sep 16 14:31:20.657571 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:20.657568    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f90242-bf5f-46fc-8acc-28dfc45ce02f-model-cache\") pod \"premium-w0-924916-kserve-58645fcd45-wc58w\" (UID: \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\") " pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"
Sep 16 14:31:20.657773 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:20.657608    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgtbb\" (UniqueName: \"kubernetes.io/projected/27f90242-bf5f-46fc-8acc-28dfc45ce02f-kube-api-access-xgtbb\") pod \"premium-w0-924916-kserve-58645fcd45-wc58w\" (UID: \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\") " pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"
Sep 16 14:31:20.657773 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:20.657660    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f90242-bf5f-46fc-8acc-28dfc45ce02f-tmp-dir\") pod \"premium-w0-924916-kserve-58645fcd45-wc58w\" (UID: \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\") " pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"
Sep 16 14:31:20.657773 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:20.657714    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f90242-bf5f-46fc-8acc-28dfc45ce02f-home\") pod \"premium-w0-924916-kserve-58645fcd45-wc58w\" (UID: \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\") " pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"
Sep 16 14:31:20.657899 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:20.657780    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f90242-bf5f-46fc-8acc-28dfc45ce02f-tls-certs\") pod \"premium-w0-924916-kserve-58645fcd45-wc58w\" (UID: \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\") " pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"
Sep 16 14:31:20.658077 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:20.658053    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f90242-bf5f-46fc-8acc-28dfc45ce02f-model-cache\") pod \"premium-w0-924916-kserve-58645fcd45-wc58w\" (UID: \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\") " pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"
Sep 16 14:31:20.658159 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:20.658099    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f90242-bf5f-46fc-8acc-28dfc45ce02f-home\") pod \"premium-w0-924916-kserve-58645fcd45-wc58w\" (UID: \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\") " pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"
Sep 16 14:31:20.658159 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:20.658123    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f90242-bf5f-46fc-8acc-28dfc45ce02f-tmp-dir\") pod \"premium-w0-924916-kserve-58645fcd45-wc58w\" (UID: \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\") " pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"
Sep 16 14:31:20.660160 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:20.660128    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f90242-bf5f-46fc-8acc-28dfc45ce02f-dshm\") pod \"premium-w0-924916-kserve-58645fcd45-wc58w\" (UID: \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\") " pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"
Sep 16 14:31:20.660426 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:20.660372    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f90242-bf5f-46fc-8acc-28dfc45ce02f-tls-certs\") pod \"premium-w0-924916-kserve-58645fcd45-wc58w\" (UID: \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\") " pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"
Sep 16 14:31:20.669816 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:20.669793    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgtbb\" (UniqueName: \"kubernetes.io/projected/27f90242-bf5f-46fc-8acc-28dfc45ce02f-kube-api-access-xgtbb\") pod \"premium-w0-924916-kserve-58645fcd45-wc58w\" (UID: \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\") " pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"
Sep 16 14:31:20.768492 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:20.768406    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"
Sep 16 14:31:21.056557 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:21.056529    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"]
Sep 16 14:31:21.056856 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:31:21.056831    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27f90242_bf5f_46fc_8acc_28dfc45ce02f.slice/crio-5b7b0eac725eb4efef7964d30ca7cc32b04b362136f643489b1a55d861f8fb4f WatchSource:0}: Error finding container 5b7b0eac725eb4efef7964d30ca7cc32b04b362136f643489b1a55d861f8fb4f: Status 404 returned error can't find the container with id 5b7b0eac725eb4efef7964d30ca7cc32b04b362136f643489b1a55d861f8fb4f
Sep 16 14:31:21.744562 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:21.744520    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w" event={"ID":"27f90242-bf5f-46fc-8acc-28dfc45ce02f","Type":"ContainerStarted","Data":"c93db5b6d58217252c274f05e4b54d460bc9c40e088bc7c49babff9fadc26f1a"}
Sep 16 14:31:21.744562 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:21.744565    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w" event={"ID":"27f90242-bf5f-46fc-8acc-28dfc45ce02f","Type":"ContainerStarted","Data":"5b7b0eac725eb4efef7964d30ca7cc32b04b362136f643489b1a55d861f8fb4f"}
Sep 16 14:31:21.771619 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:21.771570    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w" podStartSLOduration=1.771552327 podStartE2EDuration="1.771552327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:31:21.769640257 +0000 UTC m=+1030.373265288" watchObservedRunningTime="2026-09-16 14:31:21.771552327 +0000 UTC m=+1030.375177358"
Sep 16 14:31:30.769178 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:30.769146    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"
Sep 16 14:31:30.769611 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:30.769199    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"
Sep 16 14:31:30.782027 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:30.782000    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"
Sep 16 14:31:30.804094 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:31:30.804057    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"
Sep 16 14:32:00.924148 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:00.924111    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7ce904-5vc7c"]
Sep 16 14:32:00.928361 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:00.928340    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7ce904-5vc7c"
Sep 16 14:32:00.931202 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:00.931181    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4jf\""
Sep 16 14:32:00.931324 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:00.931250    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:32:00.938924 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:00.938901    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7ce904-5vc7c"]
Sep 16 14:32:00.948797 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:00.948755    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5a5203b-78c4-4a31-923e-f5ffdad54ef1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-7ce904-5vc7c\" (UID: \"f5a5203b-78c4-4a31-923e-f5ffdad54ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7ce904-5vc7c"
Sep 16 14:32:01.049977 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:01.049937    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5a5203b-78c4-4a31-923e-f5ffdad54ef1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-7ce904-5vc7c\" (UID: \"f5a5203b-78c4-4a31-923e-f5ffdad54ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7ce904-5vc7c"
Sep 16 14:32:01.050758 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:01.050735    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5a5203b-78c4-4a31-923e-f5ffdad54ef1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-7ce904-5vc7c\" (UID: \"f5a5203b-78c4-4a31-923e-f5ffdad54ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7ce904-5vc7c"
Sep 16 14:32:01.241565 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:01.241469    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7ce904-5vc7c"
Sep 16 14:32:01.524501 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:01.524380    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7ce904-5vc7c"]
Sep 16 14:32:01.528266 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:32:01.528240    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5a5203b_78c4_4a31_923e_f5ffdad54ef1.slice/crio-b885bad55cfabdd68ad0fac6fa5fd19e8ce192344582bdb9f322f7c661245e36 WatchSource:0}: Error finding container b885bad55cfabdd68ad0fac6fa5fd19e8ce192344582bdb9f322f7c661245e36: Status 404 returned error can't find the container with id b885bad55cfabdd68ad0fac6fa5fd19e8ce192344582bdb9f322f7c661245e36
Sep 16 14:32:01.966111 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:01.966075    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7ce904-5vc7c" event={"ID":"f5a5203b-78c4-4a31-923e-f5ffdad54ef1","Type":"ContainerStarted","Data":"740259bc6966d614bcd8c4196fc8445d2f1ab833d9cb5eed449fde1dcd942a3a"}
Sep 16 14:32:01.966559 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:01.966133    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7ce904-5vc7c" event={"ID":"f5a5203b-78c4-4a31-923e-f5ffdad54ef1","Type":"ContainerStarted","Data":"b885bad55cfabdd68ad0fac6fa5fd19e8ce192344582bdb9f322f7c661245e36"}
Sep 16 14:32:01.984476 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:01.984422    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7ce904-5vc7c" podStartSLOduration=1.984403657 podStartE2EDuration="1.984403657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:32:01.982440026 +0000 UTC m=+1070.586065061" watchObservedRunningTime="2026-09-16 14:32:01.984403657 +0000 UTC m=+1070.588028680"
Sep 16 14:32:02.973524 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:02.973445    2587 generic.go:356] "Generic (PLEG): container finished" podID="f5a5203b-78c4-4a31-923e-f5ffdad54ef1" containerID="740259bc6966d614bcd8c4196fc8445d2f1ab833d9cb5eed449fde1dcd942a3a" exitCode=0
Sep 16 14:32:02.973893 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:02.973522    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7ce904-5vc7c" event={"ID":"f5a5203b-78c4-4a31-923e-f5ffdad54ef1","Type":"ContainerDied","Data":"740259bc6966d614bcd8c4196fc8445d2f1ab833d9cb5eed449fde1dcd942a3a"}
Sep 16 14:32:04.124760 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:04.124732    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7ce904-5vc7c"
Sep 16 14:32:04.181514 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:04.181481    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5a5203b-78c4-4a31-923e-f5ffdad54ef1-maas-api-service-ca\") pod \"f5a5203b-78c4-4a31-923e-f5ffdad54ef1\" (UID: \"f5a5203b-78c4-4a31-923e-f5ffdad54ef1\") "
Sep 16 14:32:04.181853 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:04.181827    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5a5203b-78c4-4a31-923e-f5ffdad54ef1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f5a5203b-78c4-4a31-923e-f5ffdad54ef1" (UID: "f5a5203b-78c4-4a31-923e-f5ffdad54ef1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:32:04.282582 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:04.282496    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5a5203b-78c4-4a31-923e-f5ffdad54ef1-maas-api-service-ca\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:32:04.986995 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:04.986963    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7ce904-5vc7c"
Sep 16 14:32:04.987206 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:04.986961    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7ce904-5vc7c" event={"ID":"f5a5203b-78c4-4a31-923e-f5ffdad54ef1","Type":"ContainerDied","Data":"b885bad55cfabdd68ad0fac6fa5fd19e8ce192344582bdb9f322f7c661245e36"}
Sep 16 14:32:04.987206 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:04.987093    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b885bad55cfabdd68ad0fac6fa5fd19e8ce192344582bdb9f322f7c661245e36"
Sep 16 14:32:10.838901 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:10.838864    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g"]
Sep 16 14:32:10.839372 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:10.839282    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5a5203b-78c4-4a31-923e-f5ffdad54ef1" containerName="revoke-keys"
Sep 16 14:32:10.839372 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:10.839294    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5a5203b-78c4-4a31-923e-f5ffdad54ef1" containerName="revoke-keys"
Sep 16 14:32:10.839372 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:10.839371    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5a5203b-78c4-4a31-923e-f5ffdad54ef1" containerName="revoke-keys"
Sep 16 14:32:10.842467 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:10.842449    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g"
Sep 16 14:32:10.845034 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:10.845006    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5e366590\""
Sep 16 14:32:10.845156 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:10.845102    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5e366590-dockercfg-ljh2d\""
Sep 16 14:32:10.855741 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:10.855709    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g"]
Sep 16 14:32:10.874682 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:10.874647    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr"]
Sep 16 14:32:10.878660 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:10.878630    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr"
Sep 16 14:32:10.892846 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:10.892817    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr"]
Sep 16 14:32:10.948478 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:10.948427    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24f6a99b-d1e4-4799-8b99-89dc96a576d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr\" (UID: \"24f6a99b-d1e4-4799-8b99-89dc96a576d5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr"
Sep 16 14:32:10.948648 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:10.948532    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87627dce-3159-4ad0-aa28-d515e6e510db-plugins-config-volume\") pod \"payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g\" (UID: \"87627dce-3159-4ad0-aa28-d515e6e510db\") " pod="openshift-ingress/payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g"
Sep 16 14:32:10.948648 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:10.948608    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9rz8\" (UniqueName: \"kubernetes.io/projected/87627dce-3159-4ad0-aa28-d515e6e510db-kube-api-access-v9rz8\") pod \"payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g\" (UID: \"87627dce-3159-4ad0-aa28-d515e6e510db\") " pod="openshift-ingress/payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g"
Sep 16 14:32:10.948763 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:10.948669    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkrr7\" (UniqueName: \"kubernetes.io/projected/24f6a99b-d1e4-4799-8b99-89dc96a576d5-kube-api-access-dkrr7\") pod \"payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr\" (UID: \"24f6a99b-d1e4-4799-8b99-89dc96a576d5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr"
Sep 16 14:32:11.049327 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.049284    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87627dce-3159-4ad0-aa28-d515e6e510db-plugins-config-volume\") pod \"payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g\" (UID: \"87627dce-3159-4ad0-aa28-d515e6e510db\") " pod="openshift-ingress/payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g"
Sep 16 14:32:11.049532 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.049343    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9rz8\" (UniqueName: \"kubernetes.io/projected/87627dce-3159-4ad0-aa28-d515e6e510db-kube-api-access-v9rz8\") pod \"payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g\" (UID: \"87627dce-3159-4ad0-aa28-d515e6e510db\") " pod="openshift-ingress/payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g"
Sep 16 14:32:11.049532 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.049421    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkrr7\" (UniqueName: \"kubernetes.io/projected/24f6a99b-d1e4-4799-8b99-89dc96a576d5-kube-api-access-dkrr7\") pod \"payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr\" (UID: \"24f6a99b-d1e4-4799-8b99-89dc96a576d5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr"
Sep 16 14:32:11.049532 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.049518    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24f6a99b-d1e4-4799-8b99-89dc96a576d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr\" (UID: \"24f6a99b-d1e4-4799-8b99-89dc96a576d5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr"
Sep 16 14:32:11.050230 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.050206    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24f6a99b-d1e4-4799-8b99-89dc96a576d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr\" (UID: \"24f6a99b-d1e4-4799-8b99-89dc96a576d5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr"
Sep 16 14:32:11.050360 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.050244    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87627dce-3159-4ad0-aa28-d515e6e510db-plugins-config-volume\") pod \"payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g\" (UID: \"87627dce-3159-4ad0-aa28-d515e6e510db\") " pod="openshift-ingress/payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g"
Sep 16 14:32:11.062808 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.062774    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9rz8\" (UniqueName: \"kubernetes.io/projected/87627dce-3159-4ad0-aa28-d515e6e510db-kube-api-access-v9rz8\") pod \"payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g\" (UID: \"87627dce-3159-4ad0-aa28-d515e6e510db\") " pod="openshift-ingress/payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g"
Sep 16 14:32:11.067598 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.067565    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkrr7\" (UniqueName: \"kubernetes.io/projected/24f6a99b-d1e4-4799-8b99-89dc96a576d5-kube-api-access-dkrr7\") pod \"payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr\" (UID: \"24f6a99b-d1e4-4799-8b99-89dc96a576d5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr"
Sep 16 14:32:11.093688 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.093596    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4"]
Sep 16 14:32:11.093828 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.093808    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4" podUID="f96a7b4f-a182-4ef1-aa53-adb433fb7457" containerName="payload-processing" containerID="cri-o://6d1ff189ccac1e960b5f48ee2c8cd124faa18b6c6573f3ec0c08eedb16579e6c" gracePeriod=30
Sep 16 14:32:11.103830 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.102652    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4"]
Sep 16 14:32:11.103830 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.103126    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4" podUID="e71bd69e-4c4c-433a-85e2-c7b7d3465e43" containerName="payload-pre-processing" containerID="cri-o://0e552e71c76993840aa2274c1900f5fb72b439c5f81baf795c6062e6311c2b29" gracePeriod=30
Sep 16 14:32:11.153346 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.153314    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g"
Sep 16 14:32:11.189614 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.189579    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr"
Sep 16 14:32:11.402985 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.402955    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4"
Sep 16 14:32:11.430329 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.430291    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4"
Sep 16 14:32:11.448534 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.448507    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g"]
Sep 16 14:32:11.453472 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.453425    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96a7b4f-a182-4ef1-aa53-adb433fb7457-plugins-config-volume\") pod \"f96a7b4f-a182-4ef1-aa53-adb433fb7457\" (UID: \"f96a7b4f-a182-4ef1-aa53-adb433fb7457\") "
Sep 16 14:32:11.453621 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.453474    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25rnj\" (UniqueName: \"kubernetes.io/projected/f96a7b4f-a182-4ef1-aa53-adb433fb7457-kube-api-access-25rnj\") pod \"f96a7b4f-a182-4ef1-aa53-adb433fb7457\" (UID: \"f96a7b4f-a182-4ef1-aa53-adb433fb7457\") "
Sep 16 14:32:11.454332 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.454297    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f96a7b4f-a182-4ef1-aa53-adb433fb7457-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f96a7b4f-a182-4ef1-aa53-adb433fb7457" (UID: "f96a7b4f-a182-4ef1-aa53-adb433fb7457"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:32:11.457596 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.457568    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f96a7b4f-a182-4ef1-aa53-adb433fb7457-kube-api-access-25rnj" (OuterVolumeSpecName: "kube-api-access-25rnj") pod "f96a7b4f-a182-4ef1-aa53-adb433fb7457" (UID: "f96a7b4f-a182-4ef1-aa53-adb433fb7457"). InnerVolumeSpecName "kube-api-access-25rnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:32:11.458222 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.458202    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7ce904-5vc7c"]
Sep 16 14:32:11.461607 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.461581    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7ce904-5vc7c"]
Sep 16 14:32:11.503501 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.503471    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr"]
Sep 16 14:32:11.507509 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:32:11.507486    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f6a99b_d1e4_4799_8b99_89dc96a576d5.slice/crio-52949cc94972bb2e25beab0de15bffb95966937818262802f976a752e0e900d1 WatchSource:0}: Error finding container 52949cc94972bb2e25beab0de15bffb95966937818262802f976a752e0e900d1: Status 404 returned error can't find the container with id 52949cc94972bb2e25beab0de15bffb95966937818262802f976a752e0e900d1
Sep 16 14:32:11.554870 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.554835    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e71bd69e-4c4c-433a-85e2-c7b7d3465e43-plugins-config-volume\") pod \"e71bd69e-4c4c-433a-85e2-c7b7d3465e43\" (UID: \"e71bd69e-4c4c-433a-85e2-c7b7d3465e43\") "
Sep 16 14:32:11.555079 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.554902    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7r4g\" (UniqueName: \"kubernetes.io/projected/e71bd69e-4c4c-433a-85e2-c7b7d3465e43-kube-api-access-j7r4g\") pod \"e71bd69e-4c4c-433a-85e2-c7b7d3465e43\" (UID: \"e71bd69e-4c4c-433a-85e2-c7b7d3465e43\") "
Sep 16 14:32:11.555196 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.555150    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96a7b4f-a182-4ef1-aa53-adb433fb7457-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:32:11.555196 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.555161    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25rnj\" (UniqueName: \"kubernetes.io/projected/f96a7b4f-a182-4ef1-aa53-adb433fb7457-kube-api-access-25rnj\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:32:11.555346 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.555264    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e71bd69e-4c4c-433a-85e2-c7b7d3465e43-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e71bd69e-4c4c-433a-85e2-c7b7d3465e43" (UID: "e71bd69e-4c4c-433a-85e2-c7b7d3465e43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:32:11.557424 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.557364    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71bd69e-4c4c-433a-85e2-c7b7d3465e43-kube-api-access-j7r4g" (OuterVolumeSpecName: "kube-api-access-j7r4g") pod "e71bd69e-4c4c-433a-85e2-c7b7d3465e43" (UID: "e71bd69e-4c4c-433a-85e2-c7b7d3465e43"). InnerVolumeSpecName "kube-api-access-j7r4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:32:11.656506 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.656469    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e71bd69e-4c4c-433a-85e2-c7b7d3465e43-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:32:11.656506 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.656501    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7r4g\" (UniqueName: \"kubernetes.io/projected/e71bd69e-4c4c-433a-85e2-c7b7d3465e43-kube-api-access-j7r4g\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:32:11.904378 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:11.904339    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5a5203b-78c4-4a31-923e-f5ffdad54ef1" path="/var/lib/kubelet/pods/f5a5203b-78c4-4a31-923e-f5ffdad54ef1/volumes"
Sep 16 14:32:12.032038 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:12.031988    2587 generic.go:356] "Generic (PLEG): container finished" podID="e71bd69e-4c4c-433a-85e2-c7b7d3465e43" containerID="0e552e71c76993840aa2274c1900f5fb72b439c5f81baf795c6062e6311c2b29" exitCode=0
Sep 16 14:32:12.032038 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:12.032038    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4"
Sep 16 14:32:12.032292 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:12.032075    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4" event={"ID":"e71bd69e-4c4c-433a-85e2-c7b7d3465e43","Type":"ContainerDied","Data":"0e552e71c76993840aa2274c1900f5fb72b439c5f81baf795c6062e6311c2b29"}
Sep 16 14:32:12.032292 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:12.032148    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4" event={"ID":"e71bd69e-4c4c-433a-85e2-c7b7d3465e43","Type":"ContainerDied","Data":"f9cf84e808721a5fd1fb7baf05a5df641414207864f334ec92eb9d67e216cde4"}
Sep 16 14:32:12.032292 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:12.032187    2587 scope.go:117] "RemoveContainer" containerID="0e552e71c76993840aa2274c1900f5fb72b439c5f81baf795c6062e6311c2b29"
Sep 16 14:32:12.034862 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:12.034829    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g" event={"ID":"87627dce-3159-4ad0-aa28-d515e6e510db","Type":"ContainerStarted","Data":"1ac87a2c526b90e1f17e88b62c2a21f7bdc699c1dae05978b0aa2826962350e6"}
Sep 16 14:32:12.035003 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:12.034887    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g" event={"ID":"87627dce-3159-4ad0-aa28-d515e6e510db","Type":"ContainerStarted","Data":"b1b6bb4664adc1fffdc2acaa8e53aabfa3162ff0967e3094d4ee814227ce4992"}
Sep 16 14:32:12.035638 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:12.035615    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g"
Sep 16 14:32:12.038636 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:12.038607    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr" event={"ID":"24f6a99b-d1e4-4799-8b99-89dc96a576d5","Type":"ContainerStarted","Data":"3cee1482f6657e0c7a0bdcadc69fc34c920f42f0ef5432d7fde1ff5a336e0b9f"}
Sep 16 14:32:12.038764 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:12.038671    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr" event={"ID":"24f6a99b-d1e4-4799-8b99-89dc96a576d5","Type":"ContainerStarted","Data":"52949cc94972bb2e25beab0de15bffb95966937818262802f976a752e0e900d1"}
Sep 16 14:32:12.038764 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:12.038703    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr"
Sep 16 14:32:12.040693 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:12.040673    2587 generic.go:356] "Generic (PLEG): container finished" podID="f96a7b4f-a182-4ef1-aa53-adb433fb7457" containerID="6d1ff189ccac1e960b5f48ee2c8cd124faa18b6c6573f3ec0c08eedb16579e6c" exitCode=0
Sep 16 14:32:12.040813 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:12.040701    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4" event={"ID":"f96a7b4f-a182-4ef1-aa53-adb433fb7457","Type":"ContainerDied","Data":"6d1ff189ccac1e960b5f48ee2c8cd124faa18b6c6573f3ec0c08eedb16579e6c"}
Sep 16 14:32:12.040813 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:12.040719    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4"
Sep 16 14:32:12.040813 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:12.040732    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4" event={"ID":"f96a7b4f-a182-4ef1-aa53-adb433fb7457","Type":"ContainerDied","Data":"e340902a1a315d48de747bcbcee563f8975dec3c878ea8b00d1bc36a8d9f8453"}
Sep 16 14:32:12.045031 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:12.045006    2587 scope.go:117] "RemoveContainer" containerID="0e552e71c76993840aa2274c1900f5fb72b439c5f81baf795c6062e6311c2b29"
Sep 16 14:32:12.045367 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:32:12.045339    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e552e71c76993840aa2274c1900f5fb72b439c5f81baf795c6062e6311c2b29\": container with ID starting with 0e552e71c76993840aa2274c1900f5fb72b439c5f81baf795c6062e6311c2b29 not found: ID does not exist" containerID="0e552e71c76993840aa2274c1900f5fb72b439c5f81baf795c6062e6311c2b29"
Sep 16 14:32:12.045485 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:12.045380    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e552e71c76993840aa2274c1900f5fb72b439c5f81baf795c6062e6311c2b29"} err="failed to get container status \"0e552e71c76993840aa2274c1900f5fb72b439c5f81baf795c6062e6311c2b29\": rpc error: code = NotFound desc = could not find container \"0e552e71c76993840aa2274c1900f5fb72b439c5f81baf795c6062e6311c2b29\": container with ID starting with 0e552e71c76993840aa2274c1900f5fb72b439c5f81baf795c6062e6311c2b29 not found: ID does not exist"
Sep 16 14:32:12.045485 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:12.045423    2587 scope.go:117] "RemoveContainer" containerID="6d1ff189ccac1e960b5f48ee2c8cd124faa18b6c6573f3ec0c08eedb16579e6c"
Sep 16 14:32:12.054977 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:12.054915    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g" podStartSLOduration=2.054897481 podStartE2EDuration="2.054897481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:32:12.052889677 +0000 UTC m=+1080.656514710" watchObservedRunningTime="2026-09-16 14:32:12.054897481 +0000 UTC m=+1080.658522514"
Sep 16 14:32:12.055068 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:12.055005    2587 scope.go:117] "RemoveContainer" containerID="6d1ff189ccac1e960b5f48ee2c8cd124faa18b6c6573f3ec0c08eedb16579e6c"
Sep 16 14:32:12.055453 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:32:12.055413    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d1ff189ccac1e960b5f48ee2c8cd124faa18b6c6573f3ec0c08eedb16579e6c\": container with ID starting with 6d1ff189ccac1e960b5f48ee2c8cd124faa18b6c6573f3ec0c08eedb16579e6c not found: ID does not exist" containerID="6d1ff189ccac1e960b5f48ee2c8cd124faa18b6c6573f3ec0c08eedb16579e6c"
Sep 16 14:32:12.055543 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:12.055445    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d1ff189ccac1e960b5f48ee2c8cd124faa18b6c6573f3ec0c08eedb16579e6c"} err="failed to get container status \"6d1ff189ccac1e960b5f48ee2c8cd124faa18b6c6573f3ec0c08eedb16579e6c\": rpc error: code = NotFound desc = could not find container \"6d1ff189ccac1e960b5f48ee2c8cd124faa18b6c6573f3ec0c08eedb16579e6c\": container with ID starting with 6d1ff189ccac1e960b5f48ee2c8cd124faa18b6c6573f3ec0c08eedb16579e6c not found: ID does not exist"
Sep 16 14:32:12.067830 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:12.067803    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4"]
Sep 16 14:32:12.071921 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:12.071894    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7ce904-5fcc754865-46mq4"]
Sep 16 14:32:12.085986 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:12.085939    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr" podStartSLOduration=2.085916832 podStartE2EDuration="2.085916832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:32:12.085640388 +0000 UTC m=+1080.689265421" watchObservedRunningTime="2026-09-16 14:32:12.085916832 +0000 UTC m=+1080.689541866"
Sep 16 14:32:12.100258 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:12.100213    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4"]
Sep 16 14:32:12.105744 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:12.105718    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7ce904-5978f7886-qmgs4"]
Sep 16 14:32:13.903549 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:13.903513    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e71bd69e-4c4c-433a-85e2-c7b7d3465e43" path="/var/lib/kubelet/pods/e71bd69e-4c4c-433a-85e2-c7b7d3465e43/volumes"
Sep 16 14:32:13.903938 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:13.903874    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f96a7b4f-a182-4ef1-aa53-adb433fb7457" path="/var/lib/kubelet/pods/f96a7b4f-a182-4ef1-aa53-adb433fb7457/volumes"
Sep 16 14:32:23.052289 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:23.052253    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr"
Sep 16 14:32:24.061267 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:24.061236    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g"
Sep 16 14:32:56.114647 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:56.114609    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g"]
Sep 16 14:32:56.115218 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:56.114873    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g" podUID="87627dce-3159-4ad0-aa28-d515e6e510db" containerName="payload-processing" containerID="cri-o://1ac87a2c526b90e1f17e88b62c2a21f7bdc699c1dae05978b0aa2826962350e6" gracePeriod=30
Sep 16 14:32:56.123117 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:56.122468    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr"]
Sep 16 14:32:56.123117 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:56.122713    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr" podUID="24f6a99b-d1e4-4799-8b99-89dc96a576d5" containerName="payload-pre-processing" containerID="cri-o://3cee1482f6657e0c7a0bdcadc69fc34c920f42f0ef5432d7fde1ff5a336e0b9f" gracePeriod=30
Sep 16 14:32:56.298670 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:56.298490    2587 generic.go:356] "Generic (PLEG): container finished" podID="87627dce-3159-4ad0-aa28-d515e6e510db" containerID="1ac87a2c526b90e1f17e88b62c2a21f7bdc699c1dae05978b0aa2826962350e6" exitCode=0
Sep 16 14:32:56.298670 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:56.298581    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g" event={"ID":"87627dce-3159-4ad0-aa28-d515e6e510db","Type":"ContainerDied","Data":"1ac87a2c526b90e1f17e88b62c2a21f7bdc699c1dae05978b0aa2826962350e6"}
Sep 16 14:32:56.301166 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:56.300976    2587 generic.go:356] "Generic (PLEG): container finished" podID="24f6a99b-d1e4-4799-8b99-89dc96a576d5" containerID="3cee1482f6657e0c7a0bdcadc69fc34c920f42f0ef5432d7fde1ff5a336e0b9f" exitCode=0
Sep 16 14:32:56.301166 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:56.301082    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr" event={"ID":"24f6a99b-d1e4-4799-8b99-89dc96a576d5","Type":"ContainerDied","Data":"3cee1482f6657e0c7a0bdcadc69fc34c920f42f0ef5432d7fde1ff5a336e0b9f"}
Sep 16 14:32:56.378095 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:56.378032    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g"
Sep 16 14:32:56.419212 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:56.419187    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr"
Sep 16 14:32:56.514308 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:56.514271    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkrr7\" (UniqueName: \"kubernetes.io/projected/24f6a99b-d1e4-4799-8b99-89dc96a576d5-kube-api-access-dkrr7\") pod \"24f6a99b-d1e4-4799-8b99-89dc96a576d5\" (UID: \"24f6a99b-d1e4-4799-8b99-89dc96a576d5\") "
Sep 16 14:32:56.514554 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:56.514454    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9rz8\" (UniqueName: \"kubernetes.io/projected/87627dce-3159-4ad0-aa28-d515e6e510db-kube-api-access-v9rz8\") pod \"87627dce-3159-4ad0-aa28-d515e6e510db\" (UID: \"87627dce-3159-4ad0-aa28-d515e6e510db\") "
Sep 16 14:32:56.514554 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:56.514509    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87627dce-3159-4ad0-aa28-d515e6e510db-plugins-config-volume\") pod \"87627dce-3159-4ad0-aa28-d515e6e510db\" (UID: \"87627dce-3159-4ad0-aa28-d515e6e510db\") "
Sep 16 14:32:56.514683 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:56.514558    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24f6a99b-d1e4-4799-8b99-89dc96a576d5-plugins-config-volume\") pod \"24f6a99b-d1e4-4799-8b99-89dc96a576d5\" (UID: \"24f6a99b-d1e4-4799-8b99-89dc96a576d5\") "
Sep 16 14:32:56.514929 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:56.514901    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87627dce-3159-4ad0-aa28-d515e6e510db-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "87627dce-3159-4ad0-aa28-d515e6e510db" (UID: "87627dce-3159-4ad0-aa28-d515e6e510db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:32:56.515001 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:56.514939    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24f6a99b-d1e4-4799-8b99-89dc96a576d5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "24f6a99b-d1e4-4799-8b99-89dc96a576d5" (UID: "24f6a99b-d1e4-4799-8b99-89dc96a576d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:32:56.516552 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:56.516532    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f6a99b-d1e4-4799-8b99-89dc96a576d5-kube-api-access-dkrr7" (OuterVolumeSpecName: "kube-api-access-dkrr7") pod "24f6a99b-d1e4-4799-8b99-89dc96a576d5" (UID: "24f6a99b-d1e4-4799-8b99-89dc96a576d5"). InnerVolumeSpecName "kube-api-access-dkrr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:32:56.516682 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:56.516664    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87627dce-3159-4ad0-aa28-d515e6e510db-kube-api-access-v9rz8" (OuterVolumeSpecName: "kube-api-access-v9rz8") pod "87627dce-3159-4ad0-aa28-d515e6e510db" (UID: "87627dce-3159-4ad0-aa28-d515e6e510db"). InnerVolumeSpecName "kube-api-access-v9rz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:32:56.615967 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:56.615926    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24f6a99b-d1e4-4799-8b99-89dc96a576d5-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:32:56.615967 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:56.615958    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkrr7\" (UniqueName: \"kubernetes.io/projected/24f6a99b-d1e4-4799-8b99-89dc96a576d5-kube-api-access-dkrr7\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:32:56.615967 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:56.615971    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9rz8\" (UniqueName: \"kubernetes.io/projected/87627dce-3159-4ad0-aa28-d515e6e510db-kube-api-access-v9rz8\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:32:56.616209 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:56.615985    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87627dce-3159-4ad0-aa28-d515e6e510db-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:32:57.308037 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:57.308005    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g"
Sep 16 14:32:57.308514 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:57.308003    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g" event={"ID":"87627dce-3159-4ad0-aa28-d515e6e510db","Type":"ContainerDied","Data":"b1b6bb4664adc1fffdc2acaa8e53aabfa3162ff0967e3094d4ee814227ce4992"}
Sep 16 14:32:57.308514 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:57.308152    2587 scope.go:117] "RemoveContainer" containerID="1ac87a2c526b90e1f17e88b62c2a21f7bdc699c1dae05978b0aa2826962350e6"
Sep 16 14:32:57.309566 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:57.309527    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr" event={"ID":"24f6a99b-d1e4-4799-8b99-89dc96a576d5","Type":"ContainerDied","Data":"52949cc94972bb2e25beab0de15bffb95966937818262802f976a752e0e900d1"}
Sep 16 14:32:57.309566 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:57.309552    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr"
Sep 16 14:32:57.323188 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:57.323043    2587 scope.go:117] "RemoveContainer" containerID="3cee1482f6657e0c7a0bdcadc69fc34c920f42f0ef5432d7fde1ff5a336e0b9f"
Sep 16 14:32:57.343086 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:57.343060    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr"]
Sep 16 14:32:57.347762 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:57.347736    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5e366590-7db9dc67c4-ls8lr"]
Sep 16 14:32:57.358832 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:57.358804    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g"]
Sep 16 14:32:57.363057 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:57.363027    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5e366590-8448fbddd9-ctg5g"]
Sep 16 14:32:57.903010 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:57.902969    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24f6a99b-d1e4-4799-8b99-89dc96a576d5" path="/var/lib/kubelet/pods/24f6a99b-d1e4-4799-8b99-89dc96a576d5/volumes"
Sep 16 14:32:57.903461 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:32:57.903441    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87627dce-3159-4ad0-aa28-d515e6e510db" path="/var/lib/kubelet/pods/87627dce-3159-4ad0-aa28-d515e6e510db/volumes"
Sep 16 14:33:27.750768 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:27.750732    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk"]
Sep 16 14:33:27.751251 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:27.751133    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24f6a99b-d1e4-4799-8b99-89dc96a576d5" containerName="payload-pre-processing"
Sep 16 14:33:27.751251 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:27.751145    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="24f6a99b-d1e4-4799-8b99-89dc96a576d5" containerName="payload-pre-processing"
Sep 16 14:33:27.751251 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:27.751162    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f96a7b4f-a182-4ef1-aa53-adb433fb7457" containerName="payload-processing"
Sep 16 14:33:27.751251 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:27.751168    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="f96a7b4f-a182-4ef1-aa53-adb433fb7457" containerName="payload-processing"
Sep 16 14:33:27.751251 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:27.751176    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87627dce-3159-4ad0-aa28-d515e6e510db" containerName="payload-processing"
Sep 16 14:33:27.751251 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:27.751181    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="87627dce-3159-4ad0-aa28-d515e6e510db" containerName="payload-processing"
Sep 16 14:33:27.751251 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:27.751192    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e71bd69e-4c4c-433a-85e2-c7b7d3465e43" containerName="payload-pre-processing"
Sep 16 14:33:27.751251 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:27.751197    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="e71bd69e-4c4c-433a-85e2-c7b7d3465e43" containerName="payload-pre-processing"
Sep 16 14:33:27.751251 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:27.751252    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="e71bd69e-4c4c-433a-85e2-c7b7d3465e43" containerName="payload-pre-processing"
Sep 16 14:33:27.751600 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:27.751262    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="24f6a99b-d1e4-4799-8b99-89dc96a576d5" containerName="payload-pre-processing"
Sep 16 14:33:27.751600 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:27.751270    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="87627dce-3159-4ad0-aa28-d515e6e510db" containerName="payload-processing"
Sep 16 14:33:27.751600 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:27.751275    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="f96a7b4f-a182-4ef1-aa53-adb433fb7457" containerName="payload-processing"
Sep 16 14:33:27.755735 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:27.755718    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk"
Sep 16 14:33:27.758675 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:27.758646    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-4c2787-dockercfg-fx9kp\""
Sep 16 14:33:27.758820 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:27.758707    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-4c2787\""
Sep 16 14:33:27.765433 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:27.765410    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk"]
Sep 16 14:33:27.783857 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:27.783830    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf"]
Sep 16 14:33:27.787319 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:27.787300    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf"
Sep 16 14:33:27.797523 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:27.797498    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf"]
Sep 16 14:33:27.824029 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:27.823997    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk\" (UID: \"183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk"
Sep 16 14:33:27.824266 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:27.824246    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2stc5\" (UniqueName: \"kubernetes.io/projected/183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a-kube-api-access-2stc5\") pod \"payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk\" (UID: \"183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk"
Sep 16 14:33:27.925364 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:27.925320    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b4836a3-de08-4aa2-a2df-c519ae6f8c0e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf\" (UID: \"4b4836a3-de08-4aa2-a2df-c519ae6f8c0e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf"
Sep 16 14:33:27.925595 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:27.925377    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk\" (UID: \"183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk"
Sep 16 14:33:27.925595 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:27.925463    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2stc5\" (UniqueName: \"kubernetes.io/projected/183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a-kube-api-access-2stc5\") pod \"payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk\" (UID: \"183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk"
Sep 16 14:33:27.925595 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:27.925486    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7gbf\" (UniqueName: \"kubernetes.io/projected/4b4836a3-de08-4aa2-a2df-c519ae6f8c0e-kube-api-access-n7gbf\") pod \"payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf\" (UID: \"4b4836a3-de08-4aa2-a2df-c519ae6f8c0e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf"
Sep 16 14:33:27.926153 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:27.926128    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk\" (UID: \"183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk"
Sep 16 14:33:27.936839 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:27.936815    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2stc5\" (UniqueName: \"kubernetes.io/projected/183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a-kube-api-access-2stc5\") pod \"payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk\" (UID: \"183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk"
Sep 16 14:33:28.026241 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:28.026150    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b4836a3-de08-4aa2-a2df-c519ae6f8c0e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf\" (UID: \"4b4836a3-de08-4aa2-a2df-c519ae6f8c0e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf"
Sep 16 14:33:28.026476 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:28.026374    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7gbf\" (UniqueName: \"kubernetes.io/projected/4b4836a3-de08-4aa2-a2df-c519ae6f8c0e-kube-api-access-n7gbf\") pod \"payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf\" (UID: \"4b4836a3-de08-4aa2-a2df-c519ae6f8c0e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf"
Sep 16 14:33:28.026990 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:28.026968    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b4836a3-de08-4aa2-a2df-c519ae6f8c0e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf\" (UID: \"4b4836a3-de08-4aa2-a2df-c519ae6f8c0e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf"
Sep 16 14:33:28.035708 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:28.035679    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7gbf\" (UniqueName: \"kubernetes.io/projected/4b4836a3-de08-4aa2-a2df-c519ae6f8c0e-kube-api-access-n7gbf\") pod \"payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf\" (UID: \"4b4836a3-de08-4aa2-a2df-c519ae6f8c0e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf"
Sep 16 14:33:28.067295 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:28.067262    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk"
Sep 16 14:33:28.098601 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:28.098568    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf"
Sep 16 14:33:28.345320 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:28.345292    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk"]
Sep 16 14:33:28.347017 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:33:28.346990    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod183d6a8b_2b8c_4b0b_97c7_68a87bbe7a6a.slice/crio-931821feafbefe498a4b79092bb6d3ab17bf5b938ec4be8736accf31a567e04b WatchSource:0}: Error finding container 931821feafbefe498a4b79092bb6d3ab17bf5b938ec4be8736accf31a567e04b: Status 404 returned error can't find the container with id 931821feafbefe498a4b79092bb6d3ab17bf5b938ec4be8736accf31a567e04b
Sep 16 14:33:28.382049 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:28.382020    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf"]
Sep 16 14:33:28.383572 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:33:28.383548    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4836a3_de08_4aa2_a2df_c519ae6f8c0e.slice/crio-0d039b05956fbd51162f3b5a4ad6c2e595955f3d1665eabb3d6600736409b318 WatchSource:0}: Error finding container 0d039b05956fbd51162f3b5a4ad6c2e595955f3d1665eabb3d6600736409b318: Status 404 returned error can't find the container with id 0d039b05956fbd51162f3b5a4ad6c2e595955f3d1665eabb3d6600736409b318
Sep 16 14:33:28.481229 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:28.481176    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk" event={"ID":"183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a","Type":"ContainerStarted","Data":"33605731eaf1ab09a5cb25e238bd033c7c3a50880e8e899d7f1146c4dc988181"}
Sep 16 14:33:28.481439 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:28.481254    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk"
Sep 16 14:33:28.481439 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:28.481277    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk" event={"ID":"183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a","Type":"ContainerStarted","Data":"931821feafbefe498a4b79092bb6d3ab17bf5b938ec4be8736accf31a567e04b"}
Sep 16 14:33:28.483008 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:28.482985    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf" event={"ID":"4b4836a3-de08-4aa2-a2df-c519ae6f8c0e","Type":"ContainerStarted","Data":"8266609f6ab1ad9c3d020e588c389c91c296506a1fa1a2449d95c485082f171d"}
Sep 16 14:33:28.483112 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:28.483026    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf" event={"ID":"4b4836a3-de08-4aa2-a2df-c519ae6f8c0e","Type":"ContainerStarted","Data":"0d039b05956fbd51162f3b5a4ad6c2e595955f3d1665eabb3d6600736409b318"}
Sep 16 14:33:28.483112 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:28.483057    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf"
Sep 16 14:33:28.500840 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:28.500793    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk" podStartSLOduration=1.500778059 podStartE2EDuration="1.500778059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:33:28.498759812 +0000 UTC m=+1157.102384844" watchObservedRunningTime="2026-09-16 14:33:28.500778059 +0000 UTC m=+1157.104403091"
Sep 16 14:33:28.516451 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:28.516381    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf" podStartSLOduration=1.516366034 podStartE2EDuration="1.516366034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:33:28.515608632 +0000 UTC m=+1157.119233664" watchObservedRunningTime="2026-09-16 14:33:28.516366034 +0000 UTC m=+1157.119991067"
Sep 16 14:33:32.300309 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:32.300269    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"]
Sep 16 14:33:32.304076 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:32.304055    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"
Sep 16 14:33:32.306811 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:32.306789    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ec80c7\"/\"kube-root-ca.crt\""
Sep 16 14:33:32.307855 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:32.307834    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ec80c7\"/\"noten-model-ec80c7-kserve-self-signed-certs\""
Sep 16 14:33:32.307923 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:32.307877    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ec80c7\"/\"default-dockercfg-j4zh6\""
Sep 16 14:33:32.307988 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:32.307877    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ec80c7\"/\"openshift-service-ca.crt\""
Sep 16 14:33:32.315160 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:32.315132    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"]
Sep 16 14:33:32.469264 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:32.469226    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de3dae52-deae-4f5f-bc21-6c647914412e-tls-certs\") pod \"noten-model-ec80c7-kserve-5d77f7d95-qvs5l\" (UID: \"de3dae52-deae-4f5f-bc21-6c647914412e\") " pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"
Sep 16 14:33:32.469467 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:32.469282    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de3dae52-deae-4f5f-bc21-6c647914412e-tmp-dir\") pod \"noten-model-ec80c7-kserve-5d77f7d95-qvs5l\" (UID: \"de3dae52-deae-4f5f-bc21-6c647914412e\") " pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"
Sep 16 14:33:32.469467 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:32.469304    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de3dae52-deae-4f5f-bc21-6c647914412e-dshm\") pod \"noten-model-ec80c7-kserve-5d77f7d95-qvs5l\" (UID: \"de3dae52-deae-4f5f-bc21-6c647914412e\") " pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"
Sep 16 14:33:32.469467 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:32.469426    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztvtg\" (UniqueName: \"kubernetes.io/projected/de3dae52-deae-4f5f-bc21-6c647914412e-kube-api-access-ztvtg\") pod \"noten-model-ec80c7-kserve-5d77f7d95-qvs5l\" (UID: \"de3dae52-deae-4f5f-bc21-6c647914412e\") " pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"
Sep 16 14:33:32.469619 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:32.469478    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de3dae52-deae-4f5f-bc21-6c647914412e-home\") pod \"noten-model-ec80c7-kserve-5d77f7d95-qvs5l\" (UID: \"de3dae52-deae-4f5f-bc21-6c647914412e\") " pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"
Sep 16 14:33:32.469619 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:32.469525    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de3dae52-deae-4f5f-bc21-6c647914412e-model-cache\") pod \"noten-model-ec80c7-kserve-5d77f7d95-qvs5l\" (UID: \"de3dae52-deae-4f5f-bc21-6c647914412e\") " pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"
Sep 16 14:33:32.570977 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:32.570889    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de3dae52-deae-4f5f-bc21-6c647914412e-tls-certs\") pod \"noten-model-ec80c7-kserve-5d77f7d95-qvs5l\" (UID: \"de3dae52-deae-4f5f-bc21-6c647914412e\") " pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"
Sep 16 14:33:32.570977 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:32.570941    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de3dae52-deae-4f5f-bc21-6c647914412e-tmp-dir\") pod \"noten-model-ec80c7-kserve-5d77f7d95-qvs5l\" (UID: \"de3dae52-deae-4f5f-bc21-6c647914412e\") " pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"
Sep 16 14:33:32.571183 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:32.571055    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de3dae52-deae-4f5f-bc21-6c647914412e-dshm\") pod \"noten-model-ec80c7-kserve-5d77f7d95-qvs5l\" (UID: \"de3dae52-deae-4f5f-bc21-6c647914412e\") " pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"
Sep 16 14:33:32.571183 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:32.571096    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztvtg\" (UniqueName: \"kubernetes.io/projected/de3dae52-deae-4f5f-bc21-6c647914412e-kube-api-access-ztvtg\") pod \"noten-model-ec80c7-kserve-5d77f7d95-qvs5l\" (UID: \"de3dae52-deae-4f5f-bc21-6c647914412e\") " pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"
Sep 16 14:33:32.571183 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:32.571129    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de3dae52-deae-4f5f-bc21-6c647914412e-home\") pod \"noten-model-ec80c7-kserve-5d77f7d95-qvs5l\" (UID: \"de3dae52-deae-4f5f-bc21-6c647914412e\") " pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"
Sep 16 14:33:32.571333 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:32.571185    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de3dae52-deae-4f5f-bc21-6c647914412e-model-cache\") pod \"noten-model-ec80c7-kserve-5d77f7d95-qvs5l\" (UID: \"de3dae52-deae-4f5f-bc21-6c647914412e\") " pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"
Sep 16 14:33:32.571385 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:32.571330    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de3dae52-deae-4f5f-bc21-6c647914412e-tmp-dir\") pod \"noten-model-ec80c7-kserve-5d77f7d95-qvs5l\" (UID: \"de3dae52-deae-4f5f-bc21-6c647914412e\") " pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"
Sep 16 14:33:32.571514 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:32.571496    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de3dae52-deae-4f5f-bc21-6c647914412e-home\") pod \"noten-model-ec80c7-kserve-5d77f7d95-qvs5l\" (UID: \"de3dae52-deae-4f5f-bc21-6c647914412e\") " pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"
Sep 16 14:33:32.571586 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:32.571567    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de3dae52-deae-4f5f-bc21-6c647914412e-model-cache\") pod \"noten-model-ec80c7-kserve-5d77f7d95-qvs5l\" (UID: \"de3dae52-deae-4f5f-bc21-6c647914412e\") " pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"
Sep 16 14:33:32.573482 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:32.573454    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de3dae52-deae-4f5f-bc21-6c647914412e-dshm\") pod \"noten-model-ec80c7-kserve-5d77f7d95-qvs5l\" (UID: \"de3dae52-deae-4f5f-bc21-6c647914412e\") " pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"
Sep 16 14:33:32.573746 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:32.573726    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de3dae52-deae-4f5f-bc21-6c647914412e-tls-certs\") pod \"noten-model-ec80c7-kserve-5d77f7d95-qvs5l\" (UID: \"de3dae52-deae-4f5f-bc21-6c647914412e\") " pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"
Sep 16 14:33:32.582180 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:32.582154    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztvtg\" (UniqueName: \"kubernetes.io/projected/de3dae52-deae-4f5f-bc21-6c647914412e-kube-api-access-ztvtg\") pod \"noten-model-ec80c7-kserve-5d77f7d95-qvs5l\" (UID: \"de3dae52-deae-4f5f-bc21-6c647914412e\") " pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"
Sep 16 14:33:32.614730 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:32.614699    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"
Sep 16 14:33:32.891344 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:32.891311    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"]
Sep 16 14:33:32.893178 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:33:32.893144    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde3dae52_deae_4f5f_bc21_6c647914412e.slice/crio-c5c6904520995b45f6e9f5b91360f0b10fe4dcc309ff5318a458f315dc182092 WatchSource:0}: Error finding container c5c6904520995b45f6e9f5b91360f0b10fe4dcc309ff5318a458f315dc182092: Status 404 returned error can't find the container with id c5c6904520995b45f6e9f5b91360f0b10fe4dcc309ff5318a458f315dc182092
Sep 16 14:33:33.510713 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:33.510669    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l" event={"ID":"de3dae52-deae-4f5f-bc21-6c647914412e","Type":"ContainerStarted","Data":"d695b37b0b183618ae20e4383392e25c9c46bf1f62e3fcd5d1a0f69ac4346a55"}
Sep 16 14:33:33.510713 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:33.510712    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l" event={"ID":"de3dae52-deae-4f5f-bc21-6c647914412e","Type":"ContainerStarted","Data":"c5c6904520995b45f6e9f5b91360f0b10fe4dcc309ff5318a458f315dc182092"}
Sep 16 14:33:33.531276 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:33.531219    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l" podStartSLOduration=1.5312029169999999 podStartE2EDuration="1.531202917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:33:33.529864207 +0000 UTC m=+1162.133489250" watchObservedRunningTime="2026-09-16 14:33:33.531202917 +0000 UTC m=+1162.134827993"
Sep 16 14:33:39.490518 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:39.490488    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf"
Sep 16 14:33:39.490897 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:39.490559    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk"
Sep 16 14:33:41.799038 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:41.799006    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"]
Sep 16 14:33:41.802310 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:41.802292    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"
Sep 16 14:33:41.804810 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:41.804789    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-924916\"/\"distinct-w0-924916-kserve-self-signed-certs\""
Sep 16 14:33:41.811030 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:41.811006    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"]
Sep 16 14:33:41.863534 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:41.863501    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74140765-fb3c-4647-9dcc-4640ccf91709-model-cache\") pod \"distinct-w0-924916-kserve-85788cf7b8-7n68d\" (UID: \"74140765-fb3c-4647-9dcc-4640ccf91709\") " pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"
Sep 16 14:33:41.863534 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:41.863537    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74140765-fb3c-4647-9dcc-4640ccf91709-tmp-dir\") pod \"distinct-w0-924916-kserve-85788cf7b8-7n68d\" (UID: \"74140765-fb3c-4647-9dcc-4640ccf91709\") " pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"
Sep 16 14:33:41.863711 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:41.863575    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74140765-fb3c-4647-9dcc-4640ccf91709-home\") pod \"distinct-w0-924916-kserve-85788cf7b8-7n68d\" (UID: \"74140765-fb3c-4647-9dcc-4640ccf91709\") " pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"
Sep 16 14:33:41.863711 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:41.863597    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74140765-fb3c-4647-9dcc-4640ccf91709-tls-certs\") pod \"distinct-w0-924916-kserve-85788cf7b8-7n68d\" (UID: \"74140765-fb3c-4647-9dcc-4640ccf91709\") " pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"
Sep 16 14:33:41.863711 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:41.863666    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74140765-fb3c-4647-9dcc-4640ccf91709-dshm\") pod \"distinct-w0-924916-kserve-85788cf7b8-7n68d\" (UID: \"74140765-fb3c-4647-9dcc-4640ccf91709\") " pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"
Sep 16 14:33:41.863711 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:41.863705    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6pbh\" (UniqueName: \"kubernetes.io/projected/74140765-fb3c-4647-9dcc-4640ccf91709-kube-api-access-c6pbh\") pod \"distinct-w0-924916-kserve-85788cf7b8-7n68d\" (UID: \"74140765-fb3c-4647-9dcc-4640ccf91709\") " pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"
Sep 16 14:33:41.964459 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:41.964426    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74140765-fb3c-4647-9dcc-4640ccf91709-model-cache\") pod \"distinct-w0-924916-kserve-85788cf7b8-7n68d\" (UID: \"74140765-fb3c-4647-9dcc-4640ccf91709\") " pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"
Sep 16 14:33:41.964658 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:41.964473    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74140765-fb3c-4647-9dcc-4640ccf91709-tmp-dir\") pod \"distinct-w0-924916-kserve-85788cf7b8-7n68d\" (UID: \"74140765-fb3c-4647-9dcc-4640ccf91709\") " pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"
Sep 16 14:33:41.964658 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:41.964542    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74140765-fb3c-4647-9dcc-4640ccf91709-home\") pod \"distinct-w0-924916-kserve-85788cf7b8-7n68d\" (UID: \"74140765-fb3c-4647-9dcc-4640ccf91709\") " pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"
Sep 16 14:33:41.964658 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:41.964573    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74140765-fb3c-4647-9dcc-4640ccf91709-tls-certs\") pod \"distinct-w0-924916-kserve-85788cf7b8-7n68d\" (UID: \"74140765-fb3c-4647-9dcc-4640ccf91709\") " pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"
Sep 16 14:33:41.964658 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:41.964649    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74140765-fb3c-4647-9dcc-4640ccf91709-dshm\") pod \"distinct-w0-924916-kserve-85788cf7b8-7n68d\" (UID: \"74140765-fb3c-4647-9dcc-4640ccf91709\") " pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"
Sep 16 14:33:41.964862 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:41.964672    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6pbh\" (UniqueName: \"kubernetes.io/projected/74140765-fb3c-4647-9dcc-4640ccf91709-kube-api-access-c6pbh\") pod \"distinct-w0-924916-kserve-85788cf7b8-7n68d\" (UID: \"74140765-fb3c-4647-9dcc-4640ccf91709\") " pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"
Sep 16 14:33:41.964862 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:41.964831    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74140765-fb3c-4647-9dcc-4640ccf91709-model-cache\") pod \"distinct-w0-924916-kserve-85788cf7b8-7n68d\" (UID: \"74140765-fb3c-4647-9dcc-4640ccf91709\") " pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"
Sep 16 14:33:41.964966 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:41.964879    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74140765-fb3c-4647-9dcc-4640ccf91709-tmp-dir\") pod \"distinct-w0-924916-kserve-85788cf7b8-7n68d\" (UID: \"74140765-fb3c-4647-9dcc-4640ccf91709\") " pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"
Sep 16 14:33:41.964966 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:41.964945    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74140765-fb3c-4647-9dcc-4640ccf91709-home\") pod \"distinct-w0-924916-kserve-85788cf7b8-7n68d\" (UID: \"74140765-fb3c-4647-9dcc-4640ccf91709\") " pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"
Sep 16 14:33:41.967146 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:41.967124    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74140765-fb3c-4647-9dcc-4640ccf91709-dshm\") pod \"distinct-w0-924916-kserve-85788cf7b8-7n68d\" (UID: \"74140765-fb3c-4647-9dcc-4640ccf91709\") " pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"
Sep 16 14:33:41.967365 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:41.967346    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74140765-fb3c-4647-9dcc-4640ccf91709-tls-certs\") pod \"distinct-w0-924916-kserve-85788cf7b8-7n68d\" (UID: \"74140765-fb3c-4647-9dcc-4640ccf91709\") " pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"
Sep 16 14:33:41.978982 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:41.978962    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6pbh\" (UniqueName: \"kubernetes.io/projected/74140765-fb3c-4647-9dcc-4640ccf91709-kube-api-access-c6pbh\") pod \"distinct-w0-924916-kserve-85788cf7b8-7n68d\" (UID: \"74140765-fb3c-4647-9dcc-4640ccf91709\") " pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"
Sep 16 14:33:42.112103 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:42.112013    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"
Sep 16 14:33:42.401428 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:42.401405    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"]
Sep 16 14:33:42.402829 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:33:42.402806    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74140765_fb3c_4647_9dcc_4640ccf91709.slice/crio-b8f4784e7bfd52e76ca125502f63e1e74b95b2bf91f1139acc788e410ecf6191 WatchSource:0}: Error finding container b8f4784e7bfd52e76ca125502f63e1e74b95b2bf91f1139acc788e410ecf6191: Status 404 returned error can't find the container with id b8f4784e7bfd52e76ca125502f63e1e74b95b2bf91f1139acc788e410ecf6191
Sep 16 14:33:42.564850 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:42.564811    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d" event={"ID":"74140765-fb3c-4647-9dcc-4640ccf91709","Type":"ContainerStarted","Data":"ee7dc773495ffcc0a63c4a4032be4b2fcdf6a441402766f2a29eac77a565d69e"}
Sep 16 14:33:42.565043 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:42.564860    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d" event={"ID":"74140765-fb3c-4647-9dcc-4640ccf91709","Type":"ContainerStarted","Data":"b8f4784e7bfd52e76ca125502f63e1e74b95b2bf91f1139acc788e410ecf6191"}
Sep 16 14:33:42.583572 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:42.583521    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d" podStartSLOduration=1.5835065670000001 podStartE2EDuration="1.583506567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:33:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:33:42.58166021 +0000 UTC m=+1171.185285239" watchObservedRunningTime="2026-09-16 14:33:42.583506567 +0000 UTC m=+1171.187131599"
Sep 16 14:33:42.615884 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:42.615842    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"
Sep 16 14:33:42.616021 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:42.615894    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"
Sep 16 14:33:42.628899 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:42.628870    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"
Sep 16 14:33:43.581503 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:43.581474    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"
Sep 16 14:33:52.112379 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:52.112334    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"
Sep 16 14:33:52.112858 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:52.112384    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"
Sep 16 14:33:52.127189 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:52.127162    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"
Sep 16 14:33:52.630795 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:52.630766    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"
Sep 16 14:33:56.324287 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:33:56.324249    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"]
Sep 16 14:34:01.959769 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:01.959733    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp"]
Sep 16 14:34:01.964521 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:01.964503    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp"
Sep 16 14:34:01.968333 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:01.968311    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-2539c0-dockercfg-hj7m2\""
Sep 16 14:34:01.968477 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:01.968405    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-2539c0\""
Sep 16 14:34:01.974577 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:01.974555    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp"]
Sep 16 14:34:01.996160 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:01.996130    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx"]
Sep 16 14:34:02.000443 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:02.000421    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx"
Sep 16 14:34:02.005965 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:02.005944    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx"]
Sep 16 14:34:02.060016 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:02.059981    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q6tx\" (UniqueName: \"kubernetes.io/projected/ed2b8a7c-a772-40a9-b610-3828c5d3380c-kube-api-access-5q6tx\") pod \"payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx\" (UID: \"ed2b8a7c-a772-40a9-b610-3828c5d3380c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx"
Sep 16 14:34:02.060194 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:02.060110    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc51bfd4-1df1-481f-92d3-9e761495f65a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp\" (UID: \"fc51bfd4-1df1-481f-92d3-9e761495f65a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp"
Sep 16 14:34:02.060194 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:02.060175    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed2b8a7c-a772-40a9-b610-3828c5d3380c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx\" (UID: \"ed2b8a7c-a772-40a9-b610-3828c5d3380c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx"
Sep 16 14:34:02.060310 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:02.060221    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bngtj\" (UniqueName: \"kubernetes.io/projected/fc51bfd4-1df1-481f-92d3-9e761495f65a-kube-api-access-bngtj\") pod \"payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp\" (UID: \"fc51bfd4-1df1-481f-92d3-9e761495f65a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp"
Sep 16 14:34:02.161148 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:02.161115    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc51bfd4-1df1-481f-92d3-9e761495f65a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp\" (UID: \"fc51bfd4-1df1-481f-92d3-9e761495f65a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp"
Sep 16 14:34:02.161148 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:02.161154    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed2b8a7c-a772-40a9-b610-3828c5d3380c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx\" (UID: \"ed2b8a7c-a772-40a9-b610-3828c5d3380c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx"
Sep 16 14:34:02.161459 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:02.161184    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bngtj\" (UniqueName: \"kubernetes.io/projected/fc51bfd4-1df1-481f-92d3-9e761495f65a-kube-api-access-bngtj\") pod \"payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp\" (UID: \"fc51bfd4-1df1-481f-92d3-9e761495f65a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp"
Sep 16 14:34:02.161459 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:02.161252    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q6tx\" (UniqueName: \"kubernetes.io/projected/ed2b8a7c-a772-40a9-b610-3828c5d3380c-kube-api-access-5q6tx\") pod \"payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx\" (UID: \"ed2b8a7c-a772-40a9-b610-3828c5d3380c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx"
Sep 16 14:34:02.161829 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:02.161806    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc51bfd4-1df1-481f-92d3-9e761495f65a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp\" (UID: \"fc51bfd4-1df1-481f-92d3-9e761495f65a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp"
Sep 16 14:34:02.161959 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:02.161938    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed2b8a7c-a772-40a9-b610-3828c5d3380c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx\" (UID: \"ed2b8a7c-a772-40a9-b610-3828c5d3380c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx"
Sep 16 14:34:02.170007 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:02.169985    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q6tx\" (UniqueName: \"kubernetes.io/projected/ed2b8a7c-a772-40a9-b610-3828c5d3380c-kube-api-access-5q6tx\") pod \"payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx\" (UID: \"ed2b8a7c-a772-40a9-b610-3828c5d3380c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx"
Sep 16 14:34:02.170095 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:02.170077    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bngtj\" (UniqueName: \"kubernetes.io/projected/fc51bfd4-1df1-481f-92d3-9e761495f65a-kube-api-access-bngtj\") pod \"payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp\" (UID: \"fc51bfd4-1df1-481f-92d3-9e761495f65a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp"
Sep 16 14:34:02.276380 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:02.276288    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp"
Sep 16 14:34:02.312706 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:02.312665    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx"
Sep 16 14:34:02.557068 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:02.557047    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp"]
Sep 16 14:34:02.559809 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:34:02.559783    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc51bfd4_1df1_481f_92d3_9e761495f65a.slice/crio-9af7ecbe928a5f3b28ab626ed7c22e5d0b01ca8f32826f330f2a830a8380e27e WatchSource:0}: Error finding container 9af7ecbe928a5f3b28ab626ed7c22e5d0b01ca8f32826f330f2a830a8380e27e: Status 404 returned error can't find the container with id 9af7ecbe928a5f3b28ab626ed7c22e5d0b01ca8f32826f330f2a830a8380e27e
Sep 16 14:34:02.604413 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:02.604371    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx"]
Sep 16 14:34:02.606493 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:34:02.606468    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded2b8a7c_a772_40a9_b610_3828c5d3380c.slice/crio-db05b2e020f2a045efff7037ece49f6c36549af393d23b2e7bed28adfd9a7338 WatchSource:0}: Error finding container db05b2e020f2a045efff7037ece49f6c36549af393d23b2e7bed28adfd9a7338: Status 404 returned error can't find the container with id db05b2e020f2a045efff7037ece49f6c36549af393d23b2e7bed28adfd9a7338
Sep 16 14:34:02.679472 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:02.679436    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp" event={"ID":"fc51bfd4-1df1-481f-92d3-9e761495f65a","Type":"ContainerStarted","Data":"47b18e434d2c7c0d7530d13ddb772125a2e62532862b852d7fb414a665d73ac6"}
Sep 16 14:34:02.679651 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:02.679502    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp" event={"ID":"fc51bfd4-1df1-481f-92d3-9e761495f65a","Type":"ContainerStarted","Data":"9af7ecbe928a5f3b28ab626ed7c22e5d0b01ca8f32826f330f2a830a8380e27e"}
Sep 16 14:34:02.679651 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:02.679536    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp"
Sep 16 14:34:02.680907 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:02.680883    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx" event={"ID":"ed2b8a7c-a772-40a9-b610-3828c5d3380c","Type":"ContainerStarted","Data":"db05b2e020f2a045efff7037ece49f6c36549af393d23b2e7bed28adfd9a7338"}
Sep 16 14:34:02.698958 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:02.698902    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp" podStartSLOduration=1.698882037 podStartE2EDuration="1.698882037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:34:02.698038497 +0000 UTC m=+1191.301663528" watchObservedRunningTime="2026-09-16 14:34:02.698882037 +0000 UTC m=+1191.302507069"
Sep 16 14:34:03.689061 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:03.689020    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx" event={"ID":"ed2b8a7c-a772-40a9-b610-3828c5d3380c","Type":"ContainerStarted","Data":"eed98af2615466373b1675397d3d9b7071afa88199e4bc0b78b946f04972f830"}
Sep 16 14:34:03.689563 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:03.689284    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx"
Sep 16 14:34:03.707282 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:03.707233    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx" podStartSLOduration=2.707217588 podStartE2EDuration="2.707217588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:34:03.705758052 +0000 UTC m=+1192.309383084" watchObservedRunningTime="2026-09-16 14:34:03.707217588 +0000 UTC m=+1192.310842620"
Sep 16 14:34:11.345595 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:11.345548    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l" podUID="de3dae52-deae-4f5f-bc21-6c647914412e" containerName="main" containerID="cri-o://d695b37b0b183618ae20e4383392e25c9c46bf1f62e3fcd5d1a0f69ac4346a55" gracePeriod=45
Sep 16 14:34:11.602262 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:11.602206    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"
Sep 16 14:34:11.733346 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:11.733312    2587 generic.go:356] "Generic (PLEG): container finished" podID="de3dae52-deae-4f5f-bc21-6c647914412e" containerID="d695b37b0b183618ae20e4383392e25c9c46bf1f62e3fcd5d1a0f69ac4346a55" exitCode=0
Sep 16 14:34:11.733521 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:11.733359    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"
Sep 16 14:34:11.733521 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:11.733373    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l" event={"ID":"de3dae52-deae-4f5f-bc21-6c647914412e","Type":"ContainerDied","Data":"d695b37b0b183618ae20e4383392e25c9c46bf1f62e3fcd5d1a0f69ac4346a55"}
Sep 16 14:34:11.733521 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:11.733428    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l" event={"ID":"de3dae52-deae-4f5f-bc21-6c647914412e","Type":"ContainerDied","Data":"c5c6904520995b45f6e9f5b91360f0b10fe4dcc309ff5318a458f315dc182092"}
Sep 16 14:34:11.733521 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:11.733444    2587 scope.go:117] "RemoveContainer" containerID="d695b37b0b183618ae20e4383392e25c9c46bf1f62e3fcd5d1a0f69ac4346a55"
Sep 16 14:34:11.742339 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:11.742319    2587 scope.go:117] "RemoveContainer" containerID="d695b37b0b183618ae20e4383392e25c9c46bf1f62e3fcd5d1a0f69ac4346a55"
Sep 16 14:34:11.742643 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:34:11.742621    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d695b37b0b183618ae20e4383392e25c9c46bf1f62e3fcd5d1a0f69ac4346a55\": container with ID starting with d695b37b0b183618ae20e4383392e25c9c46bf1f62e3fcd5d1a0f69ac4346a55 not found: ID does not exist" containerID="d695b37b0b183618ae20e4383392e25c9c46bf1f62e3fcd5d1a0f69ac4346a55"
Sep 16 14:34:11.742697 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:11.742651    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d695b37b0b183618ae20e4383392e25c9c46bf1f62e3fcd5d1a0f69ac4346a55"} err="failed to get container status \"d695b37b0b183618ae20e4383392e25c9c46bf1f62e3fcd5d1a0f69ac4346a55\": rpc error: code = NotFound desc = could not find container \"d695b37b0b183618ae20e4383392e25c9c46bf1f62e3fcd5d1a0f69ac4346a55\": container with ID starting with d695b37b0b183618ae20e4383392e25c9c46bf1f62e3fcd5d1a0f69ac4346a55 not found: ID does not exist"
Sep 16 14:34:11.756941 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:11.756916    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de3dae52-deae-4f5f-bc21-6c647914412e-dshm\") pod \"de3dae52-deae-4f5f-bc21-6c647914412e\" (UID: \"de3dae52-deae-4f5f-bc21-6c647914412e\") "
Sep 16 14:34:11.757066 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:11.756966    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de3dae52-deae-4f5f-bc21-6c647914412e-tls-certs\") pod \"de3dae52-deae-4f5f-bc21-6c647914412e\" (UID: \"de3dae52-deae-4f5f-bc21-6c647914412e\") "
Sep 16 14:34:11.757066 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:11.756995    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztvtg\" (UniqueName: \"kubernetes.io/projected/de3dae52-deae-4f5f-bc21-6c647914412e-kube-api-access-ztvtg\") pod \"de3dae52-deae-4f5f-bc21-6c647914412e\" (UID: \"de3dae52-deae-4f5f-bc21-6c647914412e\") "
Sep 16 14:34:11.757066 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:11.757027    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de3dae52-deae-4f5f-bc21-6c647914412e-model-cache\") pod \"de3dae52-deae-4f5f-bc21-6c647914412e\" (UID: \"de3dae52-deae-4f5f-bc21-6c647914412e\") "
Sep 16 14:34:11.757066 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:11.757062    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de3dae52-deae-4f5f-bc21-6c647914412e-home\") pod \"de3dae52-deae-4f5f-bc21-6c647914412e\" (UID: \"de3dae52-deae-4f5f-bc21-6c647914412e\") "
Sep 16 14:34:11.757288 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:11.757095    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de3dae52-deae-4f5f-bc21-6c647914412e-tmp-dir\") pod \"de3dae52-deae-4f5f-bc21-6c647914412e\" (UID: \"de3dae52-deae-4f5f-bc21-6c647914412e\") "
Sep 16 14:34:11.757355 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:11.757314    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de3dae52-deae-4f5f-bc21-6c647914412e-home" (OuterVolumeSpecName: "home") pod "de3dae52-deae-4f5f-bc21-6c647914412e" (UID: "de3dae52-deae-4f5f-bc21-6c647914412e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:34:11.757355 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:11.757329    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de3dae52-deae-4f5f-bc21-6c647914412e-model-cache" (OuterVolumeSpecName: "model-cache") pod "de3dae52-deae-4f5f-bc21-6c647914412e" (UID: "de3dae52-deae-4f5f-bc21-6c647914412e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:34:11.757504 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:11.757486    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de3dae52-deae-4f5f-bc21-6c647914412e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "de3dae52-deae-4f5f-bc21-6c647914412e" (UID: "de3dae52-deae-4f5f-bc21-6c647914412e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:34:11.759458 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:11.759427    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de3dae52-deae-4f5f-bc21-6c647914412e-dshm" (OuterVolumeSpecName: "dshm") pod "de3dae52-deae-4f5f-bc21-6c647914412e" (UID: "de3dae52-deae-4f5f-bc21-6c647914412e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:34:11.759458 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:11.759452    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de3dae52-deae-4f5f-bc21-6c647914412e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "de3dae52-deae-4f5f-bc21-6c647914412e" (UID: "de3dae52-deae-4f5f-bc21-6c647914412e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:34:11.759628 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:11.759479    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de3dae52-deae-4f5f-bc21-6c647914412e-kube-api-access-ztvtg" (OuterVolumeSpecName: "kube-api-access-ztvtg") pod "de3dae52-deae-4f5f-bc21-6c647914412e" (UID: "de3dae52-deae-4f5f-bc21-6c647914412e"). InnerVolumeSpecName "kube-api-access-ztvtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:34:11.858344 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:11.858245    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de3dae52-deae-4f5f-bc21-6c647914412e-dshm\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:34:11.858344 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:11.858284    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de3dae52-deae-4f5f-bc21-6c647914412e-tls-certs\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:34:11.858344 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:11.858298    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztvtg\" (UniqueName: \"kubernetes.io/projected/de3dae52-deae-4f5f-bc21-6c647914412e-kube-api-access-ztvtg\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:34:11.858344 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:11.858312    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de3dae52-deae-4f5f-bc21-6c647914412e-model-cache\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:34:11.858344 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:11.858323    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de3dae52-deae-4f5f-bc21-6c647914412e-home\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:34:11.858344 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:11.858333    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de3dae52-deae-4f5f-bc21-6c647914412e-tmp-dir\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:34:12.050748 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:12.050709    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"]
Sep 16 14:34:12.055756 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:12.055726    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-ec80c7/noten-model-ec80c7-kserve-5d77f7d95-qvs5l"]
Sep 16 14:34:13.691508 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:13.691476    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp"
Sep 16 14:34:13.903836 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:13.903802    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de3dae52-deae-4f5f-bc21-6c647914412e" path="/var/lib/kubelet/pods/de3dae52-deae-4f5f-bc21-6c647914412e/volumes"
Sep 16 14:34:14.697580 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:14.697551    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx"
Sep 16 14:34:20.791991 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:20.791960    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-094dab-6hcdn"]
Sep 16 14:34:20.792407 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:20.792379    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de3dae52-deae-4f5f-bc21-6c647914412e" containerName="main"
Sep 16 14:34:20.792407 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:20.792407    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="de3dae52-deae-4f5f-bc21-6c647914412e" containerName="main"
Sep 16 14:34:20.792489 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:20.792482    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="de3dae52-deae-4f5f-bc21-6c647914412e" containerName="main"
Sep 16 14:34:20.796018 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:20.795998    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-094dab-6hcdn"
Sep 16 14:34:20.799787 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:20.799765    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:34:20.799866 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:20.799778    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4jf\""
Sep 16 14:34:20.808576 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:20.808555    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-094dab-6hcdn"]
Sep 16 14:34:20.942625 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:20.942587    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169c80f6-e153-4847-ad69-385990f6017f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-094dab-6hcdn\" (UID: \"169c80f6-e153-4847-ad69-385990f6017f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-094dab-6hcdn"
Sep 16 14:34:21.044158 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:21.044079    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169c80f6-e153-4847-ad69-385990f6017f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-094dab-6hcdn\" (UID: \"169c80f6-e153-4847-ad69-385990f6017f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-094dab-6hcdn"
Sep 16 14:34:21.044754 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:21.044736    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169c80f6-e153-4847-ad69-385990f6017f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-094dab-6hcdn\" (UID: \"169c80f6-e153-4847-ad69-385990f6017f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-094dab-6hcdn"
Sep 16 14:34:21.107032 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:21.107003    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-094dab-6hcdn"
Sep 16 14:34:21.399742 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:21.399698    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-094dab-6hcdn"]
Sep 16 14:34:21.404572 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:34:21.404544    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169c80f6_e153_4847_ad69_385990f6017f.slice/crio-e528178e7ff09d3b719e5dc959ea7e89218fb2146f4634acfdc55548dc7302cb WatchSource:0}: Error finding container e528178e7ff09d3b719e5dc959ea7e89218fb2146f4634acfdc55548dc7302cb: Status 404 returned error can't find the container with id e528178e7ff09d3b719e5dc959ea7e89218fb2146f4634acfdc55548dc7302cb
Sep 16 14:34:21.789842 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:21.789808    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-094dab-6hcdn" event={"ID":"169c80f6-e153-4847-ad69-385990f6017f","Type":"ContainerStarted","Data":"6d62814b12a27422120edc29b1d7c32d69b5434f23b5e98b1321bed3c2583d23"}
Sep 16 14:34:21.790029 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:21.789868    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-094dab-6hcdn" event={"ID":"169c80f6-e153-4847-ad69-385990f6017f","Type":"ContainerStarted","Data":"e528178e7ff09d3b719e5dc959ea7e89218fb2146f4634acfdc55548dc7302cb"}
Sep 16 14:34:21.813094 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:21.813043    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-094dab-6hcdn" podStartSLOduration=1.8130269490000002 podStartE2EDuration="1.813026949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:34:21.810491148 +0000 UTC m=+1210.414116173" watchObservedRunningTime="2026-09-16 14:34:21.813026949 +0000 UTC m=+1210.416651980"
Sep 16 14:34:22.798201 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:22.798165    2587 generic.go:356] "Generic (PLEG): container finished" podID="169c80f6-e153-4847-ad69-385990f6017f" containerID="6d62814b12a27422120edc29b1d7c32d69b5434f23b5e98b1321bed3c2583d23" exitCode=0
Sep 16 14:34:22.798375 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:22.798228    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-094dab-6hcdn" event={"ID":"169c80f6-e153-4847-ad69-385990f6017f","Type":"ContainerDied","Data":"6d62814b12a27422120edc29b1d7c32d69b5434f23b5e98b1321bed3c2583d23"}
Sep 16 14:34:23.956727 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:23.956703    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-094dab-6hcdn"
Sep 16 14:34:24.073984 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:24.073883    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169c80f6-e153-4847-ad69-385990f6017f-maas-api-service-ca\") pod \"169c80f6-e153-4847-ad69-385990f6017f\" (UID: \"169c80f6-e153-4847-ad69-385990f6017f\") "
Sep 16 14:34:24.074324 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:24.074300    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/169c80f6-e153-4847-ad69-385990f6017f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "169c80f6-e153-4847-ad69-385990f6017f" (UID: "169c80f6-e153-4847-ad69-385990f6017f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:34:24.175709 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:24.175674    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169c80f6-e153-4847-ad69-385990f6017f-maas-api-service-ca\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:34:24.812812 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:24.812783    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-094dab-6hcdn"
Sep 16 14:34:24.812983 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:24.812821    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-094dab-6hcdn" event={"ID":"169c80f6-e153-4847-ad69-385990f6017f","Type":"ContainerDied","Data":"e528178e7ff09d3b719e5dc959ea7e89218fb2146f4634acfdc55548dc7302cb"}
Sep 16 14:34:24.812983 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:24.812868    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e528178e7ff09d3b719e5dc959ea7e89218fb2146f4634acfdc55548dc7302cb"
Sep 16 14:34:33.002039 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.002007    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf"]
Sep 16 14:34:33.002621 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.002258    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf" podUID="a7b3132a-66c9-4a5d-b47b-e3cff8a2b644" containerName="payload-processing" containerID="cri-o://4c143226815d150a10f4b78ea17114099ea906a51a325ca7113a69d7f0722f16" gracePeriod=30
Sep 16 14:34:33.015678 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.015648    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55"]
Sep 16 14:34:33.015998 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.015889    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55" podUID="608bb3a3-74d2-4557-8f6a-5961db6fad36" containerName="payload-pre-processing" containerID="cri-o://89c05196e08c2ae58aaef3d4378c2b9e696bbec201bdb229994c3c09dc653176" gracePeriod=30
Sep 16 14:34:33.077879 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.077854    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7605-fcc2aa"]
Sep 16 14:34:33.078285 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.078264    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="169c80f6-e153-4847-ad69-385990f6017f" containerName="revoke-keys"
Sep 16 14:34:33.078331 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.078288    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="169c80f6-e153-4847-ad69-385990f6017f" containerName="revoke-keys"
Sep 16 14:34:33.078370 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.078359    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="169c80f6-e153-4847-ad69-385990f6017f" containerName="revoke-keys"
Sep 16 14:34:33.081492 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.081474    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7605-fcc2aa"
Sep 16 14:34:33.084494 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.084476    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gn749\""
Sep 16 14:34:33.099019 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.098990    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7605-fcc2aa"]
Sep 16 14:34:33.151304 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.151260    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq95p\" (UniqueName: \"kubernetes.io/projected/2853dd1c-58db-47b8-998d-b9147fac6f72-kube-api-access-xq95p\") pod \"test-curl-7605-fcc2aa\" (UID: \"2853dd1c-58db-47b8-998d-b9147fac6f72\") " pod="opendatahub/test-curl-7605-fcc2aa"
Sep 16 14:34:33.253189 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.252936    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq95p\" (UniqueName: \"kubernetes.io/projected/2853dd1c-58db-47b8-998d-b9147fac6f72-kube-api-access-xq95p\") pod \"test-curl-7605-fcc2aa\" (UID: \"2853dd1c-58db-47b8-998d-b9147fac6f72\") " pod="opendatahub/test-curl-7605-fcc2aa"
Sep 16 14:34:33.261968 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.261936    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq95p\" (UniqueName: \"kubernetes.io/projected/2853dd1c-58db-47b8-998d-b9147fac6f72-kube-api-access-xq95p\") pod \"test-curl-7605-fcc2aa\" (UID: \"2853dd1c-58db-47b8-998d-b9147fac6f72\") " pod="opendatahub/test-curl-7605-fcc2aa"
Sep 16 14:34:33.279143 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.279117    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf"
Sep 16 14:34:33.289867 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.289842    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-094dab-6hcdn"]
Sep 16 14:34:33.293641 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.293611    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-094dab-6hcdn"]
Sep 16 14:34:33.294007 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.293987    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55"
Sep 16 14:34:33.353451 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.353427    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/608bb3a3-74d2-4557-8f6a-5961db6fad36-plugins-config-volume\") pod \"608bb3a3-74d2-4557-8f6a-5961db6fad36\" (UID: \"608bb3a3-74d2-4557-8f6a-5961db6fad36\") "
Sep 16 14:34:33.353604 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.353494    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdcr7\" (UniqueName: \"kubernetes.io/projected/608bb3a3-74d2-4557-8f6a-5961db6fad36-kube-api-access-rdcr7\") pod \"608bb3a3-74d2-4557-8f6a-5961db6fad36\" (UID: \"608bb3a3-74d2-4557-8f6a-5961db6fad36\") "
Sep 16 14:34:33.353604 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.353585    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5vcm\" (UniqueName: \"kubernetes.io/projected/a7b3132a-66c9-4a5d-b47b-e3cff8a2b644-kube-api-access-q5vcm\") pod \"a7b3132a-66c9-4a5d-b47b-e3cff8a2b644\" (UID: \"a7b3132a-66c9-4a5d-b47b-e3cff8a2b644\") "
Sep 16 14:34:33.353677 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.353655    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b3132a-66c9-4a5d-b47b-e3cff8a2b644-plugins-config-volume\") pod \"a7b3132a-66c9-4a5d-b47b-e3cff8a2b644\" (UID: \"a7b3132a-66c9-4a5d-b47b-e3cff8a2b644\") "
Sep 16 14:34:33.353820 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.353795    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/608bb3a3-74d2-4557-8f6a-5961db6fad36-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "608bb3a3-74d2-4557-8f6a-5961db6fad36" (UID: "608bb3a3-74d2-4557-8f6a-5961db6fad36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:34:33.354036 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.354014    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7b3132a-66c9-4a5d-b47b-e3cff8a2b644-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a7b3132a-66c9-4a5d-b47b-e3cff8a2b644" (UID: "a7b3132a-66c9-4a5d-b47b-e3cff8a2b644"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:34:33.354196 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.354143    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b3132a-66c9-4a5d-b47b-e3cff8a2b644-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:34:33.354196 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.354165    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/608bb3a3-74d2-4557-8f6a-5961db6fad36-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:34:33.355727 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.355703    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/608bb3a3-74d2-4557-8f6a-5961db6fad36-kube-api-access-rdcr7" (OuterVolumeSpecName: "kube-api-access-rdcr7") pod "608bb3a3-74d2-4557-8f6a-5961db6fad36" (UID: "608bb3a3-74d2-4557-8f6a-5961db6fad36"). InnerVolumeSpecName "kube-api-access-rdcr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:34:33.355824 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.355741    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7b3132a-66c9-4a5d-b47b-e3cff8a2b644-kube-api-access-q5vcm" (OuterVolumeSpecName: "kube-api-access-q5vcm") pod "a7b3132a-66c9-4a5d-b47b-e3cff8a2b644" (UID: "a7b3132a-66c9-4a5d-b47b-e3cff8a2b644"). InnerVolumeSpecName "kube-api-access-q5vcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:34:33.435960 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.435925    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7605-fcc2aa"
Sep 16 14:34:33.455443 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.455417    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdcr7\" (UniqueName: \"kubernetes.io/projected/608bb3a3-74d2-4557-8f6a-5961db6fad36-kube-api-access-rdcr7\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:34:33.455443 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.455442    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5vcm\" (UniqueName: \"kubernetes.io/projected/a7b3132a-66c9-4a5d-b47b-e3cff8a2b644-kube-api-access-q5vcm\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:34:33.712585 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.712558    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7605-fcc2aa"]
Sep 16 14:34:33.715368 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:34:33.715339    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2853dd1c_58db_47b8_998d_b9147fac6f72.slice/crio-f5581dac9f2a2d0caa0072ab083b2312c1750906e8ce787372c83cac9b2b5bbe WatchSource:0}: Error finding container f5581dac9f2a2d0caa0072ab083b2312c1750906e8ce787372c83cac9b2b5bbe: Status 404 returned error can't find the container with id f5581dac9f2a2d0caa0072ab083b2312c1750906e8ce787372c83cac9b2b5bbe
Sep 16 14:34:33.865173 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.865138    2587 generic.go:356] "Generic (PLEG): container finished" podID="a7b3132a-66c9-4a5d-b47b-e3cff8a2b644" containerID="4c143226815d150a10f4b78ea17114099ea906a51a325ca7113a69d7f0722f16" exitCode=0
Sep 16 14:34:33.865317 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.865182    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf"
Sep 16 14:34:33.865317 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.865208    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf" event={"ID":"a7b3132a-66c9-4a5d-b47b-e3cff8a2b644","Type":"ContainerDied","Data":"4c143226815d150a10f4b78ea17114099ea906a51a325ca7113a69d7f0722f16"}
Sep 16 14:34:33.865317 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.865254    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf" event={"ID":"a7b3132a-66c9-4a5d-b47b-e3cff8a2b644","Type":"ContainerDied","Data":"aad5ad75d9c338c38cc5f7379cf8513094e8bea2ed89227e2402cb51d90f8eec"}
Sep 16 14:34:33.865317 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.865276    2587 scope.go:117] "RemoveContainer" containerID="4c143226815d150a10f4b78ea17114099ea906a51a325ca7113a69d7f0722f16"
Sep 16 14:34:33.866918 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.866889    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7605-fcc2aa" event={"ID":"2853dd1c-58db-47b8-998d-b9147fac6f72","Type":"ContainerStarted","Data":"d2f2a1200b6a381ea8afcaef92b21be202fb171e15f885d734414117686c7890"}
Sep 16 14:34:33.867052 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.866932    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7605-fcc2aa" event={"ID":"2853dd1c-58db-47b8-998d-b9147fac6f72","Type":"ContainerStarted","Data":"f5581dac9f2a2d0caa0072ab083b2312c1750906e8ce787372c83cac9b2b5bbe"}
Sep 16 14:34:33.868255 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.868235    2587 generic.go:356] "Generic (PLEG): container finished" podID="608bb3a3-74d2-4557-8f6a-5961db6fad36" containerID="89c05196e08c2ae58aaef3d4378c2b9e696bbec201bdb229994c3c09dc653176" exitCode=0
Sep 16 14:34:33.868368 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.868276    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55"
Sep 16 14:34:33.868368 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.868318    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55" event={"ID":"608bb3a3-74d2-4557-8f6a-5961db6fad36","Type":"ContainerDied","Data":"89c05196e08c2ae58aaef3d4378c2b9e696bbec201bdb229994c3c09dc653176"}
Sep 16 14:34:33.868509 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.868369    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55" event={"ID":"608bb3a3-74d2-4557-8f6a-5961db6fad36","Type":"ContainerDied","Data":"b3510bc5270da62ec6332e34579e551cdaf16149a90476045ca7816f08ebbaa5"}
Sep 16 14:34:33.874955 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.874940    2587 scope.go:117] "RemoveContainer" containerID="4c143226815d150a10f4b78ea17114099ea906a51a325ca7113a69d7f0722f16"
Sep 16 14:34:33.875254 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:34:33.875236    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c143226815d150a10f4b78ea17114099ea906a51a325ca7113a69d7f0722f16\": container with ID starting with 4c143226815d150a10f4b78ea17114099ea906a51a325ca7113a69d7f0722f16 not found: ID does not exist" containerID="4c143226815d150a10f4b78ea17114099ea906a51a325ca7113a69d7f0722f16"
Sep 16 14:34:33.875344 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.875261    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c143226815d150a10f4b78ea17114099ea906a51a325ca7113a69d7f0722f16"} err="failed to get container status \"4c143226815d150a10f4b78ea17114099ea906a51a325ca7113a69d7f0722f16\": rpc error: code = NotFound desc = could not find container \"4c143226815d150a10f4b78ea17114099ea906a51a325ca7113a69d7f0722f16\": container with ID starting with 4c143226815d150a10f4b78ea17114099ea906a51a325ca7113a69d7f0722f16 not found: ID does not exist"
Sep 16 14:34:33.875344 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.875275    2587 scope.go:117] "RemoveContainer" containerID="89c05196e08c2ae58aaef3d4378c2b9e696bbec201bdb229994c3c09dc653176"
Sep 16 14:34:33.883511 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.883496    2587 scope.go:117] "RemoveContainer" containerID="89c05196e08c2ae58aaef3d4378c2b9e696bbec201bdb229994c3c09dc653176"
Sep 16 14:34:33.883755 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:34:33.883736    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89c05196e08c2ae58aaef3d4378c2b9e696bbec201bdb229994c3c09dc653176\": container with ID starting with 89c05196e08c2ae58aaef3d4378c2b9e696bbec201bdb229994c3c09dc653176 not found: ID does not exist" containerID="89c05196e08c2ae58aaef3d4378c2b9e696bbec201bdb229994c3c09dc653176"
Sep 16 14:34:33.883802 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.883762    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89c05196e08c2ae58aaef3d4378c2b9e696bbec201bdb229994c3c09dc653176"} err="failed to get container status \"89c05196e08c2ae58aaef3d4378c2b9e696bbec201bdb229994c3c09dc653176\": rpc error: code = NotFound desc = could not find container \"89c05196e08c2ae58aaef3d4378c2b9e696bbec201bdb229994c3c09dc653176\": container with ID starting with 89c05196e08c2ae58aaef3d4378c2b9e696bbec201bdb229994c3c09dc653176 not found: ID does not exist"
Sep 16 14:34:33.902469 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.902424    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7605-fcc2aa" podStartSLOduration=0.902406093 podStartE2EDuration="902.406093ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:34:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:34:33.899456842 +0000 UTC m=+1222.503081873" watchObservedRunningTime="2026-09-16 14:34:33.902406093 +0000 UTC m=+1222.506031118"
Sep 16 14:34:33.903778 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.903759    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="169c80f6-e153-4847-ad69-385990f6017f" path="/var/lib/kubelet/pods/169c80f6-e153-4847-ad69-385990f6017f/volumes"
Sep 16 14:34:33.915364 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.915332    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55"]
Sep 16 14:34:33.918481 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.918459    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-094dab-6c6d699fd-tsv55"]
Sep 16 14:34:33.948146 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.948108    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf"]
Sep 16 14:34:33.959218 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:33.953284    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-094dab-95bc57677-7r6nf"]
Sep 16 14:34:34.592474 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:34.592442    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7605-fcc2aa"]
Sep 16 14:34:34.597261 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:34.597234    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7605-fcc2aa"]
Sep 16 14:34:34.722517 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:34.722481    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7605-f13f6f"]
Sep 16 14:34:34.723155 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:34.723135    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="608bb3a3-74d2-4557-8f6a-5961db6fad36" containerName="payload-pre-processing"
Sep 16 14:34:34.723199 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:34.723161    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="608bb3a3-74d2-4557-8f6a-5961db6fad36" containerName="payload-pre-processing"
Sep 16 14:34:34.723234 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:34.723200    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7b3132a-66c9-4a5d-b47b-e3cff8a2b644" containerName="payload-processing"
Sep 16 14:34:34.723234 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:34.723209    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7b3132a-66c9-4a5d-b47b-e3cff8a2b644" containerName="payload-processing"
Sep 16 14:34:34.723303 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:34.723236    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2853dd1c-58db-47b8-998d-b9147fac6f72" containerName="test-curl-7605-fcc2aa"
Sep 16 14:34:34.723303 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:34.723244    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="2853dd1c-58db-47b8-998d-b9147fac6f72" containerName="test-curl-7605-fcc2aa"
Sep 16 14:34:34.723370 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:34.723335    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="608bb3a3-74d2-4557-8f6a-5961db6fad36" containerName="payload-pre-processing"
Sep 16 14:34:34.723370 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:34.723349    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7b3132a-66c9-4a5d-b47b-e3cff8a2b644" containerName="payload-processing"
Sep 16 14:34:34.723370 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:34.723361    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="2853dd1c-58db-47b8-998d-b9147fac6f72" containerName="test-curl-7605-fcc2aa"
Sep 16 14:34:34.728643 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:34.728621    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7605-f13f6f"
Sep 16 14:34:34.736945 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:34.736921    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7605-f13f6f"]
Sep 16 14:34:34.766825 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:34.766797    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b9qj\" (UniqueName: \"kubernetes.io/projected/5764823f-b456-4a9a-95aa-eb8c2cd891af-kube-api-access-2b9qj\") pod \"test-curl-7605-f13f6f\" (UID: \"5764823f-b456-4a9a-95aa-eb8c2cd891af\") " pod="opendatahub/test-curl-7605-f13f6f"
Sep 16 14:34:34.867940 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:34.867907    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b9qj\" (UniqueName: \"kubernetes.io/projected/5764823f-b456-4a9a-95aa-eb8c2cd891af-kube-api-access-2b9qj\") pod \"test-curl-7605-f13f6f\" (UID: \"5764823f-b456-4a9a-95aa-eb8c2cd891af\") " pod="opendatahub/test-curl-7605-f13f6f"
Sep 16 14:34:34.876126 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:34:34.876102    2587 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7605-fcc2aa" container="test-curl-7605-fcc2aa"
Sep 16 14:34:34.879048 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:34.879028    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b9qj\" (UniqueName: \"kubernetes.io/projected/5764823f-b456-4a9a-95aa-eb8c2cd891af-kube-api-access-2b9qj\") pod \"test-curl-7605-f13f6f\" (UID: \"5764823f-b456-4a9a-95aa-eb8c2cd891af\") " pod="opendatahub/test-curl-7605-f13f6f"
Sep 16 14:34:35.040139 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:35.040067    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7605-f13f6f"
Sep 16 14:34:35.318955 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:35.318926    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7605-f13f6f"]
Sep 16 14:34:35.320604 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:34:35.320579    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5764823f_b456_4a9a_95aa_eb8c2cd891af.slice/crio-8e8d44283e0d0517ea3fa92c07f6522b2e8f78c4301ae1b1481b563d6d5f23f0 WatchSource:0}: Error finding container 8e8d44283e0d0517ea3fa92c07f6522b2e8f78c4301ae1b1481b563d6d5f23f0: Status 404 returned error can't find the container with id 8e8d44283e0d0517ea3fa92c07f6522b2e8f78c4301ae1b1481b563d6d5f23f0
Sep 16 14:34:35.882892 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:35.882856    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7605-f13f6f" event={"ID":"5764823f-b456-4a9a-95aa-eb8c2cd891af","Type":"ContainerStarted","Data":"59d2cc867854e12594f8b6e4b395ab30c3e776830d14daaee30fa7c292f18096"}
Sep 16 14:34:35.882892 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:35.882894    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7605-f13f6f" event={"ID":"5764823f-b456-4a9a-95aa-eb8c2cd891af","Type":"ContainerStarted","Data":"8e8d44283e0d0517ea3fa92c07f6522b2e8f78c4301ae1b1481b563d6d5f23f0"}
Sep 16 14:34:35.883435 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:35.883086    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7605-fcc2aa" podUID="2853dd1c-58db-47b8-998d-b9147fac6f72" containerName="test-curl-7605-fcc2aa" containerID="cri-o://d2f2a1200b6a381ea8afcaef92b21be202fb171e15f885d734414117686c7890" gracePeriod=2
Sep 16 14:34:35.903104 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:35.903076    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="608bb3a3-74d2-4557-8f6a-5961db6fad36" path="/var/lib/kubelet/pods/608bb3a3-74d2-4557-8f6a-5961db6fad36/volumes"
Sep 16 14:34:35.903459 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:35.903445    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7b3132a-66c9-4a5d-b47b-e3cff8a2b644" path="/var/lib/kubelet/pods/a7b3132a-66c9-4a5d-b47b-e3cff8a2b644/volumes"
Sep 16 14:34:35.922989 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:35.922944    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7605-f13f6f" podStartSLOduration=1.9229302860000002 podStartE2EDuration="1.922930286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:34:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:34:35.921050209 +0000 UTC m=+1224.524675242" watchObservedRunningTime="2026-09-16 14:34:35.922930286 +0000 UTC m=+1224.526555318"
Sep 16 14:34:36.680968 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:36.680935    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7605-f13f6f"]
Sep 16 14:34:36.689451 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:36.689371    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7605-f13f6f"]
Sep 16 14:34:36.890193 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:34:36.890160    2587 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7605-f13f6f" container="test-curl-7605-f13f6f"
Sep 16 14:34:37.894991 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:37.894953    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7605-f13f6f" podUID="5764823f-b456-4a9a-95aa-eb8c2cd891af" containerName="test-curl-7605-f13f6f" containerID="cri-o://59d2cc867854e12594f8b6e4b395ab30c3e776830d14daaee30fa7c292f18096" gracePeriod=2
Sep 16 14:34:37.897834 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:34:37.897808    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7605-f13f6f\" is forbidden: User \"system:node:ip-10-0-132-217.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-217.ec2.internal' and this object" podUID="5764823f-b456-4a9a-95aa-eb8c2cd891af" pod="opendatahub/test-curl-7605-f13f6f"
Sep 16 14:34:38.061444 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:38.061415    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7605-fcc2aa"
Sep 16 14:34:38.097340 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:38.097313    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xq95p\" (UniqueName: \"kubernetes.io/projected/2853dd1c-58db-47b8-998d-b9147fac6f72-kube-api-access-xq95p\") pod \"2853dd1c-58db-47b8-998d-b9147fac6f72\" (UID: \"2853dd1c-58db-47b8-998d-b9147fac6f72\") "
Sep 16 14:34:38.099512 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:38.099486    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2853dd1c-58db-47b8-998d-b9147fac6f72-kube-api-access-xq95p" (OuterVolumeSpecName: "kube-api-access-xq95p") pod "2853dd1c-58db-47b8-998d-b9147fac6f72" (UID: "2853dd1c-58db-47b8-998d-b9147fac6f72"). InnerVolumeSpecName "kube-api-access-xq95p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:34:38.199002 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:38.198971    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq95p\" (UniqueName: \"kubernetes.io/projected/2853dd1c-58db-47b8-998d-b9147fac6f72-kube-api-access-xq95p\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:34:38.902284 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:38.902240    2587 generic.go:356] "Generic (PLEG): container finished" podID="2853dd1c-58db-47b8-998d-b9147fac6f72" containerID="d2f2a1200b6a381ea8afcaef92b21be202fb171e15f885d734414117686c7890" exitCode=137
Sep 16 14:34:38.902728 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:38.902308    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7605-fcc2aa"
Sep 16 14:34:38.902728 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:38.902336    2587 scope.go:117] "RemoveContainer" containerID="d2f2a1200b6a381ea8afcaef92b21be202fb171e15f885d734414117686c7890"
Sep 16 14:34:38.912025 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:38.912009    2587 scope.go:117] "RemoveContainer" containerID="d2f2a1200b6a381ea8afcaef92b21be202fb171e15f885d734414117686c7890"
Sep 16 14:34:38.912564 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:34:38.912326    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2f2a1200b6a381ea8afcaef92b21be202fb171e15f885d734414117686c7890\": container with ID starting with d2f2a1200b6a381ea8afcaef92b21be202fb171e15f885d734414117686c7890 not found: ID does not exist" containerID="d2f2a1200b6a381ea8afcaef92b21be202fb171e15f885d734414117686c7890"
Sep 16 14:34:38.912564 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:38.912361    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2f2a1200b6a381ea8afcaef92b21be202fb171e15f885d734414117686c7890"} err="failed to get container status \"d2f2a1200b6a381ea8afcaef92b21be202fb171e15f885d734414117686c7890\": rpc error: code = NotFound desc = could not find container \"d2f2a1200b6a381ea8afcaef92b21be202fb171e15f885d734414117686c7890\": container with ID starting with d2f2a1200b6a381ea8afcaef92b21be202fb171e15f885d734414117686c7890 not found: ID does not exist"
Sep 16 14:34:38.915275 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:34:38.915252    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7605-fcc2aa\" is forbidden: User \"system:node:ip-10-0-132-217.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-217.ec2.internal' and this object" podUID="2853dd1c-58db-47b8-998d-b9147fac6f72" pod="opendatahub/test-curl-7605-fcc2aa"
Sep 16 14:34:39.903419 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:39.903376    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2853dd1c-58db-47b8-998d-b9147fac6f72" path="/var/lib/kubelet/pods/2853dd1c-58db-47b8-998d-b9147fac6f72/volumes"
Sep 16 14:34:40.081682 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:40.081656    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7605-f13f6f"
Sep 16 14:34:40.118789 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:40.118748    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2b9qj\" (UniqueName: \"kubernetes.io/projected/5764823f-b456-4a9a-95aa-eb8c2cd891af-kube-api-access-2b9qj\") pod \"5764823f-b456-4a9a-95aa-eb8c2cd891af\" (UID: \"5764823f-b456-4a9a-95aa-eb8c2cd891af\") "
Sep 16 14:34:40.121552 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:40.121526    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5764823f-b456-4a9a-95aa-eb8c2cd891af-kube-api-access-2b9qj" (OuterVolumeSpecName: "kube-api-access-2b9qj") pod "5764823f-b456-4a9a-95aa-eb8c2cd891af" (UID: "5764823f-b456-4a9a-95aa-eb8c2cd891af"). InnerVolumeSpecName "kube-api-access-2b9qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:34:40.220119 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:40.220038    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b9qj\" (UniqueName: \"kubernetes.io/projected/5764823f-b456-4a9a-95aa-eb8c2cd891af-kube-api-access-2b9qj\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:34:40.915986 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:40.915951    2587 generic.go:356] "Generic (PLEG): container finished" podID="5764823f-b456-4a9a-95aa-eb8c2cd891af" containerID="59d2cc867854e12594f8b6e4b395ab30c3e776830d14daaee30fa7c292f18096" exitCode=137
Sep 16 14:34:40.916459 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:40.915997    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7605-f13f6f"
Sep 16 14:34:40.916459 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:40.916064    2587 scope.go:117] "RemoveContainer" containerID="59d2cc867854e12594f8b6e4b395ab30c3e776830d14daaee30fa7c292f18096"
Sep 16 14:34:40.925312 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:40.925295    2587 scope.go:117] "RemoveContainer" containerID="59d2cc867854e12594f8b6e4b395ab30c3e776830d14daaee30fa7c292f18096"
Sep 16 14:34:40.925686 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:34:40.925666    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59d2cc867854e12594f8b6e4b395ab30c3e776830d14daaee30fa7c292f18096\": container with ID starting with 59d2cc867854e12594f8b6e4b395ab30c3e776830d14daaee30fa7c292f18096 not found: ID does not exist" containerID="59d2cc867854e12594f8b6e4b395ab30c3e776830d14daaee30fa7c292f18096"
Sep 16 14:34:40.925762 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:40.925695    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59d2cc867854e12594f8b6e4b395ab30c3e776830d14daaee30fa7c292f18096"} err="failed to get container status \"59d2cc867854e12594f8b6e4b395ab30c3e776830d14daaee30fa7c292f18096\": rpc error: code = NotFound desc = could not find container \"59d2cc867854e12594f8b6e4b395ab30c3e776830d14daaee30fa7c292f18096\": container with ID starting with 59d2cc867854e12594f8b6e4b395ab30c3e776830d14daaee30fa7c292f18096 not found: ID does not exist"
Sep 16 14:34:40.926544 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:34:40.926518    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7605-f13f6f\" is forbidden: User \"system:node:ip-10-0-132-217.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-217.ec2.internal' and this object" podUID="5764823f-b456-4a9a-95aa-eb8c2cd891af" pod="opendatahub/test-curl-7605-f13f6f"
Sep 16 14:34:41.679478 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:41.679445    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0b89a-t574k"]
Sep 16 14:34:41.679886 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:41.679873    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5764823f-b456-4a9a-95aa-eb8c2cd891af" containerName="test-curl-7605-f13f6f"
Sep 16 14:34:41.679928 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:41.679888    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="5764823f-b456-4a9a-95aa-eb8c2cd891af" containerName="test-curl-7605-f13f6f"
Sep 16 14:34:41.679993 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:41.679980    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="5764823f-b456-4a9a-95aa-eb8c2cd891af" containerName="test-curl-7605-f13f6f"
Sep 16 14:34:41.684473 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:41.684456    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0b89a-t574k"
Sep 16 14:34:41.687531 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:41.687509    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:34:41.687641 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:41.687538    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4jf\""
Sep 16 14:34:41.696604 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:41.696584    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0b89a-t574k"]
Sep 16 14:34:41.708258 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:34:41.708224    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7605-f13f6f\" is forbidden: User \"system:node:ip-10-0-132-217.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-217.ec2.internal' and this object" podUID="5764823f-b456-4a9a-95aa-eb8c2cd891af" pod="opendatahub/test-curl-7605-f13f6f"
Sep 16 14:34:41.735083 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:41.735056    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de41b0a-d8eb-483b-a5df-60fd784535ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d0b89a-t574k\" (UID: \"1de41b0a-d8eb-483b-a5df-60fd784535ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0b89a-t574k"
Sep 16 14:34:41.836371 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:41.836338    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de41b0a-d8eb-483b-a5df-60fd784535ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d0b89a-t574k\" (UID: \"1de41b0a-d8eb-483b-a5df-60fd784535ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0b89a-t574k"
Sep 16 14:34:41.836978 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:41.836955    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de41b0a-d8eb-483b-a5df-60fd784535ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d0b89a-t574k\" (UID: \"1de41b0a-d8eb-483b-a5df-60fd784535ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0b89a-t574k"
Sep 16 14:34:41.903365 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:41.903330    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5764823f-b456-4a9a-95aa-eb8c2cd891af" path="/var/lib/kubelet/pods/5764823f-b456-4a9a-95aa-eb8c2cd891af/volumes"
Sep 16 14:34:41.903590 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:34:41.903566    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7605-f13f6f\" is forbidden: User \"system:node:ip-10-0-132-217.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-217.ec2.internal' and this object" podUID="5764823f-b456-4a9a-95aa-eb8c2cd891af" pod="opendatahub/test-curl-7605-f13f6f"
Sep 16 14:34:41.997745 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:41.997660    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0b89a-t574k"
Sep 16 14:34:42.280135 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:42.280105    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0b89a-t574k"]
Sep 16 14:34:42.281817 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:34:42.281794    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de41b0a_d8eb_483b_a5df_60fd784535ed.slice/crio-e682b0ea7be82c04bf5306ee5eefcfbc56fa89bd874bc8a47a00ed8865f13f71 WatchSource:0}: Error finding container e682b0ea7be82c04bf5306ee5eefcfbc56fa89bd874bc8a47a00ed8865f13f71: Status 404 returned error can't find the container with id e682b0ea7be82c04bf5306ee5eefcfbc56fa89bd874bc8a47a00ed8865f13f71
Sep 16 14:34:42.931349 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:42.931321    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0b89a-t574k" event={"ID":"1de41b0a-d8eb-483b-a5df-60fd784535ed","Type":"ContainerStarted","Data":"b54644a0e8f6da41a856bdadc22ceab056fe78aacd344525c726c81b5684e834"}
Sep 16 14:34:42.931541 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:42.931373    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0b89a-t574k" event={"ID":"1de41b0a-d8eb-483b-a5df-60fd784535ed","Type":"ContainerStarted","Data":"e682b0ea7be82c04bf5306ee5eefcfbc56fa89bd874bc8a47a00ed8865f13f71"}
Sep 16 14:34:42.969090 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:42.969046    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0b89a-t574k" podStartSLOduration=1.969031392 podStartE2EDuration="1.969031392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:34:42.967785903 +0000 UTC m=+1231.571410936" watchObservedRunningTime="2026-09-16 14:34:42.969031392 +0000 UTC m=+1231.572656425"
Sep 16 14:34:43.938089 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:43.938053    2587 generic.go:356] "Generic (PLEG): container finished" podID="1de41b0a-d8eb-483b-a5df-60fd784535ed" containerID="b54644a0e8f6da41a856bdadc22ceab056fe78aacd344525c726c81b5684e834" exitCode=0
Sep 16 14:34:43.938529 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:43.938136    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0b89a-t574k" event={"ID":"1de41b0a-d8eb-483b-a5df-60fd784535ed","Type":"ContainerDied","Data":"b54644a0e8f6da41a856bdadc22ceab056fe78aacd344525c726c81b5684e834"}
Sep 16 14:34:45.082293 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:45.082257    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0b89a-t574k"
Sep 16 14:34:45.172657 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:45.172625    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de41b0a-d8eb-483b-a5df-60fd784535ed-maas-api-service-ca\") pod \"1de41b0a-d8eb-483b-a5df-60fd784535ed\" (UID: \"1de41b0a-d8eb-483b-a5df-60fd784535ed\") "
Sep 16 14:34:45.172950 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:45.172930    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1de41b0a-d8eb-483b-a5df-60fd784535ed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1de41b0a-d8eb-483b-a5df-60fd784535ed" (UID: "1de41b0a-d8eb-483b-a5df-60fd784535ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:34:45.273373 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:45.273281    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de41b0a-d8eb-483b-a5df-60fd784535ed-maas-api-service-ca\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:34:45.950321 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:45.950289    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0b89a-t574k"
Sep 16 14:34:45.950321 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:45.950306    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0b89a-t574k" event={"ID":"1de41b0a-d8eb-483b-a5df-60fd784535ed","Type":"ContainerDied","Data":"e682b0ea7be82c04bf5306ee5eefcfbc56fa89bd874bc8a47a00ed8865f13f71"}
Sep 16 14:34:45.950545 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:45.950364    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e682b0ea7be82c04bf5306ee5eefcfbc56fa89bd874bc8a47a00ed8865f13f71"
Sep 16 14:34:49.526384 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:49.526349    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2"]
Sep 16 14:34:49.526797 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:49.526782    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1de41b0a-d8eb-483b-a5df-60fd784535ed" containerName="revoke-keys"
Sep 16 14:34:49.526860 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:49.526799    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="1de41b0a-d8eb-483b-a5df-60fd784535ed" containerName="revoke-keys"
Sep 16 14:34:49.526908 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:49.526895    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="1de41b0a-d8eb-483b-a5df-60fd784535ed" containerName="revoke-keys"
Sep 16 14:34:49.530300 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:49.530284    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2"
Sep 16 14:34:49.533239 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:49.533220    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7b12b9e3\""
Sep 16 14:34:49.534261 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:49.534245    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7b12b9e3-dockercfg-s2qzg\""
Sep 16 14:34:49.550683 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:49.550657    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2"]
Sep 16 14:34:49.582506 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:49.582487    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx"]
Sep 16 14:34:49.586027 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:49.586013    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx"
Sep 16 14:34:49.602853 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:49.602827    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx"]
Sep 16 14:34:49.616659 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:49.616635    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfvsr\" (UniqueName: \"kubernetes.io/projected/ea2ff83a-e0ff-4129-a7de-cb2ae1be381e-kube-api-access-nfvsr\") pod \"payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2\" (UID: \"ea2ff83a-e0ff-4129-a7de-cb2ae1be381e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2"
Sep 16 14:34:49.616761 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:49.616731    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea2ff83a-e0ff-4129-a7de-cb2ae1be381e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2\" (UID: \"ea2ff83a-e0ff-4129-a7de-cb2ae1be381e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2"
Sep 16 14:34:49.717582 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:49.717549    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7d9238a-3482-4937-b8d0-de24f0bebd4b-plugins-config-volume\") pod \"payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx\" (UID: \"e7d9238a-3482-4937-b8d0-de24f0bebd4b\") " pod="openshift-ingress/payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx"
Sep 16 14:34:49.717748 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:49.717620    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea2ff83a-e0ff-4129-a7de-cb2ae1be381e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2\" (UID: \"ea2ff83a-e0ff-4129-a7de-cb2ae1be381e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2"
Sep 16 14:34:49.717748 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:49.717704    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfvsr\" (UniqueName: \"kubernetes.io/projected/ea2ff83a-e0ff-4129-a7de-cb2ae1be381e-kube-api-access-nfvsr\") pod \"payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2\" (UID: \"ea2ff83a-e0ff-4129-a7de-cb2ae1be381e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2"
Sep 16 14:34:49.717748 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:49.717739    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkkmr\" (UniqueName: \"kubernetes.io/projected/e7d9238a-3482-4937-b8d0-de24f0bebd4b-kube-api-access-wkkmr\") pod \"payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx\" (UID: \"e7d9238a-3482-4937-b8d0-de24f0bebd4b\") " pod="openshift-ingress/payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx"
Sep 16 14:34:49.718222 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:49.718205    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea2ff83a-e0ff-4129-a7de-cb2ae1be381e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2\" (UID: \"ea2ff83a-e0ff-4129-a7de-cb2ae1be381e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2"
Sep 16 14:34:49.727420 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:49.727377    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfvsr\" (UniqueName: \"kubernetes.io/projected/ea2ff83a-e0ff-4129-a7de-cb2ae1be381e-kube-api-access-nfvsr\") pod \"payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2\" (UID: \"ea2ff83a-e0ff-4129-a7de-cb2ae1be381e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2"
Sep 16 14:34:49.818693 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:49.818611    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7d9238a-3482-4937-b8d0-de24f0bebd4b-plugins-config-volume\") pod \"payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx\" (UID: \"e7d9238a-3482-4937-b8d0-de24f0bebd4b\") " pod="openshift-ingress/payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx"
Sep 16 14:34:49.818831 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:49.818748    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkkmr\" (UniqueName: \"kubernetes.io/projected/e7d9238a-3482-4937-b8d0-de24f0bebd4b-kube-api-access-wkkmr\") pod \"payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx\" (UID: \"e7d9238a-3482-4937-b8d0-de24f0bebd4b\") " pod="openshift-ingress/payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx"
Sep 16 14:34:49.819428 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:49.819377    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7d9238a-3482-4937-b8d0-de24f0bebd4b-plugins-config-volume\") pod \"payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx\" (UID: \"e7d9238a-3482-4937-b8d0-de24f0bebd4b\") " pod="openshift-ingress/payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx"
Sep 16 14:34:49.827967 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:49.827946    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkkmr\" (UniqueName: \"kubernetes.io/projected/e7d9238a-3482-4937-b8d0-de24f0bebd4b-kube-api-access-wkkmr\") pod \"payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx\" (UID: \"e7d9238a-3482-4937-b8d0-de24f0bebd4b\") " pod="openshift-ingress/payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx"
Sep 16 14:34:49.840808 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:49.840784    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2"
Sep 16 14:34:49.901428 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:49.901169    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx"
Sep 16 14:34:50.122271 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:50.122242    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2"]
Sep 16 14:34:50.124758 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:34:50.124726    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea2ff83a_e0ff_4129_a7de_cb2ae1be381e.slice/crio-5a81c2754e18776e91838a1472a82b96a72cacaadd17d3cc5ef0c626f0bbf5bf WatchSource:0}: Error finding container 5a81c2754e18776e91838a1472a82b96a72cacaadd17d3cc5ef0c626f0bbf5bf: Status 404 returned error can't find the container with id 5a81c2754e18776e91838a1472a82b96a72cacaadd17d3cc5ef0c626f0bbf5bf
Sep 16 14:34:50.185361 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:50.185338    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx"]
Sep 16 14:34:50.188818 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:34:50.188789    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d9238a_3482_4937_b8d0_de24f0bebd4b.slice/crio-8599b245bb2f4c0d1e74d706d7e22330c804cc31d0003f5ccaed1134aa5842d8 WatchSource:0}: Error finding container 8599b245bb2f4c0d1e74d706d7e22330c804cc31d0003f5ccaed1134aa5842d8: Status 404 returned error can't find the container with id 8599b245bb2f4c0d1e74d706d7e22330c804cc31d0003f5ccaed1134aa5842d8
Sep 16 14:34:50.983678 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:50.983629    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx" event={"ID":"e7d9238a-3482-4937-b8d0-de24f0bebd4b","Type":"ContainerStarted","Data":"c7859cf72a17cb5b63943420b00c005364084178d5d1007753ff36e9d63693a0"}
Sep 16 14:34:50.984112 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:50.983699    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx" event={"ID":"e7d9238a-3482-4937-b8d0-de24f0bebd4b","Type":"ContainerStarted","Data":"8599b245bb2f4c0d1e74d706d7e22330c804cc31d0003f5ccaed1134aa5842d8"}
Sep 16 14:34:50.984112 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:50.983838    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx"
Sep 16 14:34:50.985485 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:50.985461    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2" event={"ID":"ea2ff83a-e0ff-4129-a7de-cb2ae1be381e","Type":"ContainerStarted","Data":"b73bf6bbd79204751479c48eda76d5d19569aef2d3e77dd670cd95a6abb0e022"}
Sep 16 14:34:50.985613 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:50.985507    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2" event={"ID":"ea2ff83a-e0ff-4129-a7de-cb2ae1be381e","Type":"ContainerStarted","Data":"5a81c2754e18776e91838a1472a82b96a72cacaadd17d3cc5ef0c626f0bbf5bf"}
Sep 16 14:34:50.985613 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:50.985530    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2"
Sep 16 14:34:51.007689 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:51.007645    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx" podStartSLOduration=2.007632024 podStartE2EDuration="2.007632024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:34:51.007169009 +0000 UTC m=+1239.610794042" watchObservedRunningTime="2026-09-16 14:34:51.007632024 +0000 UTC m=+1239.611257056"
Sep 16 14:34:51.035757 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:51.035717    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2" podStartSLOduration=2.035702538 podStartE2EDuration="2.035702538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:34:51.034150328 +0000 UTC m=+1239.637775373" watchObservedRunningTime="2026-09-16 14:34:51.035702538 +0000 UTC m=+1239.639327569"
Sep 16 14:34:52.628911 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:52.628879    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0b89a-29826150-vz9g8"]
Sep 16 14:34:52.635986 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:52.635955    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0b89a-29826150-vz9g8"]
Sep 16 14:34:52.660811 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:52.660786    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk"]
Sep 16 14:34:52.661088 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:52.661063    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk" podUID="84948549-f033-4ba9-97f6-71e5284d9391" containerName="payload-processing" containerID="cri-o://eddf81146969c4521f8a55f494d7cd864c3aa12d734d117bf42cd5b2a939eebe" gracePeriod=30
Sep 16 14:34:52.678605 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:52.678578    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2"]
Sep 16 14:34:52.678845 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:52.678817    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2" podUID="3a939cd7-244b-4af2-840e-d5704891cd4e" containerName="payload-pre-processing" containerID="cri-o://e9655d2049106e69e33667eeba7afc8850053e6240586cf312fd62b6fa16ccf5" gracePeriod=30
Sep 16 14:34:52.951883 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:52.951793    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk"
Sep 16 14:34:52.953645 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:52.953609    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0b89a-t574k"]
Sep 16 14:34:52.957317 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:52.957297    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0b89a-t574k"]
Sep 16 14:34:52.970111 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:52.970095    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2"
Sep 16 14:34:52.999573 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:52.999087    2587 generic.go:356] "Generic (PLEG): container finished" podID="3a939cd7-244b-4af2-840e-d5704891cd4e" containerID="e9655d2049106e69e33667eeba7afc8850053e6240586cf312fd62b6fa16ccf5" exitCode=0
Sep 16 14:34:52.999573 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:52.999254    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2"
Sep 16 14:34:52.999573 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:52.999295    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2" event={"ID":"3a939cd7-244b-4af2-840e-d5704891cd4e","Type":"ContainerDied","Data":"e9655d2049106e69e33667eeba7afc8850053e6240586cf312fd62b6fa16ccf5"}
Sep 16 14:34:52.999573 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:52.999349    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2" event={"ID":"3a939cd7-244b-4af2-840e-d5704891cd4e","Type":"ContainerDied","Data":"2e826b7614d5b815b1aeb4a5677bdce3177ca709208bd54a06cbb18e30dea8b7"}
Sep 16 14:34:52.999573 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:52.999378    2587 scope.go:117] "RemoveContainer" containerID="e9655d2049106e69e33667eeba7afc8850053e6240586cf312fd62b6fa16ccf5"
Sep 16 14:34:53.001501 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.001434    2587 generic.go:356] "Generic (PLEG): container finished" podID="84948549-f033-4ba9-97f6-71e5284d9391" containerID="eddf81146969c4521f8a55f494d7cd864c3aa12d734d117bf42cd5b2a939eebe" exitCode=0
Sep 16 14:34:53.001501 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.001474    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk"
Sep 16 14:34:53.001669 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.001465    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk" event={"ID":"84948549-f033-4ba9-97f6-71e5284d9391","Type":"ContainerDied","Data":"eddf81146969c4521f8a55f494d7cd864c3aa12d734d117bf42cd5b2a939eebe"}
Sep 16 14:34:53.001669 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.001633    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk" event={"ID":"84948549-f033-4ba9-97f6-71e5284d9391","Type":"ContainerDied","Data":"d33e53b32ed6853482231aac91c2cc2530ce3ef49f5cb47ce4d6de026365d561"}
Sep 16 14:34:53.009529 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.009511    2587 scope.go:117] "RemoveContainer" containerID="e9655d2049106e69e33667eeba7afc8850053e6240586cf312fd62b6fa16ccf5"
Sep 16 14:34:53.009836 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:34:53.009786    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9655d2049106e69e33667eeba7afc8850053e6240586cf312fd62b6fa16ccf5\": container with ID starting with e9655d2049106e69e33667eeba7afc8850053e6240586cf312fd62b6fa16ccf5 not found: ID does not exist" containerID="e9655d2049106e69e33667eeba7afc8850053e6240586cf312fd62b6fa16ccf5"
Sep 16 14:34:53.009836 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.009811    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9655d2049106e69e33667eeba7afc8850053e6240586cf312fd62b6fa16ccf5"} err="failed to get container status \"e9655d2049106e69e33667eeba7afc8850053e6240586cf312fd62b6fa16ccf5\": rpc error: code = NotFound desc = could not find container \"e9655d2049106e69e33667eeba7afc8850053e6240586cf312fd62b6fa16ccf5\": container with ID starting with e9655d2049106e69e33667eeba7afc8850053e6240586cf312fd62b6fa16ccf5 not found: ID does not exist"
Sep 16 14:34:53.009836 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.009829    2587 scope.go:117] "RemoveContainer" containerID="eddf81146969c4521f8a55f494d7cd864c3aa12d734d117bf42cd5b2a939eebe"
Sep 16 14:34:53.019146 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.019124    2587 scope.go:117] "RemoveContainer" containerID="eddf81146969c4521f8a55f494d7cd864c3aa12d734d117bf42cd5b2a939eebe"
Sep 16 14:34:53.019521 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:34:53.019499    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eddf81146969c4521f8a55f494d7cd864c3aa12d734d117bf42cd5b2a939eebe\": container with ID starting with eddf81146969c4521f8a55f494d7cd864c3aa12d734d117bf42cd5b2a939eebe not found: ID does not exist" containerID="eddf81146969c4521f8a55f494d7cd864c3aa12d734d117bf42cd5b2a939eebe"
Sep 16 14:34:53.019601 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.019525    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eddf81146969c4521f8a55f494d7cd864c3aa12d734d117bf42cd5b2a939eebe"} err="failed to get container status \"eddf81146969c4521f8a55f494d7cd864c3aa12d734d117bf42cd5b2a939eebe\": rpc error: code = NotFound desc = could not find container \"eddf81146969c4521f8a55f494d7cd864c3aa12d734d117bf42cd5b2a939eebe\": container with ID starting with eddf81146969c4521f8a55f494d7cd864c3aa12d734d117bf42cd5b2a939eebe not found: ID does not exist"
Sep 16 14:34:53.052888 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.052868    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84948549-f033-4ba9-97f6-71e5284d9391-plugins-config-volume\") pod \"84948549-f033-4ba9-97f6-71e5284d9391\" (UID: \"84948549-f033-4ba9-97f6-71e5284d9391\") "
Sep 16 14:34:53.052998 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.052950    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a939cd7-244b-4af2-840e-d5704891cd4e-plugins-config-volume\") pod \"3a939cd7-244b-4af2-840e-d5704891cd4e\" (UID: \"3a939cd7-244b-4af2-840e-d5704891cd4e\") "
Sep 16 14:34:53.052998 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.052985    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zp5gm\" (UniqueName: \"kubernetes.io/projected/84948549-f033-4ba9-97f6-71e5284d9391-kube-api-access-zp5gm\") pod \"84948549-f033-4ba9-97f6-71e5284d9391\" (UID: \"84948549-f033-4ba9-97f6-71e5284d9391\") "
Sep 16 14:34:53.053121 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.053092    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6v7gb\" (UniqueName: \"kubernetes.io/projected/3a939cd7-244b-4af2-840e-d5704891cd4e-kube-api-access-6v7gb\") pod \"3a939cd7-244b-4af2-840e-d5704891cd4e\" (UID: \"3a939cd7-244b-4af2-840e-d5704891cd4e\") "
Sep 16 14:34:53.053264 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.053238    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84948549-f033-4ba9-97f6-71e5284d9391-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "84948549-f033-4ba9-97f6-71e5284d9391" (UID: "84948549-f033-4ba9-97f6-71e5284d9391"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:34:53.053330 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.053306    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a939cd7-244b-4af2-840e-d5704891cd4e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3a939cd7-244b-4af2-840e-d5704891cd4e" (UID: "3a939cd7-244b-4af2-840e-d5704891cd4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:34:53.053962 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.053933    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84948549-f033-4ba9-97f6-71e5284d9391-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:34:53.053962 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.053955    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a939cd7-244b-4af2-840e-d5704891cd4e-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:34:53.055296 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.055274    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a939cd7-244b-4af2-840e-d5704891cd4e-kube-api-access-6v7gb" (OuterVolumeSpecName: "kube-api-access-6v7gb") pod "3a939cd7-244b-4af2-840e-d5704891cd4e" (UID: "3a939cd7-244b-4af2-840e-d5704891cd4e"). InnerVolumeSpecName "kube-api-access-6v7gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:34:53.055772 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.055750    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84948549-f033-4ba9-97f6-71e5284d9391-kube-api-access-zp5gm" (OuterVolumeSpecName: "kube-api-access-zp5gm") pod "84948549-f033-4ba9-97f6-71e5284d9391" (UID: "84948549-f033-4ba9-97f6-71e5284d9391"). InnerVolumeSpecName "kube-api-access-zp5gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:34:53.155167 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.155139    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp5gm\" (UniqueName: \"kubernetes.io/projected/84948549-f033-4ba9-97f6-71e5284d9391-kube-api-access-zp5gm\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:34:53.155167 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.155165    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v7gb\" (UniqueName: \"kubernetes.io/projected/3a939cd7-244b-4af2-840e-d5704891cd4e-kube-api-access-6v7gb\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:34:53.349867 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.349838    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2"]
Sep 16 14:34:53.359978 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.359953    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d0b89a-686b784b8b-sd8w2"]
Sep 16 14:34:53.377724 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.377699    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk"]
Sep 16 14:34:53.382331 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.382308    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d0b89a-7bbd5994bc-zw6tk"]
Sep 16 14:34:53.903314 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.903287    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1de41b0a-d8eb-483b-a5df-60fd784535ed" path="/var/lib/kubelet/pods/1de41b0a-d8eb-483b-a5df-60fd784535ed/volumes"
Sep 16 14:34:53.903686 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.903657    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a939cd7-244b-4af2-840e-d5704891cd4e" path="/var/lib/kubelet/pods/3a939cd7-244b-4af2-840e-d5704891cd4e/volumes"
Sep 16 14:34:53.903895 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.903881    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82a6caff-e0df-4d1e-9bab-56052ba7ddfe" path="/var/lib/kubelet/pods/82a6caff-e0df-4d1e-9bab-56052ba7ddfe/volumes"
Sep 16 14:34:53.904160 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:34:53.904149    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84948549-f033-4ba9-97f6-71e5284d9391" path="/var/lib/kubelet/pods/84948549-f033-4ba9-97f6-71e5284d9391/volumes"
Sep 16 14:35:01.993605 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:01.993570    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx"
Sep 16 14:35:01.994054 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:01.993754    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2"
Sep 16 14:35:11.984133 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:11.984107    2587 scope.go:117] "RemoveContainer" containerID="4611c4f2c63fcdc46e8d649b1e763c29f44e93171e73f95ffee0d33193ccdd3b"
Sep 16 14:35:39.735119 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:39.735082    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj"]
Sep 16 14:35:39.735610 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:39.735510    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a939cd7-244b-4af2-840e-d5704891cd4e" containerName="payload-pre-processing"
Sep 16 14:35:39.735610 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:39.735523    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a939cd7-244b-4af2-840e-d5704891cd4e" containerName="payload-pre-processing"
Sep 16 14:35:39.735610 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:39.735535    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84948549-f033-4ba9-97f6-71e5284d9391" containerName="payload-processing"
Sep 16 14:35:39.735610 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:39.735541    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="84948549-f033-4ba9-97f6-71e5284d9391" containerName="payload-processing"
Sep 16 14:35:39.735742 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:39.735624    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a939cd7-244b-4af2-840e-d5704891cd4e" containerName="payload-pre-processing"
Sep 16 14:35:39.735742 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:39.735632    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="84948549-f033-4ba9-97f6-71e5284d9391" containerName="payload-processing"
Sep 16 14:35:39.738653 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:39.738638    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj"
Sep 16 14:35:39.742073 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:39.742054    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-9f88e1\""
Sep 16 14:35:39.757329 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:39.757304    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj"]
Sep 16 14:35:39.763810 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:39.763782    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6"]
Sep 16 14:35:39.767277 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:39.767257    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6"
Sep 16 14:35:39.795501 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:39.795475    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6"]
Sep 16 14:35:39.891117 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:39.891090    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d0b1b65-75e8-4e27-8fdd-0061cf297144-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6\" (UID: \"6d0b1b65-75e8-4e27-8fdd-0061cf297144\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6"
Sep 16 14:35:39.891282 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:39.891142    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s2kr\" (UniqueName: \"kubernetes.io/projected/51fc9064-7c2f-4b29-80a2-bd9747098af1-kube-api-access-5s2kr\") pod \"payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj\" (UID: \"51fc9064-7c2f-4b29-80a2-bd9747098af1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj"
Sep 16 14:35:39.891282 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:39.891225    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51fc9064-7c2f-4b29-80a2-bd9747098af1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj\" (UID: \"51fc9064-7c2f-4b29-80a2-bd9747098af1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj"
Sep 16 14:35:39.891364 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:39.891335    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl9vt\" (UniqueName: \"kubernetes.io/projected/6d0b1b65-75e8-4e27-8fdd-0061cf297144-kube-api-access-pl9vt\") pod \"payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6\" (UID: \"6d0b1b65-75e8-4e27-8fdd-0061cf297144\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6"
Sep 16 14:35:39.992263 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:39.992174    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl9vt\" (UniqueName: \"kubernetes.io/projected/6d0b1b65-75e8-4e27-8fdd-0061cf297144-kube-api-access-pl9vt\") pod \"payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6\" (UID: \"6d0b1b65-75e8-4e27-8fdd-0061cf297144\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6"
Sep 16 14:35:39.992263 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:39.992219    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d0b1b65-75e8-4e27-8fdd-0061cf297144-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6\" (UID: \"6d0b1b65-75e8-4e27-8fdd-0061cf297144\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6"
Sep 16 14:35:39.992560 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:39.992291    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s2kr\" (UniqueName: \"kubernetes.io/projected/51fc9064-7c2f-4b29-80a2-bd9747098af1-kube-api-access-5s2kr\") pod \"payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj\" (UID: \"51fc9064-7c2f-4b29-80a2-bd9747098af1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj"
Sep 16 14:35:39.992560 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:39.992385    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51fc9064-7c2f-4b29-80a2-bd9747098af1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj\" (UID: \"51fc9064-7c2f-4b29-80a2-bd9747098af1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj"
Sep 16 14:35:39.993067 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:39.993046    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51fc9064-7c2f-4b29-80a2-bd9747098af1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj\" (UID: \"51fc9064-7c2f-4b29-80a2-bd9747098af1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj"
Sep 16 14:35:39.993115 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:39.993076    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d0b1b65-75e8-4e27-8fdd-0061cf297144-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6\" (UID: \"6d0b1b65-75e8-4e27-8fdd-0061cf297144\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6"
Sep 16 14:35:40.030531 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:40.030505    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl9vt\" (UniqueName: \"kubernetes.io/projected/6d0b1b65-75e8-4e27-8fdd-0061cf297144-kube-api-access-pl9vt\") pod \"payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6\" (UID: \"6d0b1b65-75e8-4e27-8fdd-0061cf297144\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6"
Sep 16 14:35:40.032288 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:40.032270    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s2kr\" (UniqueName: \"kubernetes.io/projected/51fc9064-7c2f-4b29-80a2-bd9747098af1-kube-api-access-5s2kr\") pod \"payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj\" (UID: \"51fc9064-7c2f-4b29-80a2-bd9747098af1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj"
Sep 16 14:35:40.049227 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:40.049203    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj"
Sep 16 14:35:40.076534 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:40.076471    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6"
Sep 16 14:35:40.349906 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:40.349879    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj"]
Sep 16 14:35:40.352132 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:35:40.352109    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fc9064_7c2f_4b29_80a2_bd9747098af1.slice/crio-370b9a770b7aa775a7e11cca7ceaef2b9a2662e4b8df06252569adf03c0ad7bc WatchSource:0}: Error finding container 370b9a770b7aa775a7e11cca7ceaef2b9a2662e4b8df06252569adf03c0ad7bc: Status 404 returned error can't find the container with id 370b9a770b7aa775a7e11cca7ceaef2b9a2662e4b8df06252569adf03c0ad7bc
Sep 16 14:35:40.396372 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:40.396343    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6"]
Sep 16 14:35:40.397213 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:35:40.397193    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d0b1b65_75e8_4e27_8fdd_0061cf297144.slice/crio-19d03fd830cebb8a70c9235c03f632e93c37ec075b85d5b99e2fbeb5e7f3220a WatchSource:0}: Error finding container 19d03fd830cebb8a70c9235c03f632e93c37ec075b85d5b99e2fbeb5e7f3220a: Status 404 returned error can't find the container with id 19d03fd830cebb8a70c9235c03f632e93c37ec075b85d5b99e2fbeb5e7f3220a
Sep 16 14:35:41.278367 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:41.278326    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6" event={"ID":"6d0b1b65-75e8-4e27-8fdd-0061cf297144","Type":"ContainerStarted","Data":"5a057ac2ba9b84c3fb55a84551aa6dbf445cb7c9a860b8616b7cfa5e9d612725"}
Sep 16 14:35:41.278825 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:41.278382    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6" event={"ID":"6d0b1b65-75e8-4e27-8fdd-0061cf297144","Type":"ContainerStarted","Data":"19d03fd830cebb8a70c9235c03f632e93c37ec075b85d5b99e2fbeb5e7f3220a"}
Sep 16 14:35:41.278825 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:41.278434    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6"
Sep 16 14:35:41.279965 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:41.279939    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj" event={"ID":"51fc9064-7c2f-4b29-80a2-bd9747098af1","Type":"ContainerStarted","Data":"13d8de68ab1d044925ffc747d90f8a60c17d5ecc5778fb035b19ea6bc73dd508"}
Sep 16 14:35:41.280066 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:41.279984    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj" event={"ID":"51fc9064-7c2f-4b29-80a2-bd9747098af1","Type":"ContainerStarted","Data":"370b9a770b7aa775a7e11cca7ceaef2b9a2662e4b8df06252569adf03c0ad7bc"}
Sep 16 14:35:41.280066 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:41.280020    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj"
Sep 16 14:35:41.308643 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:41.308596    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6" podStartSLOduration=2.308581219 podStartE2EDuration="2.308581219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:35:41.307781171 +0000 UTC m=+1289.911406203" watchObservedRunningTime="2026-09-16 14:35:41.308581219 +0000 UTC m=+1289.912206250"
Sep 16 14:35:41.342948 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:41.342897    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj" podStartSLOduration=2.342883122 podStartE2EDuration="2.342883122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:35:41.340252738 +0000 UTC m=+1289.943877771" watchObservedRunningTime="2026-09-16 14:35:41.342883122 +0000 UTC m=+1289.946508177"
Sep 16 14:35:52.287836 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:52.287807    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj"
Sep 16 14:35:52.288225 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:35:52.287875    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6"
Sep 16 14:36:11.565329 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:36:11.565291    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl"]
Sep 16 14:36:11.569540 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:36:11.569517    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl"
Sep 16 14:36:11.572493 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:36:11.572475    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-055ce3-dockercfg-s78p7\""
Sep 16 14:36:11.572747 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:36:11.572729    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-055ce3\""
Sep 16 14:36:11.584083 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:36:11.584064    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl"]
Sep 16 14:36:11.692078 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:36:11.692042    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff38b70d-db5f-4464-acf6-bbbaeb77f70c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl\" (UID: \"ff38b70d-db5f-4464-acf6-bbbaeb77f70c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl"
Sep 16 14:36:11.692220 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:36:11.692097    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnr8s\" (UniqueName: \"kubernetes.io/projected/ff38b70d-db5f-4464-acf6-bbbaeb77f70c-kube-api-access-rnr8s\") pod \"payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl\" (UID: \"ff38b70d-db5f-4464-acf6-bbbaeb77f70c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl"
Sep 16 14:36:11.792872 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:36:11.792841    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff38b70d-db5f-4464-acf6-bbbaeb77f70c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl\" (UID: \"ff38b70d-db5f-4464-acf6-bbbaeb77f70c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl"
Sep 16 14:36:11.793031 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:36:11.792901    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnr8s\" (UniqueName: \"kubernetes.io/projected/ff38b70d-db5f-4464-acf6-bbbaeb77f70c-kube-api-access-rnr8s\") pod \"payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl\" (UID: \"ff38b70d-db5f-4464-acf6-bbbaeb77f70c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl"
Sep 16 14:36:11.795584 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:36:11.795565    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-055ce3\""
Sep 16 14:36:11.802490 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:36:11.802462    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnr8s\" (UniqueName: \"kubernetes.io/projected/ff38b70d-db5f-4464-acf6-bbbaeb77f70c-kube-api-access-rnr8s\") pod \"payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl\" (UID: \"ff38b70d-db5f-4464-acf6-bbbaeb77f70c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl"
Sep 16 14:36:11.803602 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:36:11.803584    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff38b70d-db5f-4464-acf6-bbbaeb77f70c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl\" (UID: \"ff38b70d-db5f-4464-acf6-bbbaeb77f70c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl"
Sep 16 14:36:11.883362 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:36:11.883280    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-055ce3-dockercfg-s78p7\""
Sep 16 14:36:11.890465 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:36:11.890442    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl"
Sep 16 14:36:12.032163 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:36:12.032130    2587 scope.go:117] "RemoveContainer" containerID="b6ffe6e36775ca2d8333b4a0b371bbdcf980137333889ce7defacd385c8ef2b3"
Sep 16 14:36:12.041369 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:36:12.041350    2587 scope.go:117] "RemoveContainer" containerID="f96779c179520859ed64ec8b52810fa6e82a81d652e03bf0f4b0362442149e27"
Sep 16 14:36:12.049603 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:36:12.049586    2587 scope.go:117] "RemoveContainer" containerID="76c9b2587d03b96653af49aa716d5499c076c0864e77eb5f0003bf635bce745b"
Sep 16 14:36:12.182684 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:36:12.182654    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl"]
Sep 16 14:36:12.185096 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:36:12.185063    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff38b70d_db5f_4464_acf6_bbbaeb77f70c.slice/crio-c7bb8e2e205d8d5f3f6944caf7da403c5eb2073b51dd6d3fdc66ffe7693e5578 WatchSource:0}: Error finding container c7bb8e2e205d8d5f3f6944caf7da403c5eb2073b51dd6d3fdc66ffe7693e5578: Status 404 returned error can't find the container with id c7bb8e2e205d8d5f3f6944caf7da403c5eb2073b51dd6d3fdc66ffe7693e5578
Sep 16 14:36:12.455672 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:36:12.455591    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl" event={"ID":"ff38b70d-db5f-4464-acf6-bbbaeb77f70c","Type":"ContainerStarted","Data":"6958e2e31a73960eab29156c180e1f69afff15802e8fdf38a19d2c5df2047bf0"}
Sep 16 14:36:12.455672 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:36:12.455646    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl" event={"ID":"ff38b70d-db5f-4464-acf6-bbbaeb77f70c","Type":"ContainerStarted","Data":"c7bb8e2e205d8d5f3f6944caf7da403c5eb2073b51dd6d3fdc66ffe7693e5578"}
Sep 16 14:36:12.455893 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:36:12.455714    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl"
Sep 16 14:36:12.500966 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:36:12.500921    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl" podStartSLOduration=1.500906113 podStartE2EDuration="1.500906113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:36:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:36:12.498583283 +0000 UTC m=+1321.102208316" watchObservedRunningTime="2026-09-16 14:36:12.500906113 +0000 UTC m=+1321.104531154"
Sep 16 14:36:23.463961 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:36:23.463931    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl"
Sep 16 14:37:22.589519 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:22.589488    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7b12b9e3-flf9k"]
Sep 16 14:37:22.593253 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:22.593234    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7b12b9e3-flf9k"
Sep 16 14:37:22.595602 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:22.595575    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:37:22.595742 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:22.595685    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4jf\""
Sep 16 14:37:22.600547 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:22.600470    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7b12b9e3-flf9k"]
Sep 16 14:37:22.667538 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:22.667500    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f582e46-e8f4-4244-b9cc-e338d41dde18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7b12b9e3-flf9k\" (UID: \"6f582e46-e8f4-4244-b9cc-e338d41dde18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7b12b9e3-flf9k"
Sep 16 14:37:22.768604 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:22.768556    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f582e46-e8f4-4244-b9cc-e338d41dde18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7b12b9e3-flf9k\" (UID: \"6f582e46-e8f4-4244-b9cc-e338d41dde18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7b12b9e3-flf9k"
Sep 16 14:37:22.769220 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:22.769193    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f582e46-e8f4-4244-b9cc-e338d41dde18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7b12b9e3-flf9k\" (UID: \"6f582e46-e8f4-4244-b9cc-e338d41dde18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7b12b9e3-flf9k"
Sep 16 14:37:22.906057 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:22.906024    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7b12b9e3-flf9k"
Sep 16 14:37:23.185627 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:23.185601    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7b12b9e3-flf9k"]
Sep 16 14:37:23.188223 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:37:23.188198    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f582e46_e8f4_4244_b9cc_e338d41dde18.slice/crio-680c3ec8dd259396f172ed38cb4ccd14f0275beb9e0a2e0f5f41280b5693d638 WatchSource:0}: Error finding container 680c3ec8dd259396f172ed38cb4ccd14f0275beb9e0a2e0f5f41280b5693d638: Status 404 returned error can't find the container with id 680c3ec8dd259396f172ed38cb4ccd14f0275beb9e0a2e0f5f41280b5693d638
Sep 16 14:37:23.850227 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:23.850139    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7b12b9e3-flf9k" event={"ID":"6f582e46-e8f4-4244-b9cc-e338d41dde18","Type":"ContainerStarted","Data":"564d951e0312c4b09f914a517e45adc833fb156c7f120a20b02f7f064b743830"}
Sep 16 14:37:23.850227 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:23.850197    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7b12b9e3-flf9k" event={"ID":"6f582e46-e8f4-4244-b9cc-e338d41dde18","Type":"ContainerStarted","Data":"680c3ec8dd259396f172ed38cb4ccd14f0275beb9e0a2e0f5f41280b5693d638"}
Sep 16 14:37:23.875594 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:23.875546    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7b12b9e3-flf9k" podStartSLOduration=1.8755272870000002 podStartE2EDuration="1.875527287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:37:23.873287701 +0000 UTC m=+1392.476912732" watchObservedRunningTime="2026-09-16 14:37:23.875527287 +0000 UTC m=+1392.479152414"
Sep 16 14:37:24.857293 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:24.857258    2587 generic.go:356] "Generic (PLEG): container finished" podID="6f582e46-e8f4-4244-b9cc-e338d41dde18" containerID="564d951e0312c4b09f914a517e45adc833fb156c7f120a20b02f7f064b743830" exitCode=0
Sep 16 14:37:24.857684 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:24.857351    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7b12b9e3-flf9k" event={"ID":"6f582e46-e8f4-4244-b9cc-e338d41dde18","Type":"ContainerDied","Data":"564d951e0312c4b09f914a517e45adc833fb156c7f120a20b02f7f064b743830"}
Sep 16 14:37:26.014741 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:26.014707    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7b12b9e3-flf9k"
Sep 16 14:37:26.099447 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:26.099374    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f582e46-e8f4-4244-b9cc-e338d41dde18-maas-api-service-ca\") pod \"6f582e46-e8f4-4244-b9cc-e338d41dde18\" (UID: \"6f582e46-e8f4-4244-b9cc-e338d41dde18\") "
Sep 16 14:37:26.099820 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:26.099794    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f582e46-e8f4-4244-b9cc-e338d41dde18-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6f582e46-e8f4-4244-b9cc-e338d41dde18" (UID: "6f582e46-e8f4-4244-b9cc-e338d41dde18"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:37:26.201020 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:26.200984    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f582e46-e8f4-4244-b9cc-e338d41dde18-maas-api-service-ca\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:37:26.870612 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:26.870582    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7b12b9e3-flf9k"
Sep 16 14:37:26.870791 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:26.870615    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7b12b9e3-flf9k" event={"ID":"6f582e46-e8f4-4244-b9cc-e338d41dde18","Type":"ContainerDied","Data":"680c3ec8dd259396f172ed38cb4ccd14f0275beb9e0a2e0f5f41280b5693d638"}
Sep 16 14:37:26.870791 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:26.870656    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="680c3ec8dd259396f172ed38cb4ccd14f0275beb9e0a2e0f5f41280b5693d638"
Sep 16 14:37:38.188611 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.188569    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx"]
Sep 16 14:37:38.189183 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.188875    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx" podUID="e7d9238a-3482-4937-b8d0-de24f0bebd4b" containerName="payload-processing" containerID="cri-o://c7859cf72a17cb5b63943420b00c005364084178d5d1007753ff36e9d63693a0" gracePeriod=30
Sep 16 14:37:38.195294 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.195258    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2"]
Sep 16 14:37:38.195554 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.195510    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2" podUID="ea2ff83a-e0ff-4129-a7de-cb2ae1be381e" containerName="payload-pre-processing" containerID="cri-o://b73bf6bbd79204751479c48eda76d5d19569aef2d3e77dd670cd95a6abb0e022" gracePeriod=30
Sep 16 14:37:38.393426 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.393362    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7b12b9e3-flf9k"]
Sep 16 14:37:38.395601 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.395575    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7b12b9e3-flf9k"]
Sep 16 14:37:38.452183 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.452119    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx"
Sep 16 14:37:38.483065 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.483041    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2"
Sep 16 14:37:38.525761 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.525721    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7d9238a-3482-4937-b8d0-de24f0bebd4b-plugins-config-volume\") pod \"e7d9238a-3482-4937-b8d0-de24f0bebd4b\" (UID: \"e7d9238a-3482-4937-b8d0-de24f0bebd4b\") "
Sep 16 14:37:38.525953 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.525831    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkkmr\" (UniqueName: \"kubernetes.io/projected/e7d9238a-3482-4937-b8d0-de24f0bebd4b-kube-api-access-wkkmr\") pod \"e7d9238a-3482-4937-b8d0-de24f0bebd4b\" (UID: \"e7d9238a-3482-4937-b8d0-de24f0bebd4b\") "
Sep 16 14:37:38.526189 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.526158    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7d9238a-3482-4937-b8d0-de24f0bebd4b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e7d9238a-3482-4937-b8d0-de24f0bebd4b" (UID: "e7d9238a-3482-4937-b8d0-de24f0bebd4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:37:38.528157 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.528129    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d9238a-3482-4937-b8d0-de24f0bebd4b-kube-api-access-wkkmr" (OuterVolumeSpecName: "kube-api-access-wkkmr") pod "e7d9238a-3482-4937-b8d0-de24f0bebd4b" (UID: "e7d9238a-3482-4937-b8d0-de24f0bebd4b"). InnerVolumeSpecName "kube-api-access-wkkmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:37:38.627095 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.627053    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfvsr\" (UniqueName: \"kubernetes.io/projected/ea2ff83a-e0ff-4129-a7de-cb2ae1be381e-kube-api-access-nfvsr\") pod \"ea2ff83a-e0ff-4129-a7de-cb2ae1be381e\" (UID: \"ea2ff83a-e0ff-4129-a7de-cb2ae1be381e\") "
Sep 16 14:37:38.627285 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.627111    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea2ff83a-e0ff-4129-a7de-cb2ae1be381e-plugins-config-volume\") pod \"ea2ff83a-e0ff-4129-a7de-cb2ae1be381e\" (UID: \"ea2ff83a-e0ff-4129-a7de-cb2ae1be381e\") "
Sep 16 14:37:38.627349 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.627296    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7d9238a-3482-4937-b8d0-de24f0bebd4b-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:37:38.627349 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.627306    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkkmr\" (UniqueName: \"kubernetes.io/projected/e7d9238a-3482-4937-b8d0-de24f0bebd4b-kube-api-access-wkkmr\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:37:38.627580 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.627553    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea2ff83a-e0ff-4129-a7de-cb2ae1be381e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ea2ff83a-e0ff-4129-a7de-cb2ae1be381e" (UID: "ea2ff83a-e0ff-4129-a7de-cb2ae1be381e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:37:38.629290 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.629264    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea2ff83a-e0ff-4129-a7de-cb2ae1be381e-kube-api-access-nfvsr" (OuterVolumeSpecName: "kube-api-access-nfvsr") pod "ea2ff83a-e0ff-4129-a7de-cb2ae1be381e" (UID: "ea2ff83a-e0ff-4129-a7de-cb2ae1be381e"). InnerVolumeSpecName "kube-api-access-nfvsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:37:38.727903 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.727816    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfvsr\" (UniqueName: \"kubernetes.io/projected/ea2ff83a-e0ff-4129-a7de-cb2ae1be381e-kube-api-access-nfvsr\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:37:38.727903 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.727841    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea2ff83a-e0ff-4129-a7de-cb2ae1be381e-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:37:38.938475 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.938437    2587 generic.go:356] "Generic (PLEG): container finished" podID="ea2ff83a-e0ff-4129-a7de-cb2ae1be381e" containerID="b73bf6bbd79204751479c48eda76d5d19569aef2d3e77dd670cd95a6abb0e022" exitCode=0
Sep 16 14:37:38.938672 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.938498    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2"
Sep 16 14:37:38.938672 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.938532    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2" event={"ID":"ea2ff83a-e0ff-4129-a7de-cb2ae1be381e","Type":"ContainerDied","Data":"b73bf6bbd79204751479c48eda76d5d19569aef2d3e77dd670cd95a6abb0e022"}
Sep 16 14:37:38.938672 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.938599    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2" event={"ID":"ea2ff83a-e0ff-4129-a7de-cb2ae1be381e","Type":"ContainerDied","Data":"5a81c2754e18776e91838a1472a82b96a72cacaadd17d3cc5ef0c626f0bbf5bf"}
Sep 16 14:37:38.938672 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.938627    2587 scope.go:117] "RemoveContainer" containerID="b73bf6bbd79204751479c48eda76d5d19569aef2d3e77dd670cd95a6abb0e022"
Sep 16 14:37:38.940156 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.940128    2587 generic.go:356] "Generic (PLEG): container finished" podID="e7d9238a-3482-4937-b8d0-de24f0bebd4b" containerID="c7859cf72a17cb5b63943420b00c005364084178d5d1007753ff36e9d63693a0" exitCode=0
Sep 16 14:37:38.940290 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.940194    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx" event={"ID":"e7d9238a-3482-4937-b8d0-de24f0bebd4b","Type":"ContainerDied","Data":"c7859cf72a17cb5b63943420b00c005364084178d5d1007753ff36e9d63693a0"}
Sep 16 14:37:38.940290 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.940233    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx" event={"ID":"e7d9238a-3482-4937-b8d0-de24f0bebd4b","Type":"ContainerDied","Data":"8599b245bb2f4c0d1e74d706d7e22330c804cc31d0003f5ccaed1134aa5842d8"}
Sep 16 14:37:38.940435 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.940293    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx"
Sep 16 14:37:38.949512 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.949494    2587 scope.go:117] "RemoveContainer" containerID="b73bf6bbd79204751479c48eda76d5d19569aef2d3e77dd670cd95a6abb0e022"
Sep 16 14:37:38.949802 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:37:38.949782    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b73bf6bbd79204751479c48eda76d5d19569aef2d3e77dd670cd95a6abb0e022\": container with ID starting with b73bf6bbd79204751479c48eda76d5d19569aef2d3e77dd670cd95a6abb0e022 not found: ID does not exist" containerID="b73bf6bbd79204751479c48eda76d5d19569aef2d3e77dd670cd95a6abb0e022"
Sep 16 14:37:38.949894 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.949812    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b73bf6bbd79204751479c48eda76d5d19569aef2d3e77dd670cd95a6abb0e022"} err="failed to get container status \"b73bf6bbd79204751479c48eda76d5d19569aef2d3e77dd670cd95a6abb0e022\": rpc error: code = NotFound desc = could not find container \"b73bf6bbd79204751479c48eda76d5d19569aef2d3e77dd670cd95a6abb0e022\": container with ID starting with b73bf6bbd79204751479c48eda76d5d19569aef2d3e77dd670cd95a6abb0e022 not found: ID does not exist"
Sep 16 14:37:38.949894 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.949828    2587 scope.go:117] "RemoveContainer" containerID="c7859cf72a17cb5b63943420b00c005364084178d5d1007753ff36e9d63693a0"
Sep 16 14:37:38.958379 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.958362    2587 scope.go:117] "RemoveContainer" containerID="c7859cf72a17cb5b63943420b00c005364084178d5d1007753ff36e9d63693a0"
Sep 16 14:37:38.958648 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:37:38.958632    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7859cf72a17cb5b63943420b00c005364084178d5d1007753ff36e9d63693a0\": container with ID starting with c7859cf72a17cb5b63943420b00c005364084178d5d1007753ff36e9d63693a0 not found: ID does not exist" containerID="c7859cf72a17cb5b63943420b00c005364084178d5d1007753ff36e9d63693a0"
Sep 16 14:37:38.958715 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.958652    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7859cf72a17cb5b63943420b00c005364084178d5d1007753ff36e9d63693a0"} err="failed to get container status \"c7859cf72a17cb5b63943420b00c005364084178d5d1007753ff36e9d63693a0\": rpc error: code = NotFound desc = could not find container \"c7859cf72a17cb5b63943420b00c005364084178d5d1007753ff36e9d63693a0\": container with ID starting with c7859cf72a17cb5b63943420b00c005364084178d5d1007753ff36e9d63693a0 not found: ID does not exist"
Sep 16 14:37:38.973464 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.973437    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx"]
Sep 16 14:37:38.984323 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.984269    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7b12b9e3-756f69697-rsnrx"]
Sep 16 14:37:38.996259 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:38.996232    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2"]
Sep 16 14:37:39.000563 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:39.000537    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7b12b9e3-59f86fcc58-st5f2"]
Sep 16 14:37:39.902911 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:39.902878    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f582e46-e8f4-4244-b9cc-e338d41dde18" path="/var/lib/kubelet/pods/6f582e46-e8f4-4244-b9cc-e338d41dde18/volumes"
Sep 16 14:37:39.903265 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:39.903238    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7d9238a-3482-4937-b8d0-de24f0bebd4b" path="/var/lib/kubelet/pods/e7d9238a-3482-4937-b8d0-de24f0bebd4b/volumes"
Sep 16 14:37:39.903602 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:37:39.903587    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea2ff83a-e0ff-4129-a7de-cb2ae1be381e" path="/var/lib/kubelet/pods/ea2ff83a-e0ff-4129-a7de-cb2ae1be381e/volumes"
Sep 16 14:38:12.089273 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:12.089237    2587 scope.go:117] "RemoveContainer" containerID="740259bc6966d614bcd8c4196fc8445d2f1ab833d9cb5eed449fde1dcd942a3a"
Sep 16 14:38:36.851891 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.851860    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"]
Sep 16 14:38:36.852346 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.852332    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea2ff83a-e0ff-4129-a7de-cb2ae1be381e" containerName="payload-pre-processing"
Sep 16 14:38:36.852414 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.852348    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea2ff83a-e0ff-4129-a7de-cb2ae1be381e" containerName="payload-pre-processing"
Sep 16 14:38:36.852414 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.852363    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7d9238a-3482-4937-b8d0-de24f0bebd4b" containerName="payload-processing"
Sep 16 14:38:36.852414 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.852368    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7d9238a-3482-4937-b8d0-de24f0bebd4b" containerName="payload-processing"
Sep 16 14:38:36.852414 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.852375    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f582e46-e8f4-4244-b9cc-e338d41dde18" containerName="revoke-keys"
Sep 16 14:38:36.852414 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.852380    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f582e46-e8f4-4244-b9cc-e338d41dde18" containerName="revoke-keys"
Sep 16 14:38:36.852602 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.852484    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7d9238a-3482-4937-b8d0-de24f0bebd4b" containerName="payload-processing"
Sep 16 14:38:36.852602 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.852494    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea2ff83a-e0ff-4129-a7de-cb2ae1be381e" containerName="payload-pre-processing"
Sep 16 14:38:36.852602 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.852500    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f582e46-e8f4-4244-b9cc-e338d41dde18" containerName="revoke-keys"
Sep 16 14:38:36.856800 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.856784    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"
Sep 16 14:38:36.860002 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.859980    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5d484d\"/\"distinct-2-w4-5d484d-kserve-self-signed-certs\""
Sep 16 14:38:36.869200 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.869179    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7373abcb-c40e-4f4b-bd7a-102f243bf1db-home\") pod \"distinct-2-w4-5d484d-kserve-c6777d797-76965\" (UID: \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\") " pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"
Sep 16 14:38:36.869293 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.869212    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7373abcb-c40e-4f4b-bd7a-102f243bf1db-tls-certs\") pod \"distinct-2-w4-5d484d-kserve-c6777d797-76965\" (UID: \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\") " pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"
Sep 16 14:38:36.869293 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.869253    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzkl6\" (UniqueName: \"kubernetes.io/projected/7373abcb-c40e-4f4b-bd7a-102f243bf1db-kube-api-access-pzkl6\") pod \"distinct-2-w4-5d484d-kserve-c6777d797-76965\" (UID: \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\") " pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"
Sep 16 14:38:36.869293 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.869277    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7373abcb-c40e-4f4b-bd7a-102f243bf1db-model-cache\") pod \"distinct-2-w4-5d484d-kserve-c6777d797-76965\" (UID: \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\") " pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"
Sep 16 14:38:36.869470 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.869304    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7373abcb-c40e-4f4b-bd7a-102f243bf1db-dshm\") pod \"distinct-2-w4-5d484d-kserve-c6777d797-76965\" (UID: \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\") " pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"
Sep 16 14:38:36.869470 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.869327    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7373abcb-c40e-4f4b-bd7a-102f243bf1db-tmp-dir\") pod \"distinct-2-w4-5d484d-kserve-c6777d797-76965\" (UID: \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\") " pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"
Sep 16 14:38:36.877532 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.877506    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"]
Sep 16 14:38:36.970029 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.969995    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7373abcb-c40e-4f4b-bd7a-102f243bf1db-model-cache\") pod \"distinct-2-w4-5d484d-kserve-c6777d797-76965\" (UID: \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\") " pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"
Sep 16 14:38:36.970212 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.970059    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7373abcb-c40e-4f4b-bd7a-102f243bf1db-dshm\") pod \"distinct-2-w4-5d484d-kserve-c6777d797-76965\" (UID: \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\") " pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"
Sep 16 14:38:36.970212 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.970097    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7373abcb-c40e-4f4b-bd7a-102f243bf1db-tmp-dir\") pod \"distinct-2-w4-5d484d-kserve-c6777d797-76965\" (UID: \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\") " pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"
Sep 16 14:38:36.970212 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.970137    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7373abcb-c40e-4f4b-bd7a-102f243bf1db-home\") pod \"distinct-2-w4-5d484d-kserve-c6777d797-76965\" (UID: \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\") " pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"
Sep 16 14:38:36.970380 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.970282    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7373abcb-c40e-4f4b-bd7a-102f243bf1db-tls-certs\") pod \"distinct-2-w4-5d484d-kserve-c6777d797-76965\" (UID: \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\") " pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"
Sep 16 14:38:36.970380 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.970372    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzkl6\" (UniqueName: \"kubernetes.io/projected/7373abcb-c40e-4f4b-bd7a-102f243bf1db-kube-api-access-pzkl6\") pod \"distinct-2-w4-5d484d-kserve-c6777d797-76965\" (UID: \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\") " pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"
Sep 16 14:38:36.970723 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.970522    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7373abcb-c40e-4f4b-bd7a-102f243bf1db-model-cache\") pod \"distinct-2-w4-5d484d-kserve-c6777d797-76965\" (UID: \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\") " pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"
Sep 16 14:38:36.970855 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.970724    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7373abcb-c40e-4f4b-bd7a-102f243bf1db-home\") pod \"distinct-2-w4-5d484d-kserve-c6777d797-76965\" (UID: \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\") " pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"
Sep 16 14:38:36.970855 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.970757    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7373abcb-c40e-4f4b-bd7a-102f243bf1db-tmp-dir\") pod \"distinct-2-w4-5d484d-kserve-c6777d797-76965\" (UID: \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\") " pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"
Sep 16 14:38:36.972563 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.972536    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7373abcb-c40e-4f4b-bd7a-102f243bf1db-dshm\") pod \"distinct-2-w4-5d484d-kserve-c6777d797-76965\" (UID: \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\") " pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"
Sep 16 14:38:36.973105 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.973082    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7373abcb-c40e-4f4b-bd7a-102f243bf1db-tls-certs\") pod \"distinct-2-w4-5d484d-kserve-c6777d797-76965\" (UID: \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\") " pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"
Sep 16 14:38:36.994118 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:36.994094    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzkl6\" (UniqueName: \"kubernetes.io/projected/7373abcb-c40e-4f4b-bd7a-102f243bf1db-kube-api-access-pzkl6\") pod \"distinct-2-w4-5d484d-kserve-c6777d797-76965\" (UID: \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\") " pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"
Sep 16 14:38:37.166586 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:37.166558    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"
Sep 16 14:38:37.443945 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:37.443918    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"]
Sep 16 14:38:37.445901 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:38:37.445872    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7373abcb_c40e_4f4b_bd7a_102f243bf1db.slice/crio-e67435be1fd2bdc21130c47da0906a3f21ac19859ee1450b13786648f4e0bf5c WatchSource:0}: Error finding container e67435be1fd2bdc21130c47da0906a3f21ac19859ee1450b13786648f4e0bf5c: Status 404 returned error can't find the container with id e67435be1fd2bdc21130c47da0906a3f21ac19859ee1450b13786648f4e0bf5c
Sep 16 14:38:38.264915 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:38.264870    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965" event={"ID":"7373abcb-c40e-4f4b-bd7a-102f243bf1db","Type":"ContainerStarted","Data":"681af4ad85f93beb5127e57af68128c138996d58d69c4b507ec0bb3c224141ff"}
Sep 16 14:38:38.264915 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:38.264917    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965" event={"ID":"7373abcb-c40e-4f4b-bd7a-102f243bf1db","Type":"ContainerStarted","Data":"e67435be1fd2bdc21130c47da0906a3f21ac19859ee1450b13786648f4e0bf5c"}
Sep 16 14:38:38.291857 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:38.291801    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965" podStartSLOduration=2.291783459 podStartE2EDuration="2.291783459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:38:38.289658783 +0000 UTC m=+1466.893283815" watchObservedRunningTime="2026-09-16 14:38:38.291783459 +0000 UTC m=+1466.895408491"
Sep 16 14:38:47.166854 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:47.166818    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"
Sep 16 14:38:47.167363 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:47.166866    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"
Sep 16 14:38:47.179838 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:47.179814    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"
Sep 16 14:38:47.325665 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:38:47.325635    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"
Sep 16 14:40:00.499662 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:00.499579    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-924916-hqz9q"]
Sep 16 14:40:00.503153 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:00.503136    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-924916-hqz9q"
Sep 16 14:40:00.506337 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:00.506316    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:40:00.506479 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:00.506405    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4jf\""
Sep 16 14:40:00.514096 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:00.514067    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-924916-hqz9q"]
Sep 16 14:40:00.565916 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:00.565873    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29a06fa6-6d4d-4040-b815-bfc781ae8cdc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-924916-hqz9q\" (UID: \"29a06fa6-6d4d-4040-b815-bfc781ae8cdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-924916-hqz9q"
Sep 16 14:40:00.666719 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:00.666672    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29a06fa6-6d4d-4040-b815-bfc781ae8cdc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-924916-hqz9q\" (UID: \"29a06fa6-6d4d-4040-b815-bfc781ae8cdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-924916-hqz9q"
Sep 16 14:40:00.667292 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:00.667270    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29a06fa6-6d4d-4040-b815-bfc781ae8cdc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-924916-hqz9q\" (UID: \"29a06fa6-6d4d-4040-b815-bfc781ae8cdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-924916-hqz9q"
Sep 16 14:40:00.815889 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:00.815802    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-924916-hqz9q"
Sep 16 14:40:01.091284 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:01.091260    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-924916-hqz9q"]
Sep 16 14:40:01.092596 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:40:01.092573    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a06fa6_6d4d_4040_b815_bfc781ae8cdc.slice/crio-43d74885341b8dd70ab886e452792d3346a2349ef43b73dd490361bbe8a2925f WatchSource:0}: Error finding container 43d74885341b8dd70ab886e452792d3346a2349ef43b73dd490361bbe8a2925f: Status 404 returned error can't find the container with id 43d74885341b8dd70ab886e452792d3346a2349ef43b73dd490361bbe8a2925f
Sep 16 14:40:01.727130 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:01.727045    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-924916-hqz9q" event={"ID":"29a06fa6-6d4d-4040-b815-bfc781ae8cdc","Type":"ContainerStarted","Data":"39e858dc5f29b964dc80feeaf30e795159b22501d463c96a6f08ed9f059fceb0"}
Sep 16 14:40:01.727130 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:01.727113    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-924916-hqz9q" event={"ID":"29a06fa6-6d4d-4040-b815-bfc781ae8cdc","Type":"ContainerStarted","Data":"43d74885341b8dd70ab886e452792d3346a2349ef43b73dd490361bbe8a2925f"}
Sep 16 14:40:01.750014 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:01.749966    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-924916-hqz9q" podStartSLOduration=1.749949261 podStartE2EDuration="1.749949261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:40:01.747454314 +0000 UTC m=+1550.351079346" watchObservedRunningTime="2026-09-16 14:40:01.749949261 +0000 UTC m=+1550.353574330"
Sep 16 14:40:02.733760 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:02.733720    2587 generic.go:356] "Generic (PLEG): container finished" podID="29a06fa6-6d4d-4040-b815-bfc781ae8cdc" containerID="39e858dc5f29b964dc80feeaf30e795159b22501d463c96a6f08ed9f059fceb0" exitCode=0
Sep 16 14:40:02.734153 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:02.733804    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-924916-hqz9q" event={"ID":"29a06fa6-6d4d-4040-b815-bfc781ae8cdc","Type":"ContainerDied","Data":"39e858dc5f29b964dc80feeaf30e795159b22501d463c96a6f08ed9f059fceb0"}
Sep 16 14:40:03.879343 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:03.879320    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-924916-hqz9q"
Sep 16 14:40:03.994378 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:03.994285    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29a06fa6-6d4d-4040-b815-bfc781ae8cdc-maas-api-service-ca\") pod \"29a06fa6-6d4d-4040-b815-bfc781ae8cdc\" (UID: \"29a06fa6-6d4d-4040-b815-bfc781ae8cdc\") "
Sep 16 14:40:03.994664 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:03.994638    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a06fa6-6d4d-4040-b815-bfc781ae8cdc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "29a06fa6-6d4d-4040-b815-bfc781ae8cdc" (UID: "29a06fa6-6d4d-4040-b815-bfc781ae8cdc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:40:04.095886 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:04.095855    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29a06fa6-6d4d-4040-b815-bfc781ae8cdc-maas-api-service-ca\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:40:04.745885 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:04.745861    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-924916-hqz9q"
Sep 16 14:40:04.745885 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:04.745865    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-924916-hqz9q" event={"ID":"29a06fa6-6d4d-4040-b815-bfc781ae8cdc","Type":"ContainerDied","Data":"43d74885341b8dd70ab886e452792d3346a2349ef43b73dd490361bbe8a2925f"}
Sep 16 14:40:04.746157 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:04.745909    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43d74885341b8dd70ab886e452792d3346a2349ef43b73dd490361bbe8a2925f"
Sep 16 14:40:05.346829 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:05.346790    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"]
Sep 16 14:40:05.370488 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:05.370457    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"]
Sep 16 14:40:20.366857 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.366794    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d" podUID="74140765-fb3c-4647-9dcc-4640ccf91709" containerName="main" containerID="cri-o://ee7dc773495ffcc0a63c4a4032be4b2fcdf6a441402766f2a29eac77a565d69e" gracePeriod=45
Sep 16 14:40:20.389549 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.389510    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w" podUID="27f90242-bf5f-46fc-8acc-28dfc45ce02f" containerName="main" containerID="cri-o://c93db5b6d58217252c274f05e4b54d460bc9c40e088bc7c49babff9fadc26f1a" gracePeriod=45
Sep 16 14:40:20.670531 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.670506    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"
Sep 16 14:40:20.673180 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.673159    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"
Sep 16 14:40:20.758502 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.758470    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74140765-fb3c-4647-9dcc-4640ccf91709-home\") pod \"74140765-fb3c-4647-9dcc-4640ccf91709\" (UID: \"74140765-fb3c-4647-9dcc-4640ccf91709\") "
Sep 16 14:40:20.758689 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.758510    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74140765-fb3c-4647-9dcc-4640ccf91709-model-cache\") pod \"74140765-fb3c-4647-9dcc-4640ccf91709\" (UID: \"74140765-fb3c-4647-9dcc-4640ccf91709\") "
Sep 16 14:40:20.758689 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.758535    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74140765-fb3c-4647-9dcc-4640ccf91709-tls-certs\") pod \"74140765-fb3c-4647-9dcc-4640ccf91709\" (UID: \"74140765-fb3c-4647-9dcc-4640ccf91709\") "
Sep 16 14:40:20.758689 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.758571    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f90242-bf5f-46fc-8acc-28dfc45ce02f-dshm\") pod \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\" (UID: \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\") "
Sep 16 14:40:20.758865 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.758732    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f90242-bf5f-46fc-8acc-28dfc45ce02f-tls-certs\") pod \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\" (UID: \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\") "
Sep 16 14:40:20.758865 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.758758    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74140765-fb3c-4647-9dcc-4640ccf91709-home" (OuterVolumeSpecName: "home") pod "74140765-fb3c-4647-9dcc-4640ccf91709" (UID: "74140765-fb3c-4647-9dcc-4640ccf91709"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:40:20.758865 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.758776    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74140765-fb3c-4647-9dcc-4640ccf91709-model-cache" (OuterVolumeSpecName: "model-cache") pod "74140765-fb3c-4647-9dcc-4640ccf91709" (UID: "74140765-fb3c-4647-9dcc-4640ccf91709"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:40:20.758865 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.758795    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f90242-bf5f-46fc-8acc-28dfc45ce02f-tmp-dir\") pod \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\" (UID: \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\") "
Sep 16 14:40:20.758865 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.758831    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f90242-bf5f-46fc-8acc-28dfc45ce02f-home\") pod \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\" (UID: \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\") "
Sep 16 14:40:20.759092 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.758926    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgtbb\" (UniqueName: \"kubernetes.io/projected/27f90242-bf5f-46fc-8acc-28dfc45ce02f-kube-api-access-xgtbb\") pod \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\" (UID: \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\") "
Sep 16 14:40:20.759092 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.758965    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6pbh\" (UniqueName: \"kubernetes.io/projected/74140765-fb3c-4647-9dcc-4640ccf91709-kube-api-access-c6pbh\") pod \"74140765-fb3c-4647-9dcc-4640ccf91709\" (UID: \"74140765-fb3c-4647-9dcc-4640ccf91709\") "
Sep 16 14:40:20.759092 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.758994    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74140765-fb3c-4647-9dcc-4640ccf91709-tmp-dir\") pod \"74140765-fb3c-4647-9dcc-4640ccf91709\" (UID: \"74140765-fb3c-4647-9dcc-4640ccf91709\") "
Sep 16 14:40:20.759092 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.759022    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f90242-bf5f-46fc-8acc-28dfc45ce02f-model-cache\") pod \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\" (UID: \"27f90242-bf5f-46fc-8acc-28dfc45ce02f\") "
Sep 16 14:40:20.759092 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.759068    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74140765-fb3c-4647-9dcc-4640ccf91709-dshm\") pod \"74140765-fb3c-4647-9dcc-4640ccf91709\" (UID: \"74140765-fb3c-4647-9dcc-4640ccf91709\") "
Sep 16 14:40:20.759092 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.759036    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f90242-bf5f-46fc-8acc-28dfc45ce02f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "27f90242-bf5f-46fc-8acc-28dfc45ce02f" (UID: "27f90242-bf5f-46fc-8acc-28dfc45ce02f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:40:20.759447 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.759102    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f90242-bf5f-46fc-8acc-28dfc45ce02f-home" (OuterVolumeSpecName: "home") pod "27f90242-bf5f-46fc-8acc-28dfc45ce02f" (UID: "27f90242-bf5f-46fc-8acc-28dfc45ce02f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:40:20.759447 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.759271    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74140765-fb3c-4647-9dcc-4640ccf91709-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "74140765-fb3c-4647-9dcc-4640ccf91709" (UID: "74140765-fb3c-4647-9dcc-4640ccf91709"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:40:20.759629 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.759505    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f90242-bf5f-46fc-8acc-28dfc45ce02f-model-cache" (OuterVolumeSpecName: "model-cache") pod "27f90242-bf5f-46fc-8acc-28dfc45ce02f" (UID: "27f90242-bf5f-46fc-8acc-28dfc45ce02f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:40:20.759793 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.759656    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74140765-fb3c-4647-9dcc-4640ccf91709-tmp-dir\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:40:20.759793 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.759677    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f90242-bf5f-46fc-8acc-28dfc45ce02f-model-cache\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:40:20.759793 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.759689    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74140765-fb3c-4647-9dcc-4640ccf91709-home\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:40:20.759793 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.759701    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74140765-fb3c-4647-9dcc-4640ccf91709-model-cache\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:40:20.759793 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.759715    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f90242-bf5f-46fc-8acc-28dfc45ce02f-tmp-dir\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:40:20.759793 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.759726    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f90242-bf5f-46fc-8acc-28dfc45ce02f-home\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:40:20.761230 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.761187    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27f90242-bf5f-46fc-8acc-28dfc45ce02f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27f90242-bf5f-46fc-8acc-28dfc45ce02f" (UID: "27f90242-bf5f-46fc-8acc-28dfc45ce02f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:40:20.761353 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.761310    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f90242-bf5f-46fc-8acc-28dfc45ce02f-kube-api-access-xgtbb" (OuterVolumeSpecName: "kube-api-access-xgtbb") pod "27f90242-bf5f-46fc-8acc-28dfc45ce02f" (UID: "27f90242-bf5f-46fc-8acc-28dfc45ce02f"). InnerVolumeSpecName "kube-api-access-xgtbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:40:20.761512 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.761456    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74140765-fb3c-4647-9dcc-4640ccf91709-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "74140765-fb3c-4647-9dcc-4640ccf91709" (UID: "74140765-fb3c-4647-9dcc-4640ccf91709"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:40:20.761841 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.761821    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74140765-fb3c-4647-9dcc-4640ccf91709-dshm" (OuterVolumeSpecName: "dshm") pod "74140765-fb3c-4647-9dcc-4640ccf91709" (UID: "74140765-fb3c-4647-9dcc-4640ccf91709"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:40:20.761896 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.761821    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f90242-bf5f-46fc-8acc-28dfc45ce02f-dshm" (OuterVolumeSpecName: "dshm") pod "27f90242-bf5f-46fc-8acc-28dfc45ce02f" (UID: "27f90242-bf5f-46fc-8acc-28dfc45ce02f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:40:20.761934 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.761922    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74140765-fb3c-4647-9dcc-4640ccf91709-kube-api-access-c6pbh" (OuterVolumeSpecName: "kube-api-access-c6pbh") pod "74140765-fb3c-4647-9dcc-4640ccf91709" (UID: "74140765-fb3c-4647-9dcc-4640ccf91709"). InnerVolumeSpecName "kube-api-access-c6pbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:40:20.840794 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.840760    2587 generic.go:356] "Generic (PLEG): container finished" podID="27f90242-bf5f-46fc-8acc-28dfc45ce02f" containerID="c93db5b6d58217252c274f05e4b54d460bc9c40e088bc7c49babff9fadc26f1a" exitCode=0
Sep 16 14:40:20.840972 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.840803    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"
Sep 16 14:40:20.840972 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.840850    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w" event={"ID":"27f90242-bf5f-46fc-8acc-28dfc45ce02f","Type":"ContainerDied","Data":"c93db5b6d58217252c274f05e4b54d460bc9c40e088bc7c49babff9fadc26f1a"}
Sep 16 14:40:20.840972 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.840906    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w" event={"ID":"27f90242-bf5f-46fc-8acc-28dfc45ce02f","Type":"ContainerDied","Data":"5b7b0eac725eb4efef7964d30ca7cc32b04b362136f643489b1a55d861f8fb4f"}
Sep 16 14:40:20.840972 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.840931    2587 scope.go:117] "RemoveContainer" containerID="c93db5b6d58217252c274f05e4b54d460bc9c40e088bc7c49babff9fadc26f1a"
Sep 16 14:40:20.842369 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.842343    2587 generic.go:356] "Generic (PLEG): container finished" podID="74140765-fb3c-4647-9dcc-4640ccf91709" containerID="ee7dc773495ffcc0a63c4a4032be4b2fcdf6a441402766f2a29eac77a565d69e" exitCode=0
Sep 16 14:40:20.842516 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.842382    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"
Sep 16 14:40:20.842516 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.842437    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d" event={"ID":"74140765-fb3c-4647-9dcc-4640ccf91709","Type":"ContainerDied","Data":"ee7dc773495ffcc0a63c4a4032be4b2fcdf6a441402766f2a29eac77a565d69e"}
Sep 16 14:40:20.842516 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.842474    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d" event={"ID":"74140765-fb3c-4647-9dcc-4640ccf91709","Type":"ContainerDied","Data":"b8f4784e7bfd52e76ca125502f63e1e74b95b2bf91f1139acc788e410ecf6191"}
Sep 16 14:40:20.851194 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.851172    2587 scope.go:117] "RemoveContainer" containerID="c93db5b6d58217252c274f05e4b54d460bc9c40e088bc7c49babff9fadc26f1a"
Sep 16 14:40:20.851538 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:40:20.851521    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c93db5b6d58217252c274f05e4b54d460bc9c40e088bc7c49babff9fadc26f1a\": container with ID starting with c93db5b6d58217252c274f05e4b54d460bc9c40e088bc7c49babff9fadc26f1a not found: ID does not exist" containerID="c93db5b6d58217252c274f05e4b54d460bc9c40e088bc7c49babff9fadc26f1a"
Sep 16 14:40:20.851628 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.851545    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c93db5b6d58217252c274f05e4b54d460bc9c40e088bc7c49babff9fadc26f1a"} err="failed to get container status \"c93db5b6d58217252c274f05e4b54d460bc9c40e088bc7c49babff9fadc26f1a\": rpc error: code = NotFound desc = could not find container \"c93db5b6d58217252c274f05e4b54d460bc9c40e088bc7c49babff9fadc26f1a\": container with ID starting with c93db5b6d58217252c274f05e4b54d460bc9c40e088bc7c49babff9fadc26f1a not found: ID does not exist"
Sep 16 14:40:20.851628 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.851562    2587 scope.go:117] "RemoveContainer" containerID="ee7dc773495ffcc0a63c4a4032be4b2fcdf6a441402766f2a29eac77a565d69e"
Sep 16 14:40:20.859768 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.859750    2587 scope.go:117] "RemoveContainer" containerID="ee7dc773495ffcc0a63c4a4032be4b2fcdf6a441402766f2a29eac77a565d69e"
Sep 16 14:40:20.860064 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:40:20.860046    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee7dc773495ffcc0a63c4a4032be4b2fcdf6a441402766f2a29eac77a565d69e\": container with ID starting with ee7dc773495ffcc0a63c4a4032be4b2fcdf6a441402766f2a29eac77a565d69e not found: ID does not exist" containerID="ee7dc773495ffcc0a63c4a4032be4b2fcdf6a441402766f2a29eac77a565d69e"
Sep 16 14:40:20.860121 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.860071    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee7dc773495ffcc0a63c4a4032be4b2fcdf6a441402766f2a29eac77a565d69e"} err="failed to get container status \"ee7dc773495ffcc0a63c4a4032be4b2fcdf6a441402766f2a29eac77a565d69e\": rpc error: code = NotFound desc = could not find container \"ee7dc773495ffcc0a63c4a4032be4b2fcdf6a441402766f2a29eac77a565d69e\": container with ID starting with ee7dc773495ffcc0a63c4a4032be4b2fcdf6a441402766f2a29eac77a565d69e not found: ID does not exist"
Sep 16 14:40:20.860206 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.860190    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgtbb\" (UniqueName: \"kubernetes.io/projected/27f90242-bf5f-46fc-8acc-28dfc45ce02f-kube-api-access-xgtbb\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:40:20.860240 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.860211    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6pbh\" (UniqueName: \"kubernetes.io/projected/74140765-fb3c-4647-9dcc-4640ccf91709-kube-api-access-c6pbh\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:40:20.860240 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.860220    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74140765-fb3c-4647-9dcc-4640ccf91709-dshm\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:40:20.860240 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.860230    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74140765-fb3c-4647-9dcc-4640ccf91709-tls-certs\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:40:20.860240 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.860240    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f90242-bf5f-46fc-8acc-28dfc45ce02f-dshm\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:40:20.860365 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.860247    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f90242-bf5f-46fc-8acc-28dfc45ce02f-tls-certs\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:40:20.868988 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.868965    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"]
Sep 16 14:40:20.873459 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.873438    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-924916/distinct-w0-924916-kserve-85788cf7b8-7n68d"]
Sep 16 14:40:20.885225 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.885203    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"]
Sep 16 14:40:20.889761 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:20.889740    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-924916/premium-w0-924916-kserve-58645fcd45-wc58w"]
Sep 16 14:40:21.256607 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:21.256572    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-924916-29826150-gdtmq"]
Sep 16 14:40:21.258567 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:21.258547    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-924916-29826150-gdtmq"]
Sep 16 14:40:21.287957 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:21.287927    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf"]
Sep 16 14:40:21.288207 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:21.288179    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf" podUID="a004d1e5-6df4-4e40-996d-849c2e3e043f" containerName="payload-pre-processing" containerID="cri-o://4ca63b58ddcfb266bc3e593e0b82e01df937bb9d7bee35830d674b2aa2a36268" gracePeriod=30
Sep 16 14:40:21.481534 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:21.481501    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-924916-hqz9q"]
Sep 16 14:40:21.484623 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:21.484597    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-924916-hqz9q"]
Sep 16 14:40:21.533594 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:21.533569    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf"
Sep 16 14:40:21.668740 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:21.668701    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xmqn\" (UniqueName: \"kubernetes.io/projected/a004d1e5-6df4-4e40-996d-849c2e3e043f-kube-api-access-5xmqn\") pod \"a004d1e5-6df4-4e40-996d-849c2e3e043f\" (UID: \"a004d1e5-6df4-4e40-996d-849c2e3e043f\") "
Sep 16 14:40:21.668740 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:21.668744    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a004d1e5-6df4-4e40-996d-849c2e3e043f-plugins-config-volume\") pod \"a004d1e5-6df4-4e40-996d-849c2e3e043f\" (UID: \"a004d1e5-6df4-4e40-996d-849c2e3e043f\") "
Sep 16 14:40:21.669175 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:21.669148    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a004d1e5-6df4-4e40-996d-849c2e3e043f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a004d1e5-6df4-4e40-996d-849c2e3e043f" (UID: "a004d1e5-6df4-4e40-996d-849c2e3e043f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:40:21.671171 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:21.671150    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a004d1e5-6df4-4e40-996d-849c2e3e043f-kube-api-access-5xmqn" (OuterVolumeSpecName: "kube-api-access-5xmqn") pod "a004d1e5-6df4-4e40-996d-849c2e3e043f" (UID: "a004d1e5-6df4-4e40-996d-849c2e3e043f"). InnerVolumeSpecName "kube-api-access-5xmqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:40:21.770239 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:21.770154    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xmqn\" (UniqueName: \"kubernetes.io/projected/a004d1e5-6df4-4e40-996d-849c2e3e043f-kube-api-access-5xmqn\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:40:21.770239 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:21.770187    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a004d1e5-6df4-4e40-996d-849c2e3e043f-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:40:21.852306 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:21.852274    2587 generic.go:356] "Generic (PLEG): container finished" podID="a004d1e5-6df4-4e40-996d-849c2e3e043f" containerID="4ca63b58ddcfb266bc3e593e0b82e01df937bb9d7bee35830d674b2aa2a36268" exitCode=0
Sep 16 14:40:21.852516 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:21.852326    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf"
Sep 16 14:40:21.852516 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:21.852345    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf" event={"ID":"a004d1e5-6df4-4e40-996d-849c2e3e043f","Type":"ContainerDied","Data":"4ca63b58ddcfb266bc3e593e0b82e01df937bb9d7bee35830d674b2aa2a36268"}
Sep 16 14:40:21.852516 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:21.852385    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf" event={"ID":"a004d1e5-6df4-4e40-996d-849c2e3e043f","Type":"ContainerDied","Data":"225beb2b3c35c20ec2aef9e565a4a264125ba9ae7aa9211f7ce8ea58f7d55d9d"}
Sep 16 14:40:21.852516 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:21.852434    2587 scope.go:117] "RemoveContainer" containerID="4ca63b58ddcfb266bc3e593e0b82e01df937bb9d7bee35830d674b2aa2a36268"
Sep 16 14:40:21.862090 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:21.862071    2587 scope.go:117] "RemoveContainer" containerID="4ca63b58ddcfb266bc3e593e0b82e01df937bb9d7bee35830d674b2aa2a36268"
Sep 16 14:40:21.862381 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:40:21.862363    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ca63b58ddcfb266bc3e593e0b82e01df937bb9d7bee35830d674b2aa2a36268\": container with ID starting with 4ca63b58ddcfb266bc3e593e0b82e01df937bb9d7bee35830d674b2aa2a36268 not found: ID does not exist" containerID="4ca63b58ddcfb266bc3e593e0b82e01df937bb9d7bee35830d674b2aa2a36268"
Sep 16 14:40:21.862472 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:21.862405    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ca63b58ddcfb266bc3e593e0b82e01df937bb9d7bee35830d674b2aa2a36268"} err="failed to get container status \"4ca63b58ddcfb266bc3e593e0b82e01df937bb9d7bee35830d674b2aa2a36268\": rpc error: code = NotFound desc = could not find container \"4ca63b58ddcfb266bc3e593e0b82e01df937bb9d7bee35830d674b2aa2a36268\": container with ID starting with 4ca63b58ddcfb266bc3e593e0b82e01df937bb9d7bee35830d674b2aa2a36268 not found: ID does not exist"
Sep 16 14:40:21.877572 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:21.877547    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf"]
Sep 16 14:40:21.886484 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:21.886458    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-924916-575fb4b9c4-hr4wf"]
Sep 16 14:40:21.904625 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:21.904601    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27f90242-bf5f-46fc-8acc-28dfc45ce02f" path="/var/lib/kubelet/pods/27f90242-bf5f-46fc-8acc-28dfc45ce02f/volumes"
Sep 16 14:40:21.904947 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:21.904935    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29a06fa6-6d4d-4040-b815-bfc781ae8cdc" path="/var/lib/kubelet/pods/29a06fa6-6d4d-4040-b815-bfc781ae8cdc/volumes"
Sep 16 14:40:21.905274 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:21.905261    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74140765-fb3c-4647-9dcc-4640ccf91709" path="/var/lib/kubelet/pods/74140765-fb3c-4647-9dcc-4640ccf91709/volumes"
Sep 16 14:40:21.905532 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:21.905520    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a96c4bb-f7b3-4f0a-8870-a5a3cc813c34" path="/var/lib/kubelet/pods/9a96c4bb-f7b3-4f0a-8870-a5a3cc813c34/volumes"
Sep 16 14:40:21.906085 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:21.906071    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a004d1e5-6df4-4e40-996d-849c2e3e043f" path="/var/lib/kubelet/pods/a004d1e5-6df4-4e40-996d-849c2e3e043f/volumes"
Sep 16 14:40:42.253814 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:42.253781    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb"]
Sep 16 14:40:42.254456 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:42.254436    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a004d1e5-6df4-4e40-996d-849c2e3e043f" containerName="payload-pre-processing"
Sep 16 14:40:42.254551 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:42.254459    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="a004d1e5-6df4-4e40-996d-849c2e3e043f" containerName="payload-pre-processing"
Sep 16 14:40:42.254551 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:42.254471    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29a06fa6-6d4d-4040-b815-bfc781ae8cdc" containerName="revoke-keys"
Sep 16 14:40:42.254551 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:42.254479    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="29a06fa6-6d4d-4040-b815-bfc781ae8cdc" containerName="revoke-keys"
Sep 16 14:40:42.254551 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:42.254486    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74140765-fb3c-4647-9dcc-4640ccf91709" containerName="main"
Sep 16 14:40:42.254551 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:42.254494    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="74140765-fb3c-4647-9dcc-4640ccf91709" containerName="main"
Sep 16 14:40:42.254551 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:42.254517    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27f90242-bf5f-46fc-8acc-28dfc45ce02f" containerName="main"
Sep 16 14:40:42.254551 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:42.254524    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="27f90242-bf5f-46fc-8acc-28dfc45ce02f" containerName="main"
Sep 16 14:40:42.254885 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:42.254642    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="27f90242-bf5f-46fc-8acc-28dfc45ce02f" containerName="main"
Sep 16 14:40:42.254885 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:42.254654    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="74140765-fb3c-4647-9dcc-4640ccf91709" containerName="main"
Sep 16 14:40:42.254885 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:42.254666    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="a004d1e5-6df4-4e40-996d-849c2e3e043f" containerName="payload-pre-processing"
Sep 16 14:40:42.254885 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:42.254678    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="29a06fa6-6d4d-4040-b815-bfc781ae8cdc" containerName="revoke-keys"
Sep 16 14:40:42.257985 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:42.257956    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb"
Sep 16 14:40:42.262623 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:42.262599    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f891c2-dockercfg-ndp85\""
Sep 16 14:40:42.262823 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:42.262620    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f891c2\""
Sep 16 14:40:42.270445 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:42.270416    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb"]
Sep 16 14:40:42.357546 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:42.357514    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c10e808-dfba-4583-9e13-512680a60a84-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb\" (UID: \"6c10e808-dfba-4583-9e13-512680a60a84\") " pod="openshift-ingress/payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb"
Sep 16 14:40:42.357722 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:42.357592    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrhjb\" (UniqueName: \"kubernetes.io/projected/6c10e808-dfba-4583-9e13-512680a60a84-kube-api-access-vrhjb\") pod \"payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb\" (UID: \"6c10e808-dfba-4583-9e13-512680a60a84\") " pod="openshift-ingress/payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb"
Sep 16 14:40:42.458210 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:42.458177    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrhjb\" (UniqueName: \"kubernetes.io/projected/6c10e808-dfba-4583-9e13-512680a60a84-kube-api-access-vrhjb\") pod \"payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb\" (UID: \"6c10e808-dfba-4583-9e13-512680a60a84\") " pod="openshift-ingress/payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb"
Sep 16 14:40:42.458425 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:42.458255    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c10e808-dfba-4583-9e13-512680a60a84-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb\" (UID: \"6c10e808-dfba-4583-9e13-512680a60a84\") " pod="openshift-ingress/payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb"
Sep 16 14:40:42.458967 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:42.458946    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c10e808-dfba-4583-9e13-512680a60a84-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb\" (UID: \"6c10e808-dfba-4583-9e13-512680a60a84\") " pod="openshift-ingress/payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb"
Sep 16 14:40:42.466475 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:42.466455    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrhjb\" (UniqueName: \"kubernetes.io/projected/6c10e808-dfba-4583-9e13-512680a60a84-kube-api-access-vrhjb\") pod \"payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb\" (UID: \"6c10e808-dfba-4583-9e13-512680a60a84\") " pod="openshift-ingress/payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb"
Sep 16 14:40:42.571055 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:42.570973    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb"
Sep 16 14:40:42.865616 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:42.865421    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb"]
Sep 16 14:40:42.868323 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:40:42.868294    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c10e808_dfba_4583_9e13_512680a60a84.slice/crio-2969e9e791b54184342828521bac820a728fa972f8eb351d897fbd97c71e9e9e WatchSource:0}: Error finding container 2969e9e791b54184342828521bac820a728fa972f8eb351d897fbd97c71e9e9e: Status 404 returned error can't find the container with id 2969e9e791b54184342828521bac820a728fa972f8eb351d897fbd97c71e9e9e
Sep 16 14:40:42.972307 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:42.972272    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb" event={"ID":"6c10e808-dfba-4583-9e13-512680a60a84","Type":"ContainerStarted","Data":"e25dda6ebc86d58b386ccd5db8f315ed4742e14fd00ad22f5e4547f44f213479"}
Sep 16 14:40:42.972508 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:42.972327    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb" event={"ID":"6c10e808-dfba-4583-9e13-512680a60a84","Type":"ContainerStarted","Data":"2969e9e791b54184342828521bac820a728fa972f8eb351d897fbd97c71e9e9e"}
Sep 16 14:40:42.972508 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:42.972354    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb"
Sep 16 14:40:43.006176 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:43.006118    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb" podStartSLOduration=1.00610009 podStartE2EDuration="1.00610009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:40:42.993899292 +0000 UTC m=+1591.597524324" watchObservedRunningTime="2026-09-16 14:40:43.00610009 +0000 UTC m=+1591.609725122"
Sep 16 14:40:53.979549 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:40:53.979520    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb"
Sep 16 14:41:09.652108 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:09.652028    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71d83f-zpjl4"]
Sep 16 14:41:09.655853 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:09.655834    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71d83f-zpjl4"
Sep 16 14:41:09.660910 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:09.660885    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4jf\""
Sep 16 14:41:09.661123 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:09.661109    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:41:09.731271 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:09.731231    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71d83f-zpjl4"]
Sep 16 14:41:09.820491 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:09.820451    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf0c664-9c4f-40b0-b65c-b1956275d87d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-71d83f-zpjl4\" (UID: \"4bf0c664-9c4f-40b0-b65c-b1956275d87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71d83f-zpjl4"
Sep 16 14:41:09.921817 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:09.921728    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf0c664-9c4f-40b0-b65c-b1956275d87d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-71d83f-zpjl4\" (UID: \"4bf0c664-9c4f-40b0-b65c-b1956275d87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71d83f-zpjl4"
Sep 16 14:41:09.922434 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:09.922377    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf0c664-9c4f-40b0-b65c-b1956275d87d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-71d83f-zpjl4\" (UID: \"4bf0c664-9c4f-40b0-b65c-b1956275d87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71d83f-zpjl4"
Sep 16 14:41:09.966216 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:09.966178    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71d83f-zpjl4"
Sep 16 14:41:10.245845 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:10.245815    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71d83f-zpjl4"]
Sep 16 14:41:10.247544 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:41:10.247517    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf0c664_9c4f_40b0_b65c_b1956275d87d.slice/crio-19e0b2f0ea9d160de7c35039af01aaca7dce92c89f8f11e19149fc9a3c234087 WatchSource:0}: Error finding container 19e0b2f0ea9d160de7c35039af01aaca7dce92c89f8f11e19149fc9a3c234087: Status 404 returned error can't find the container with id 19e0b2f0ea9d160de7c35039af01aaca7dce92c89f8f11e19149fc9a3c234087
Sep 16 14:41:11.130537 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:11.130496    2587 generic.go:356] "Generic (PLEG): container finished" podID="4bf0c664-9c4f-40b0-b65c-b1956275d87d" containerID="dc938f882afa65bd67023abc8896b344107ab5a0ae310d467b5bbfeed22dfb78" exitCode=0
Sep 16 14:41:11.130956 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:11.130558    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71d83f-zpjl4" event={"ID":"4bf0c664-9c4f-40b0-b65c-b1956275d87d","Type":"ContainerDied","Data":"dc938f882afa65bd67023abc8896b344107ab5a0ae310d467b5bbfeed22dfb78"}
Sep 16 14:41:11.130956 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:11.130609    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71d83f-zpjl4" event={"ID":"4bf0c664-9c4f-40b0-b65c-b1956275d87d","Type":"ContainerStarted","Data":"19e0b2f0ea9d160de7c35039af01aaca7dce92c89f8f11e19149fc9a3c234087"}
Sep 16 14:41:12.134893 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:12.134863    2587 scope.go:117] "RemoveContainer" containerID="6d62814b12a27422120edc29b1d7c32d69b5434f23b5e98b1321bed3c2583d23"
Sep 16 14:41:12.146778 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:12.146674    2587 scope.go:117] "RemoveContainer" containerID="5b879fca25bff2b978931a8297bea0f44ea89d882f6570fe4cd0a0938842c519"
Sep 16 14:41:12.156851 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:12.156830    2587 scope.go:117] "RemoveContainer" containerID="b54644a0e8f6da41a856bdadc22ceab056fe78aacd344525c726c81b5684e834"
Sep 16 14:41:12.288833 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:12.288808    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71d83f-zpjl4"
Sep 16 14:41:12.448415 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:12.448363    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf0c664-9c4f-40b0-b65c-b1956275d87d-maas-api-service-ca\") pod \"4bf0c664-9c4f-40b0-b65c-b1956275d87d\" (UID: \"4bf0c664-9c4f-40b0-b65c-b1956275d87d\") "
Sep 16 14:41:12.448724 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:12.448702    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bf0c664-9c4f-40b0-b65c-b1956275d87d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4bf0c664-9c4f-40b0-b65c-b1956275d87d" (UID: "4bf0c664-9c4f-40b0-b65c-b1956275d87d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:41:12.550065 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:12.550019    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf0c664-9c4f-40b0-b65c-b1956275d87d-maas-api-service-ca\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:41:13.142299 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:13.142272    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71d83f-zpjl4"
Sep 16 14:41:13.142299 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:13.142281    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71d83f-zpjl4" event={"ID":"4bf0c664-9c4f-40b0-b65c-b1956275d87d","Type":"ContainerDied","Data":"19e0b2f0ea9d160de7c35039af01aaca7dce92c89f8f11e19149fc9a3c234087"}
Sep 16 14:41:13.142878 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:13.142339    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19e0b2f0ea9d160de7c35039af01aaca7dce92c89f8f11e19149fc9a3c234087"
Sep 16 14:41:25.837609 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:25.837573    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71d83f-zpjl4"]
Sep 16 14:41:25.841777 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:25.841746    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71d83f-zpjl4"]
Sep 16 14:41:25.902622 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:25.902593    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bf0c664-9c4f-40b0-b65c-b1956275d87d" path="/var/lib/kubelet/pods/4bf0c664-9c4f-40b0-b65c-b1956275d87d/volumes"
Sep 16 14:41:33.761526 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:33.761494    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f891c2-j8qgm"]
Sep 16 14:41:33.761958 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:33.761941    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bf0c664-9c4f-40b0-b65c-b1956275d87d" containerName="revoke-keys"
Sep 16 14:41:33.761958 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:33.761958    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bf0c664-9c4f-40b0-b65c-b1956275d87d" containerName="revoke-keys"
Sep 16 14:41:33.762059 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:33.762046    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bf0c664-9c4f-40b0-b65c-b1956275d87d" containerName="revoke-keys"
Sep 16 14:41:33.766256 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:33.766237    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f891c2-j8qgm"
Sep 16 14:41:33.769092 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:33.769076    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:41:33.769175 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:33.769077    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4jf\""
Sep 16 14:41:33.780319 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:33.780296    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f891c2-j8qgm"]
Sep 16 14:41:33.844506 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:33.844472    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76c71ab-4b2e-4258-be11-a54bd2380d33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f891c2-j8qgm\" (UID: \"b76c71ab-4b2e-4258-be11-a54bd2380d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f891c2-j8qgm"
Sep 16 14:41:33.945949 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:33.945915    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76c71ab-4b2e-4258-be11-a54bd2380d33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f891c2-j8qgm\" (UID: \"b76c71ab-4b2e-4258-be11-a54bd2380d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f891c2-j8qgm"
Sep 16 14:41:33.946724 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:33.946704    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76c71ab-4b2e-4258-be11-a54bd2380d33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f891c2-j8qgm\" (UID: \"b76c71ab-4b2e-4258-be11-a54bd2380d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f891c2-j8qgm"
Sep 16 14:41:34.076227 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:34.076133    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f891c2-j8qgm"
Sep 16 14:41:34.375424 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:34.375380    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f891c2-j8qgm"]
Sep 16 14:41:34.376991 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:41:34.376966    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76c71ab_4b2e_4258_be11_a54bd2380d33.slice/crio-af650cdf53a72338d5af98b16537816aba127b0224c5306fc1e596b3c9bba002 WatchSource:0}: Error finding container af650cdf53a72338d5af98b16537816aba127b0224c5306fc1e596b3c9bba002: Status 404 returned error can't find the container with id af650cdf53a72338d5af98b16537816aba127b0224c5306fc1e596b3c9bba002
Sep 16 14:41:35.260898 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:35.260863    2587 generic.go:356] "Generic (PLEG): container finished" podID="b76c71ab-4b2e-4258-be11-a54bd2380d33" containerID="5341bdf9548bc057c2908e962c8ac1edcbbc4cf9bb1e81c0072c7bbb6d844427" exitCode=0
Sep 16 14:41:35.261264 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:35.260943    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f891c2-j8qgm" event={"ID":"b76c71ab-4b2e-4258-be11-a54bd2380d33","Type":"ContainerDied","Data":"5341bdf9548bc057c2908e962c8ac1edcbbc4cf9bb1e81c0072c7bbb6d844427"}
Sep 16 14:41:35.261264 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:35.260994    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f891c2-j8qgm" event={"ID":"b76c71ab-4b2e-4258-be11-a54bd2380d33","Type":"ContainerStarted","Data":"af650cdf53a72338d5af98b16537816aba127b0224c5306fc1e596b3c9bba002"}
Sep 16 14:41:36.412839 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:36.412814    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f891c2-j8qgm"
Sep 16 14:41:36.470746 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:36.470716    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76c71ab-4b2e-4258-be11-a54bd2380d33-maas-api-service-ca\") pod \"b76c71ab-4b2e-4258-be11-a54bd2380d33\" (UID: \"b76c71ab-4b2e-4258-be11-a54bd2380d33\") "
Sep 16 14:41:36.471031 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:36.471010    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b76c71ab-4b2e-4258-be11-a54bd2380d33-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b76c71ab-4b2e-4258-be11-a54bd2380d33" (UID: "b76c71ab-4b2e-4258-be11-a54bd2380d33"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:41:36.572542 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:36.572450    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76c71ab-4b2e-4258-be11-a54bd2380d33-maas-api-service-ca\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:41:37.273056 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:37.273020    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f891c2-j8qgm"
Sep 16 14:41:37.273260 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:37.273053    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f891c2-j8qgm" event={"ID":"b76c71ab-4b2e-4258-be11-a54bd2380d33","Type":"ContainerDied","Data":"af650cdf53a72338d5af98b16537816aba127b0224c5306fc1e596b3c9bba002"}
Sep 16 14:41:37.273260 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:37.273102    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af650cdf53a72338d5af98b16537816aba127b0224c5306fc1e596b3c9bba002"
Sep 16 14:41:37.723679 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:37.723650    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5d484d-t4tdf"]
Sep 16 14:41:37.724066 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:37.724053    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b76c71ab-4b2e-4258-be11-a54bd2380d33" containerName="revoke-keys"
Sep 16 14:41:37.724107 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:37.724068    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="b76c71ab-4b2e-4258-be11-a54bd2380d33" containerName="revoke-keys"
Sep 16 14:41:37.724144 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:37.724139    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="b76c71ab-4b2e-4258-be11-a54bd2380d33" containerName="revoke-keys"
Sep 16 14:41:37.727230 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:37.727212    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5d484d-t4tdf"
Sep 16 14:41:37.730111 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:37.730084    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:41:37.730243 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:37.730150    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4jf\""
Sep 16 14:41:37.735457 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:37.735429    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5d484d-t4tdf"]
Sep 16 14:41:37.785635 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:37.785602    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca3af92-f3f3-4047-9401-953ca88efd9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-5d484d-t4tdf\" (UID: \"dca3af92-f3f3-4047-9401-953ca88efd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5d484d-t4tdf"
Sep 16 14:41:37.886414 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:37.886344    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca3af92-f3f3-4047-9401-953ca88efd9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-5d484d-t4tdf\" (UID: \"dca3af92-f3f3-4047-9401-953ca88efd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5d484d-t4tdf"
Sep 16 14:41:37.886975 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:37.886956    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca3af92-f3f3-4047-9401-953ca88efd9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-5d484d-t4tdf\" (UID: \"dca3af92-f3f3-4047-9401-953ca88efd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5d484d-t4tdf"
Sep 16 14:41:38.040279 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:38.040191    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5d484d-t4tdf"
Sep 16 14:41:38.319972 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:38.319942    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5d484d-t4tdf"]
Sep 16 14:41:38.322238 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:41:38.322210    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca3af92_f3f3_4047_9401_953ca88efd9e.slice/crio-ae3098325d50e981540f3bac29c32ba2b4ca44de916792636cd768b3bb9aea30 WatchSource:0}: Error finding container ae3098325d50e981540f3bac29c32ba2b4ca44de916792636cd768b3bb9aea30: Status 404 returned error can't find the container with id ae3098325d50e981540f3bac29c32ba2b4ca44de916792636cd768b3bb9aea30
Sep 16 14:41:39.285807 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:39.285719    2587 generic.go:356] "Generic (PLEG): container finished" podID="dca3af92-f3f3-4047-9401-953ca88efd9e" containerID="3d596ea3f82a3619a445e873d55743bfa1c684a9003daad445333d282ca2ccb4" exitCode=0
Sep 16 14:41:39.286192 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:39.285809    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5d484d-t4tdf" event={"ID":"dca3af92-f3f3-4047-9401-953ca88efd9e","Type":"ContainerDied","Data":"3d596ea3f82a3619a445e873d55743bfa1c684a9003daad445333d282ca2ccb4"}
Sep 16 14:41:39.286192 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:39.285869    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5d484d-t4tdf" event={"ID":"dca3af92-f3f3-4047-9401-953ca88efd9e","Type":"ContainerStarted","Data":"ae3098325d50e981540f3bac29c32ba2b4ca44de916792636cd768b3bb9aea30"}
Sep 16 14:41:40.435454 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:40.435424    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5d484d-t4tdf"
Sep 16 14:41:40.512492 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:40.512458    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca3af92-f3f3-4047-9401-953ca88efd9e-maas-api-service-ca\") pod \"dca3af92-f3f3-4047-9401-953ca88efd9e\" (UID: \"dca3af92-f3f3-4047-9401-953ca88efd9e\") "
Sep 16 14:41:40.512822 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:40.512800    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dca3af92-f3f3-4047-9401-953ca88efd9e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dca3af92-f3f3-4047-9401-953ca88efd9e" (UID: "dca3af92-f3f3-4047-9401-953ca88efd9e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:41:40.613745 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:40.613657    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca3af92-f3f3-4047-9401-953ca88efd9e-maas-api-service-ca\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:41:41.296942 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:41.296912    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5d484d-t4tdf"
Sep 16 14:41:41.297115 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:41.296963    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5d484d-t4tdf" event={"ID":"dca3af92-f3f3-4047-9401-953ca88efd9e","Type":"ContainerDied","Data":"ae3098325d50e981540f3bac29c32ba2b4ca44de916792636cd768b3bb9aea30"}
Sep 16 14:41:41.297115 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:41.297011    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae3098325d50e981540f3bac29c32ba2b4ca44de916792636cd768b3bb9aea30"
Sep 16 14:41:42.698898 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:42.698856    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"]
Sep 16 14:41:42.732656 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:42.732627    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"]
Sep 16 14:41:52.672026 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:52.671982    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb"]
Sep 16 14:41:52.672499 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:52.672267    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb" podUID="6c10e808-dfba-4583-9e13-512680a60a84" containerName="payload-processing" containerID="cri-o://e25dda6ebc86d58b386ccd5db8f315ed4742e14fd00ad22f5e4547f44f213479" gracePeriod=30
Sep 16 14:41:52.866518 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:52.866482    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f891c2-j8qgm"]
Sep 16 14:41:52.869280 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:52.869255    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f891c2-j8qgm"]
Sep 16 14:41:52.915359 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:52.915332    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb"
Sep 16 14:41:53.029419 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:53.029303    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c10e808-dfba-4583-9e13-512680a60a84-plugins-config-volume\") pod \"6c10e808-dfba-4583-9e13-512680a60a84\" (UID: \"6c10e808-dfba-4583-9e13-512680a60a84\") "
Sep 16 14:41:53.029419 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:53.029361    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrhjb\" (UniqueName: \"kubernetes.io/projected/6c10e808-dfba-4583-9e13-512680a60a84-kube-api-access-vrhjb\") pod \"6c10e808-dfba-4583-9e13-512680a60a84\" (UID: \"6c10e808-dfba-4583-9e13-512680a60a84\") "
Sep 16 14:41:53.029800 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:53.029774    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c10e808-dfba-4583-9e13-512680a60a84-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6c10e808-dfba-4583-9e13-512680a60a84" (UID: "6c10e808-dfba-4583-9e13-512680a60a84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:41:53.031852 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:53.031826    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c10e808-dfba-4583-9e13-512680a60a84-kube-api-access-vrhjb" (OuterVolumeSpecName: "kube-api-access-vrhjb") pod "6c10e808-dfba-4583-9e13-512680a60a84" (UID: "6c10e808-dfba-4583-9e13-512680a60a84"). InnerVolumeSpecName "kube-api-access-vrhjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:41:53.131346 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:53.131315    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c10e808-dfba-4583-9e13-512680a60a84-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:41:53.131346 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:53.131345    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrhjb\" (UniqueName: \"kubernetes.io/projected/6c10e808-dfba-4583-9e13-512680a60a84-kube-api-access-vrhjb\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:41:53.371299 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:53.371207    2587 generic.go:356] "Generic (PLEG): container finished" podID="6c10e808-dfba-4583-9e13-512680a60a84" containerID="e25dda6ebc86d58b386ccd5db8f315ed4742e14fd00ad22f5e4547f44f213479" exitCode=0
Sep 16 14:41:53.371299 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:53.371255    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb"
Sep 16 14:41:53.371530 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:53.371291    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb" event={"ID":"6c10e808-dfba-4583-9e13-512680a60a84","Type":"ContainerDied","Data":"e25dda6ebc86d58b386ccd5db8f315ed4742e14fd00ad22f5e4547f44f213479"}
Sep 16 14:41:53.371530 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:53.371349    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb" event={"ID":"6c10e808-dfba-4583-9e13-512680a60a84","Type":"ContainerDied","Data":"2969e9e791b54184342828521bac820a728fa972f8eb351d897fbd97c71e9e9e"}
Sep 16 14:41:53.371530 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:53.371372    2587 scope.go:117] "RemoveContainer" containerID="e25dda6ebc86d58b386ccd5db8f315ed4742e14fd00ad22f5e4547f44f213479"
Sep 16 14:41:53.381268 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:53.381250    2587 scope.go:117] "RemoveContainer" containerID="e25dda6ebc86d58b386ccd5db8f315ed4742e14fd00ad22f5e4547f44f213479"
Sep 16 14:41:53.381552 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:41:53.381533    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e25dda6ebc86d58b386ccd5db8f315ed4742e14fd00ad22f5e4547f44f213479\": container with ID starting with e25dda6ebc86d58b386ccd5db8f315ed4742e14fd00ad22f5e4547f44f213479 not found: ID does not exist" containerID="e25dda6ebc86d58b386ccd5db8f315ed4742e14fd00ad22f5e4547f44f213479"
Sep 16 14:41:53.381607 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:53.381561    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25dda6ebc86d58b386ccd5db8f315ed4742e14fd00ad22f5e4547f44f213479"} err="failed to get container status \"e25dda6ebc86d58b386ccd5db8f315ed4742e14fd00ad22f5e4547f44f213479\": rpc error: code = NotFound desc = could not find container \"e25dda6ebc86d58b386ccd5db8f315ed4742e14fd00ad22f5e4547f44f213479\": container with ID starting with e25dda6ebc86d58b386ccd5db8f315ed4742e14fd00ad22f5e4547f44f213479 not found: ID does not exist"
Sep 16 14:41:53.394304 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:53.394282    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb"]
Sep 16 14:41:53.398350 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:53.398329    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f891c2-7695b99dd7-dlspb"]
Sep 16 14:41:53.904356 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:53.904324    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c10e808-dfba-4583-9e13-512680a60a84" path="/var/lib/kubelet/pods/6c10e808-dfba-4583-9e13-512680a60a84/volumes"
Sep 16 14:41:53.904725 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:53.904689    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b76c71ab-4b2e-4258-be11-a54bd2380d33" path="/var/lib/kubelet/pods/b76c71ab-4b2e-4258-be11-a54bd2380d33/volumes"
Sep 16 14:41:56.239554 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:56.239522    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-5d484d-29826150-6xckn"]
Sep 16 14:41:56.241689 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:56.241669    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-5d484d-29826150-6xckn"]
Sep 16 14:41:56.267384 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:56.267360    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6"]
Sep 16 14:41:56.267641 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:56.267616    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6" podUID="28382e17-12f4-4f98-bb9b-53c579f10dfb" containerName="payload-processing" containerID="cri-o://37171d84b8b7f641cbeec9cfabf5e08862d6726bba64258d99cdaa132d3198f8" gracePeriod=30
Sep 16 14:41:56.271820 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:56.271797    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz"]
Sep 16 14:41:56.272145 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:56.272117    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz" podUID="1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c" containerName="payload-pre-processing" containerID="cri-o://05e6df28836a7473dc818bd449ee4a2925529fd0bb733f23ab6d3cd0dd3cf35d" gracePeriod=30
Sep 16 14:41:56.391281 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:56.391253    2587 generic.go:356] "Generic (PLEG): container finished" podID="28382e17-12f4-4f98-bb9b-53c579f10dfb" containerID="37171d84b8b7f641cbeec9cfabf5e08862d6726bba64258d99cdaa132d3198f8" exitCode=0
Sep 16 14:41:56.391430 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:56.391326    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6" event={"ID":"28382e17-12f4-4f98-bb9b-53c579f10dfb","Type":"ContainerDied","Data":"37171d84b8b7f641cbeec9cfabf5e08862d6726bba64258d99cdaa132d3198f8"}
Sep 16 14:41:56.393546 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:56.393525    2587 generic.go:356] "Generic (PLEG): container finished" podID="1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c" containerID="05e6df28836a7473dc818bd449ee4a2925529fd0bb733f23ab6d3cd0dd3cf35d" exitCode=0
Sep 16 14:41:56.393675 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:56.393606    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz" event={"ID":"1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c","Type":"ContainerDied","Data":"05e6df28836a7473dc818bd449ee4a2925529fd0bb733f23ab6d3cd0dd3cf35d"}
Sep 16 14:41:56.454724 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:56.454653    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5d484d-t4tdf"]
Sep 16 14:41:56.459042 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:56.459017    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-5d484d-t4tdf"]
Sep 16 14:41:56.564751 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:56.564707    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6"
Sep 16 14:41:56.568408 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:56.568357    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz"
Sep 16 14:41:56.667020 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:56.666985    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs759\" (UniqueName: \"kubernetes.io/projected/1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c-kube-api-access-hs759\") pod \"1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c\" (UID: \"1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c\") "
Sep 16 14:41:56.667185 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:56.667110    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c-plugins-config-volume\") pod \"1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c\" (UID: \"1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c\") "
Sep 16 14:41:56.667185 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:56.667159    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swgrv\" (UniqueName: \"kubernetes.io/projected/28382e17-12f4-4f98-bb9b-53c579f10dfb-kube-api-access-swgrv\") pod \"28382e17-12f4-4f98-bb9b-53c579f10dfb\" (UID: \"28382e17-12f4-4f98-bb9b-53c579f10dfb\") "
Sep 16 14:41:56.667324 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:56.667302    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28382e17-12f4-4f98-bb9b-53c579f10dfb-plugins-config-volume\") pod \"28382e17-12f4-4f98-bb9b-53c579f10dfb\" (UID: \"28382e17-12f4-4f98-bb9b-53c579f10dfb\") "
Sep 16 14:41:56.667657 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:56.667525    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c" (UID: "1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:41:56.667960 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:56.667697    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28382e17-12f4-4f98-bb9b-53c579f10dfb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "28382e17-12f4-4f98-bb9b-53c579f10dfb" (UID: "28382e17-12f4-4f98-bb9b-53c579f10dfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:41:56.669362 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:56.669337    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28382e17-12f4-4f98-bb9b-53c579f10dfb-kube-api-access-swgrv" (OuterVolumeSpecName: "kube-api-access-swgrv") pod "28382e17-12f4-4f98-bb9b-53c579f10dfb" (UID: "28382e17-12f4-4f98-bb9b-53c579f10dfb"). InnerVolumeSpecName "kube-api-access-swgrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:41:56.669748 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:56.669721    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c-kube-api-access-hs759" (OuterVolumeSpecName: "kube-api-access-hs759") pod "1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c" (UID: "1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c"). InnerVolumeSpecName "kube-api-access-hs759". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:41:56.768961 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:56.768929    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:41:56.768961 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:56.768960    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swgrv\" (UniqueName: \"kubernetes.io/projected/28382e17-12f4-4f98-bb9b-53c579f10dfb-kube-api-access-swgrv\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:41:56.769151 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:56.768972    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28382e17-12f4-4f98-bb9b-53c579f10dfb-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:41:56.769151 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:56.768984    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs759\" (UniqueName: \"kubernetes.io/projected/1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c-kube-api-access-hs759\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:41:57.400578 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:57.400548    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6"
Sep 16 14:41:57.400578 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:57.400567    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6" event={"ID":"28382e17-12f4-4f98-bb9b-53c579f10dfb","Type":"ContainerDied","Data":"88f8dfff57f994bcecb95487dfcd4cd9854a74bffcffba650741ed24b10114a1"}
Sep 16 14:41:57.401084 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:57.400649    2587 scope.go:117] "RemoveContainer" containerID="37171d84b8b7f641cbeec9cfabf5e08862d6726bba64258d99cdaa132d3198f8"
Sep 16 14:41:57.402297 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:57.402219    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz"
Sep 16 14:41:57.402297 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:57.402212    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz" event={"ID":"1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c","Type":"ContainerDied","Data":"21d393ecbf48406c1062b178058b12c6f9a247aa4c22a406574123a6e5f64354"}
Sep 16 14:41:57.411598 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:57.411577    2587 scope.go:117] "RemoveContainer" containerID="05e6df28836a7473dc818bd449ee4a2925529fd0bb733f23ab6d3cd0dd3cf35d"
Sep 16 14:41:57.435182 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:57.435160    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz"]
Sep 16 14:41:57.443145 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:57.443119    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-5d484d-b7f9f6786-f7dsz"]
Sep 16 14:41:57.454025 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:57.454003    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6"]
Sep 16 14:41:57.462242 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:57.462219    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-5d484d-d76945b69-7sxj6"]
Sep 16 14:41:57.719187 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:57.719086    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965" podUID="7373abcb-c40e-4f4b-bd7a-102f243bf1db" containerName="main" containerID="cri-o://681af4ad85f93beb5127e57af68128c138996d58d69c4b507ec0bb3c224141ff" gracePeriod=45
Sep 16 14:41:57.751601 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:57.751564    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4" podUID="6edfb182-3de4-4361-ab1a-2c70fb152fc7" containerName="main" containerID="cri-o://4394d64ef49dd238ae6e7f6444af77c9b794a6bd9788b05be9f383b66474500e" gracePeriod=45
Sep 16 14:41:57.906944 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:57.905382    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c" path="/var/lib/kubelet/pods/1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c/volumes"
Sep 16 14:41:57.906944 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:57.905825    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28382e17-12f4-4f98-bb9b-53c579f10dfb" path="/var/lib/kubelet/pods/28382e17-12f4-4f98-bb9b-53c579f10dfb/volumes"
Sep 16 14:41:57.906944 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:57.906092    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4759566e-b40f-421b-b998-e5bbc1738a1c" path="/var/lib/kubelet/pods/4759566e-b40f-421b-b998-e5bbc1738a1c/volumes"
Sep 16 14:41:57.906944 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:57.906382    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dca3af92-f3f3-4047-9401-953ca88efd9e" path="/var/lib/kubelet/pods/dca3af92-f3f3-4047-9401-953ca88efd9e/volumes"
Sep 16 14:41:58.013408 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.013362    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"
Sep 16 14:41:58.025902 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.025879    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"
Sep 16 14:41:58.185050 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.185016    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7373abcb-c40e-4f4b-bd7a-102f243bf1db-home\") pod \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\" (UID: \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\") "
Sep 16 14:41:58.185202 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.185083    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rq8tq\" (UniqueName: \"kubernetes.io/projected/6edfb182-3de4-4361-ab1a-2c70fb152fc7-kube-api-access-rq8tq\") pod \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\" (UID: \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\") "
Sep 16 14:41:58.185202 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.185127    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6edfb182-3de4-4361-ab1a-2c70fb152fc7-home\") pod \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\" (UID: \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\") "
Sep 16 14:41:58.185202 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.185162    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzkl6\" (UniqueName: \"kubernetes.io/projected/7373abcb-c40e-4f4b-bd7a-102f243bf1db-kube-api-access-pzkl6\") pod \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\" (UID: \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\") "
Sep 16 14:41:58.185362 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.185212    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6edfb182-3de4-4361-ab1a-2c70fb152fc7-tls-certs\") pod \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\" (UID: \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\") "
Sep 16 14:41:58.185362 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.185238    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6edfb182-3de4-4361-ab1a-2c70fb152fc7-dshm\") pod \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\" (UID: \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\") "
Sep 16 14:41:58.185362 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.185349    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7373abcb-c40e-4f4b-bd7a-102f243bf1db-home" (OuterVolumeSpecName: "home") pod "7373abcb-c40e-4f4b-bd7a-102f243bf1db" (UID: "7373abcb-c40e-4f4b-bd7a-102f243bf1db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:41:58.185570 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.185360    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edfb182-3de4-4361-ab1a-2c70fb152fc7-tmp-dir\") pod \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\" (UID: \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\") "
Sep 16 14:41:58.185570 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.185416    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6edfb182-3de4-4361-ab1a-2c70fb152fc7-home" (OuterVolumeSpecName: "home") pod "6edfb182-3de4-4361-ab1a-2c70fb152fc7" (UID: "6edfb182-3de4-4361-ab1a-2c70fb152fc7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:41:58.185570 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.185434    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7373abcb-c40e-4f4b-bd7a-102f243bf1db-tls-certs\") pod \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\" (UID: \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\") "
Sep 16 14:41:58.185570 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.185473    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7373abcb-c40e-4f4b-bd7a-102f243bf1db-dshm\") pod \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\" (UID: \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\") "
Sep 16 14:41:58.185570 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.185510    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7373abcb-c40e-4f4b-bd7a-102f243bf1db-model-cache\") pod \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\" (UID: \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\") "
Sep 16 14:41:58.185570 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.185541    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6edfb182-3de4-4361-ab1a-2c70fb152fc7-model-cache\") pod \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\" (UID: \"6edfb182-3de4-4361-ab1a-2c70fb152fc7\") "
Sep 16 14:41:58.185877 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.185592    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7373abcb-c40e-4f4b-bd7a-102f243bf1db-tmp-dir\") pod \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\" (UID: \"7373abcb-c40e-4f4b-bd7a-102f243bf1db\") "
Sep 16 14:41:58.185877 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.185662    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6edfb182-3de4-4361-ab1a-2c70fb152fc7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6edfb182-3de4-4361-ab1a-2c70fb152fc7" (UID: "6edfb182-3de4-4361-ab1a-2c70fb152fc7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:41:58.186151 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.185955    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7373abcb-c40e-4f4b-bd7a-102f243bf1db-home\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:41:58.186151 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.186076    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6edfb182-3de4-4361-ab1a-2c70fb152fc7-home\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:41:58.186151 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.186136    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edfb182-3de4-4361-ab1a-2c70fb152fc7-tmp-dir\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:41:58.186622 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.186328    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7373abcb-c40e-4f4b-bd7a-102f243bf1db-model-cache" (OuterVolumeSpecName: "model-cache") pod "7373abcb-c40e-4f4b-bd7a-102f243bf1db" (UID: "7373abcb-c40e-4f4b-bd7a-102f243bf1db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:41:58.186622 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.186552    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6edfb182-3de4-4361-ab1a-2c70fb152fc7-model-cache" (OuterVolumeSpecName: "model-cache") pod "6edfb182-3de4-4361-ab1a-2c70fb152fc7" (UID: "6edfb182-3de4-4361-ab1a-2c70fb152fc7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:41:58.186835 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.186752    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7373abcb-c40e-4f4b-bd7a-102f243bf1db-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7373abcb-c40e-4f4b-bd7a-102f243bf1db" (UID: "7373abcb-c40e-4f4b-bd7a-102f243bf1db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:41:58.187886 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.187861    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6edfb182-3de4-4361-ab1a-2c70fb152fc7-dshm" (OuterVolumeSpecName: "dshm") pod "6edfb182-3de4-4361-ab1a-2c70fb152fc7" (UID: "6edfb182-3de4-4361-ab1a-2c70fb152fc7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:41:58.187988 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.187923    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6edfb182-3de4-4361-ab1a-2c70fb152fc7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6edfb182-3de4-4361-ab1a-2c70fb152fc7" (UID: "6edfb182-3de4-4361-ab1a-2c70fb152fc7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:41:58.188096 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.188077    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6edfb182-3de4-4361-ab1a-2c70fb152fc7-kube-api-access-rq8tq" (OuterVolumeSpecName: "kube-api-access-rq8tq") pod "6edfb182-3de4-4361-ab1a-2c70fb152fc7" (UID: "6edfb182-3de4-4361-ab1a-2c70fb152fc7"). InnerVolumeSpecName "kube-api-access-rq8tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:41:58.188157 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.188106    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7373abcb-c40e-4f4b-bd7a-102f243bf1db-kube-api-access-pzkl6" (OuterVolumeSpecName: "kube-api-access-pzkl6") pod "7373abcb-c40e-4f4b-bd7a-102f243bf1db" (UID: "7373abcb-c40e-4f4b-bd7a-102f243bf1db"). InnerVolumeSpecName "kube-api-access-pzkl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:41:58.188449 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.188434    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7373abcb-c40e-4f4b-bd7a-102f243bf1db-dshm" (OuterVolumeSpecName: "dshm") pod "7373abcb-c40e-4f4b-bd7a-102f243bf1db" (UID: "7373abcb-c40e-4f4b-bd7a-102f243bf1db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:41:58.188709 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.188696    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7373abcb-c40e-4f4b-bd7a-102f243bf1db-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7373abcb-c40e-4f4b-bd7a-102f243bf1db" (UID: "7373abcb-c40e-4f4b-bd7a-102f243bf1db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:41:58.287461 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.287370    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzkl6\" (UniqueName: \"kubernetes.io/projected/7373abcb-c40e-4f4b-bd7a-102f243bf1db-kube-api-access-pzkl6\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:41:58.287461 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.287414    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6edfb182-3de4-4361-ab1a-2c70fb152fc7-tls-certs\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:41:58.287461 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.287424    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6edfb182-3de4-4361-ab1a-2c70fb152fc7-dshm\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:41:58.287461 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.287431    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7373abcb-c40e-4f4b-bd7a-102f243bf1db-tls-certs\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:41:58.287461 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.287439    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7373abcb-c40e-4f4b-bd7a-102f243bf1db-dshm\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:41:58.287461 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.287446    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7373abcb-c40e-4f4b-bd7a-102f243bf1db-model-cache\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:41:58.287461 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.287453    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6edfb182-3de4-4361-ab1a-2c70fb152fc7-model-cache\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:41:58.287461 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.287460    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7373abcb-c40e-4f4b-bd7a-102f243bf1db-tmp-dir\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:41:58.287461 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.287469    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq8tq\" (UniqueName: \"kubernetes.io/projected/6edfb182-3de4-4361-ab1a-2c70fb152fc7-kube-api-access-rq8tq\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:41:58.409252 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.409214    2587 generic.go:356] "Generic (PLEG): container finished" podID="6edfb182-3de4-4361-ab1a-2c70fb152fc7" containerID="4394d64ef49dd238ae6e7f6444af77c9b794a6bd9788b05be9f383b66474500e" exitCode=0
Sep 16 14:41:58.409801 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.409258    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"
Sep 16 14:41:58.409801 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.409304    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4" event={"ID":"6edfb182-3de4-4361-ab1a-2c70fb152fc7","Type":"ContainerDied","Data":"4394d64ef49dd238ae6e7f6444af77c9b794a6bd9788b05be9f383b66474500e"}
Sep 16 14:41:58.409801 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.409357    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4" event={"ID":"6edfb182-3de4-4361-ab1a-2c70fb152fc7","Type":"ContainerDied","Data":"098b12b172796a6cf6877304751e12c7bdbd30e6d874923e572c31d25df8d3bc"}
Sep 16 14:41:58.409801 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.409413    2587 scope.go:117] "RemoveContainer" containerID="4394d64ef49dd238ae6e7f6444af77c9b794a6bd9788b05be9f383b66474500e"
Sep 16 14:41:58.412042 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.412012    2587 generic.go:356] "Generic (PLEG): container finished" podID="7373abcb-c40e-4f4b-bd7a-102f243bf1db" containerID="681af4ad85f93beb5127e57af68128c138996d58d69c4b507ec0bb3c224141ff" exitCode=0
Sep 16 14:41:58.412151 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.412061    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"
Sep 16 14:41:58.412151 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.412091    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965" event={"ID":"7373abcb-c40e-4f4b-bd7a-102f243bf1db","Type":"ContainerDied","Data":"681af4ad85f93beb5127e57af68128c138996d58d69c4b507ec0bb3c224141ff"}
Sep 16 14:41:58.412151 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.412128    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965" event={"ID":"7373abcb-c40e-4f4b-bd7a-102f243bf1db","Type":"ContainerDied","Data":"e67435be1fd2bdc21130c47da0906a3f21ac19859ee1450b13786648f4e0bf5c"}
Sep 16 14:41:58.422042 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.422019    2587 scope.go:117] "RemoveContainer" containerID="4394d64ef49dd238ae6e7f6444af77c9b794a6bd9788b05be9f383b66474500e"
Sep 16 14:41:58.422453 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:41:58.422428    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4394d64ef49dd238ae6e7f6444af77c9b794a6bd9788b05be9f383b66474500e\": container with ID starting with 4394d64ef49dd238ae6e7f6444af77c9b794a6bd9788b05be9f383b66474500e not found: ID does not exist" containerID="4394d64ef49dd238ae6e7f6444af77c9b794a6bd9788b05be9f383b66474500e"
Sep 16 14:41:58.422586 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.422462    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4394d64ef49dd238ae6e7f6444af77c9b794a6bd9788b05be9f383b66474500e"} err="failed to get container status \"4394d64ef49dd238ae6e7f6444af77c9b794a6bd9788b05be9f383b66474500e\": rpc error: code = NotFound desc = could not find container \"4394d64ef49dd238ae6e7f6444af77c9b794a6bd9788b05be9f383b66474500e\": container with ID starting with 4394d64ef49dd238ae6e7f6444af77c9b794a6bd9788b05be9f383b66474500e not found: ID does not exist"
Sep 16 14:41:58.422586 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.422485    2587 scope.go:117] "RemoveContainer" containerID="681af4ad85f93beb5127e57af68128c138996d58d69c4b507ec0bb3c224141ff"
Sep 16 14:41:58.431254 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.431229    2587 scope.go:117] "RemoveContainer" containerID="681af4ad85f93beb5127e57af68128c138996d58d69c4b507ec0bb3c224141ff"
Sep 16 14:41:58.431746 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:41:58.431722    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"681af4ad85f93beb5127e57af68128c138996d58d69c4b507ec0bb3c224141ff\": container with ID starting with 681af4ad85f93beb5127e57af68128c138996d58d69c4b507ec0bb3c224141ff not found: ID does not exist" containerID="681af4ad85f93beb5127e57af68128c138996d58d69c4b507ec0bb3c224141ff"
Sep 16 14:41:58.431848 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.431757    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"681af4ad85f93beb5127e57af68128c138996d58d69c4b507ec0bb3c224141ff"} err="failed to get container status \"681af4ad85f93beb5127e57af68128c138996d58d69c4b507ec0bb3c224141ff\": rpc error: code = NotFound desc = could not find container \"681af4ad85f93beb5127e57af68128c138996d58d69c4b507ec0bb3c224141ff\": container with ID starting with 681af4ad85f93beb5127e57af68128c138996d58d69c4b507ec0bb3c224141ff not found: ID does not exist"
Sep 16 14:41:58.433506 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.433485    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"]
Sep 16 14:41:58.437781 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.437753    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-5d484d/premium-w4-5d484d-kserve-765d7d889f-xpsn4"]
Sep 16 14:41:58.450276 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.450235    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"]
Sep 16 14:41:58.455768 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:58.455738    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-5d484d/distinct-2-w4-5d484d-kserve-c6777d797-76965"]
Sep 16 14:41:59.903090 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:59.903056    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6edfb182-3de4-4361-ab1a-2c70fb152fc7" path="/var/lib/kubelet/pods/6edfb182-3de4-4361-ab1a-2c70fb152fc7/volumes"
Sep 16 14:41:59.903490 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:41:59.903481    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7373abcb-c40e-4f4b-bd7a-102f243bf1db" path="/var/lib/kubelet/pods/7373abcb-c40e-4f4b-bd7a-102f243bf1db/volumes"
Sep 16 14:42:00.633995 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:00.633961    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c2787-fbltg"]
Sep 16 14:42:00.634377 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:00.634361    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28382e17-12f4-4f98-bb9b-53c579f10dfb" containerName="payload-processing"
Sep 16 14:42:00.634460 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:00.634379    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="28382e17-12f4-4f98-bb9b-53c579f10dfb" containerName="payload-processing"
Sep 16 14:42:00.634460 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:00.634407    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7373abcb-c40e-4f4b-bd7a-102f243bf1db" containerName="main"
Sep 16 14:42:00.634460 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:00.634413    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="7373abcb-c40e-4f4b-bd7a-102f243bf1db" containerName="main"
Sep 16 14:42:00.634460 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:00.634421    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c" containerName="payload-pre-processing"
Sep 16 14:42:00.634460 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:00.634429    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c" containerName="payload-pre-processing"
Sep 16 14:42:00.634460 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:00.634445    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6edfb182-3de4-4361-ab1a-2c70fb152fc7" containerName="main"
Sep 16 14:42:00.634460 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:00.634450    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="6edfb182-3de4-4361-ab1a-2c70fb152fc7" containerName="main"
Sep 16 14:42:00.634460 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:00.634460    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c10e808-dfba-4583-9e13-512680a60a84" containerName="payload-processing"
Sep 16 14:42:00.634732 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:00.634465    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c10e808-dfba-4583-9e13-512680a60a84" containerName="payload-processing"
Sep 16 14:42:00.634732 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:00.634474    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dca3af92-f3f3-4047-9401-953ca88efd9e" containerName="revoke-keys"
Sep 16 14:42:00.634732 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:00.634479    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="dca3af92-f3f3-4047-9401-953ca88efd9e" containerName="revoke-keys"
Sep 16 14:42:00.634732 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:00.634532    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c10e808-dfba-4583-9e13-512680a60a84" containerName="payload-processing"
Sep 16 14:42:00.634732 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:00.634539    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="6edfb182-3de4-4361-ab1a-2c70fb152fc7" containerName="main"
Sep 16 14:42:00.634732 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:00.634544    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="7373abcb-c40e-4f4b-bd7a-102f243bf1db" containerName="main"
Sep 16 14:42:00.634732 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:00.634551    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="dca3af92-f3f3-4047-9401-953ca88efd9e" containerName="revoke-keys"
Sep 16 14:42:00.634732 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:00.634558    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ad5aeae-ed4e-45be-8e9b-0cae2f37e77c" containerName="payload-pre-processing"
Sep 16 14:42:00.634732 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:00.634563    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="28382e17-12f4-4f98-bb9b-53c579f10dfb" containerName="payload-processing"
Sep 16 14:42:00.639233 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:00.639213    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c2787-fbltg"
Sep 16 14:42:00.641870 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:00.641849    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:42:00.641951 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:00.641849    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4jf\""
Sep 16 14:42:00.652522 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:00.652500    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c2787-fbltg"]
Sep 16 14:42:00.812724 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:00.812667    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7065f22b-b0be-42d4-9520-0a9bafe167cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4c2787-fbltg\" (UID: \"7065f22b-b0be-42d4-9520-0a9bafe167cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c2787-fbltg"
Sep 16 14:42:00.913588 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:00.913538    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7065f22b-b0be-42d4-9520-0a9bafe167cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4c2787-fbltg\" (UID: \"7065f22b-b0be-42d4-9520-0a9bafe167cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c2787-fbltg"
Sep 16 14:42:00.914158 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:00.914138    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7065f22b-b0be-42d4-9520-0a9bafe167cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4c2787-fbltg\" (UID: \"7065f22b-b0be-42d4-9520-0a9bafe167cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c2787-fbltg"
Sep 16 14:42:00.949596 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:00.949562    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c2787-fbltg"
Sep 16 14:42:01.225258 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:01.225232    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c2787-fbltg"]
Sep 16 14:42:01.227377 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:42:01.227351    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7065f22b_b0be_42d4_9520_0a9bafe167cc.slice/crio-2a58c780156ed5478693e8cc4d49c15b24986c5faa9da242173fc4a0276e91c1 WatchSource:0}: Error finding container 2a58c780156ed5478693e8cc4d49c15b24986c5faa9da242173fc4a0276e91c1: Status 404 returned error can't find the container with id 2a58c780156ed5478693e8cc4d49c15b24986c5faa9da242173fc4a0276e91c1
Sep 16 14:42:01.433112 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:01.433062    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c2787-fbltg" event={"ID":"7065f22b-b0be-42d4-9520-0a9bafe167cc","Type":"ContainerStarted","Data":"2a58c780156ed5478693e8cc4d49c15b24986c5faa9da242173fc4a0276e91c1"}
Sep 16 14:42:02.440972 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:02.440932    2587 generic.go:356] "Generic (PLEG): container finished" podID="7065f22b-b0be-42d4-9520-0a9bafe167cc" containerID="175ba647863178fcbb0fe7e07930c8d3cbb848b1821df99f294c8cceb081bdf2" exitCode=0
Sep 16 14:42:02.441376 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:02.441011    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c2787-fbltg" event={"ID":"7065f22b-b0be-42d4-9520-0a9bafe167cc","Type":"ContainerDied","Data":"175ba647863178fcbb0fe7e07930c8d3cbb848b1821df99f294c8cceb081bdf2"}
Sep 16 14:42:03.597269 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:03.597245    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c2787-fbltg"
Sep 16 14:42:03.740288 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:03.740206    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7065f22b-b0be-42d4-9520-0a9bafe167cc-maas-api-service-ca\") pod \"7065f22b-b0be-42d4-9520-0a9bafe167cc\" (UID: \"7065f22b-b0be-42d4-9520-0a9bafe167cc\") "
Sep 16 14:42:03.740585 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:03.740565    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7065f22b-b0be-42d4-9520-0a9bafe167cc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7065f22b-b0be-42d4-9520-0a9bafe167cc" (UID: "7065f22b-b0be-42d4-9520-0a9bafe167cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:42:03.841368 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:03.841327    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7065f22b-b0be-42d4-9520-0a9bafe167cc-maas-api-service-ca\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:42:04.453304 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:04.453264    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c2787-fbltg" event={"ID":"7065f22b-b0be-42d4-9520-0a9bafe167cc","Type":"ContainerDied","Data":"2a58c780156ed5478693e8cc4d49c15b24986c5faa9da242173fc4a0276e91c1"}
Sep 16 14:42:04.453304 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:04.453288    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c2787-fbltg"
Sep 16 14:42:04.453549 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:04.453324    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a58c780156ed5478693e8cc4d49c15b24986c5faa9da242173fc4a0276e91c1"
Sep 16 14:42:11.198009 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.197975    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf"]
Sep 16 14:42:11.198438 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.198194    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf" podUID="4b4836a3-de08-4aa2-a2df-c519ae6f8c0e" containerName="payload-processing" containerID="cri-o://8266609f6ab1ad9c3d020e588c389c91c296506a1fa1a2449d95c485082f171d" gracePeriod=30
Sep 16 14:42:11.203118 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.203075    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk"]
Sep 16 14:42:11.203552 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.203523    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk" podUID="183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a" containerName="payload-pre-processing" containerID="cri-o://33605731eaf1ab09a5cb25e238bd033c7c3a50880e8e899d7f1146c4dc988181" gracePeriod=30
Sep 16 14:42:11.361372 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.361333    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c2787-fbltg"]
Sep 16 14:42:11.365264 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.365218    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4c2787-fbltg"]
Sep 16 14:42:11.474882 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.474848    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf"
Sep 16 14:42:11.491613 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.491588    2587 generic.go:356] "Generic (PLEG): container finished" podID="183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a" containerID="33605731eaf1ab09a5cb25e238bd033c7c3a50880e8e899d7f1146c4dc988181" exitCode=0
Sep 16 14:42:11.491737 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.491673    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk" event={"ID":"183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a","Type":"ContainerDied","Data":"33605731eaf1ab09a5cb25e238bd033c7c3a50880e8e899d7f1146c4dc988181"}
Sep 16 14:42:11.491815 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.491736    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk" event={"ID":"183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a","Type":"ContainerDied","Data":"931821feafbefe498a4b79092bb6d3ab17bf5b938ec4be8736accf31a567e04b"}
Sep 16 14:42:11.491815 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.491761    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="931821feafbefe498a4b79092bb6d3ab17bf5b938ec4be8736accf31a567e04b"
Sep 16 14:42:11.493136 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.493112    2587 generic.go:356] "Generic (PLEG): container finished" podID="4b4836a3-de08-4aa2-a2df-c519ae6f8c0e" containerID="8266609f6ab1ad9c3d020e588c389c91c296506a1fa1a2449d95c485082f171d" exitCode=0
Sep 16 14:42:11.493215 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.493153    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf"
Sep 16 14:42:11.493251 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.493154    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf" event={"ID":"4b4836a3-de08-4aa2-a2df-c519ae6f8c0e","Type":"ContainerDied","Data":"8266609f6ab1ad9c3d020e588c389c91c296506a1fa1a2449d95c485082f171d"}
Sep 16 14:42:11.493297 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.493284    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf" event={"ID":"4b4836a3-de08-4aa2-a2df-c519ae6f8c0e","Type":"ContainerDied","Data":"0d039b05956fbd51162f3b5a4ad6c2e595955f3d1665eabb3d6600736409b318"}
Sep 16 14:42:11.493333 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.493313    2587 scope.go:117] "RemoveContainer" containerID="8266609f6ab1ad9c3d020e588c389c91c296506a1fa1a2449d95c485082f171d"
Sep 16 14:42:11.504200 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.504175    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk"
Sep 16 14:42:11.504972 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.504952    2587 scope.go:117] "RemoveContainer" containerID="8266609f6ab1ad9c3d020e588c389c91c296506a1fa1a2449d95c485082f171d"
Sep 16 14:42:11.505277 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:42:11.505258    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8266609f6ab1ad9c3d020e588c389c91c296506a1fa1a2449d95c485082f171d\": container with ID starting with 8266609f6ab1ad9c3d020e588c389c91c296506a1fa1a2449d95c485082f171d not found: ID does not exist" containerID="8266609f6ab1ad9c3d020e588c389c91c296506a1fa1a2449d95c485082f171d"
Sep 16 14:42:11.505341 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.505288    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8266609f6ab1ad9c3d020e588c389c91c296506a1fa1a2449d95c485082f171d"} err="failed to get container status \"8266609f6ab1ad9c3d020e588c389c91c296506a1fa1a2449d95c485082f171d\": rpc error: code = NotFound desc = could not find container \"8266609f6ab1ad9c3d020e588c389c91c296506a1fa1a2449d95c485082f171d\": container with ID starting with 8266609f6ab1ad9c3d020e588c389c91c296506a1fa1a2449d95c485082f171d not found: ID does not exist"
Sep 16 14:42:11.513202 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.513177    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7gbf\" (UniqueName: \"kubernetes.io/projected/4b4836a3-de08-4aa2-a2df-c519ae6f8c0e-kube-api-access-n7gbf\") pod \"4b4836a3-de08-4aa2-a2df-c519ae6f8c0e\" (UID: \"4b4836a3-de08-4aa2-a2df-c519ae6f8c0e\") "
Sep 16 14:42:11.513304 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.513285    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b4836a3-de08-4aa2-a2df-c519ae6f8c0e-plugins-config-volume\") pod \"4b4836a3-de08-4aa2-a2df-c519ae6f8c0e\" (UID: \"4b4836a3-de08-4aa2-a2df-c519ae6f8c0e\") "
Sep 16 14:42:11.513656 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.513634    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b4836a3-de08-4aa2-a2df-c519ae6f8c0e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b4836a3-de08-4aa2-a2df-c519ae6f8c0e" (UID: "4b4836a3-de08-4aa2-a2df-c519ae6f8c0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:42:11.515304 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.515284    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4836a3-de08-4aa2-a2df-c519ae6f8c0e-kube-api-access-n7gbf" (OuterVolumeSpecName: "kube-api-access-n7gbf") pod "4b4836a3-de08-4aa2-a2df-c519ae6f8c0e" (UID: "4b4836a3-de08-4aa2-a2df-c519ae6f8c0e"). InnerVolumeSpecName "kube-api-access-n7gbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:42:11.614406 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.614363    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a-plugins-config-volume\") pod \"183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a\" (UID: \"183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a\") "
Sep 16 14:42:11.614576 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.614484    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2stc5\" (UniqueName: \"kubernetes.io/projected/183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a-kube-api-access-2stc5\") pod \"183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a\" (UID: \"183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a\") "
Sep 16 14:42:11.614723 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.614701    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a" (UID: "183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:42:11.614762 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.614731    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7gbf\" (UniqueName: \"kubernetes.io/projected/4b4836a3-de08-4aa2-a2df-c519ae6f8c0e-kube-api-access-n7gbf\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:42:11.614762 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.614749    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b4836a3-de08-4aa2-a2df-c519ae6f8c0e-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:42:11.616672 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.616652    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a-kube-api-access-2stc5" (OuterVolumeSpecName: "kube-api-access-2stc5") pod "183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a" (UID: "183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a"). InnerVolumeSpecName "kube-api-access-2stc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:42:11.715361 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.715296    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:42:11.715361 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.715322    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2stc5\" (UniqueName: \"kubernetes.io/projected/183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a-kube-api-access-2stc5\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:42:11.818105 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.818068    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf"]
Sep 16 14:42:11.824981 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.824953    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4c2787-668fbc77c5-4vdsf"]
Sep 16 14:42:11.908919 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.908883    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b4836a3-de08-4aa2-a2df-c519ae6f8c0e" path="/var/lib/kubelet/pods/4b4836a3-de08-4aa2-a2df-c519ae6f8c0e/volumes"
Sep 16 14:42:11.909223 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:11.909209    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7065f22b-b0be-42d4-9520-0a9bafe167cc" path="/var/lib/kubelet/pods/7065f22b-b0be-42d4-9520-0a9bafe167cc/volumes"
Sep 16 14:42:12.199675 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:12.199644    2587 scope.go:117] "RemoveContainer" containerID="33605731eaf1ab09a5cb25e238bd033c7c3a50880e8e899d7f1146c4dc988181"
Sep 16 14:42:12.208472 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:12.208449    2587 scope.go:117] "RemoveContainer" containerID="f09eb55be577e2c916402f6fd7f50505169f332ecfa63e05d660795d408eee2d"
Sep 16 14:42:12.499092 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:12.499014    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk"
Sep 16 14:42:12.530693 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:12.530655    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk"]
Sep 16 14:42:12.534902 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:12.534872    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4c2787-677955c5cd-z4nmk"]
Sep 16 14:42:13.903080 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:13.903039    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a" path="/var/lib/kubelet/pods/183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a/volumes"
Sep 16 14:42:16.805249 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:16.805215    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7608-84c18e"]
Sep 16 14:42:16.805646 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:16.805607    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7065f22b-b0be-42d4-9520-0a9bafe167cc" containerName="revoke-keys"
Sep 16 14:42:16.805646 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:16.805620    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="7065f22b-b0be-42d4-9520-0a9bafe167cc" containerName="revoke-keys"
Sep 16 14:42:16.805646 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:16.805635    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a" containerName="payload-pre-processing"
Sep 16 14:42:16.805646 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:16.805642    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a" containerName="payload-pre-processing"
Sep 16 14:42:16.805778 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:16.805652    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b4836a3-de08-4aa2-a2df-c519ae6f8c0e" containerName="payload-processing"
Sep 16 14:42:16.805778 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:16.805660    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b4836a3-de08-4aa2-a2df-c519ae6f8c0e" containerName="payload-processing"
Sep 16 14:42:16.805778 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:16.805725    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b4836a3-de08-4aa2-a2df-c519ae6f8c0e" containerName="payload-processing"
Sep 16 14:42:16.805778 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:16.805733    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="7065f22b-b0be-42d4-9520-0a9bafe167cc" containerName="revoke-keys"
Sep 16 14:42:16.805778 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:16.805740    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="183d6a8b-2b8c-4b0b-97c7-68a87bbe7a6a" containerName="payload-pre-processing"
Sep 16 14:42:16.810434 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:16.810416    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-84c18e"
Sep 16 14:42:16.813099 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:16.813080    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gn749\""
Sep 16 14:42:16.818793 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:16.818771    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7608-84c18e"]
Sep 16 14:42:16.861485 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:16.861447    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvhtd\" (UniqueName: \"kubernetes.io/projected/9568c301-f45f-4d25-8b47-89ad95867bf1-kube-api-access-zvhtd\") pod \"test-curl-7608-84c18e\" (UID: \"9568c301-f45f-4d25-8b47-89ad95867bf1\") " pod="opendatahub/test-curl-7608-84c18e"
Sep 16 14:42:16.962378 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:16.962329    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvhtd\" (UniqueName: \"kubernetes.io/projected/9568c301-f45f-4d25-8b47-89ad95867bf1-kube-api-access-zvhtd\") pod \"test-curl-7608-84c18e\" (UID: \"9568c301-f45f-4d25-8b47-89ad95867bf1\") " pod="opendatahub/test-curl-7608-84c18e"
Sep 16 14:42:16.971150 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:16.971125    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvhtd\" (UniqueName: \"kubernetes.io/projected/9568c301-f45f-4d25-8b47-89ad95867bf1-kube-api-access-zvhtd\") pod \"test-curl-7608-84c18e\" (UID: \"9568c301-f45f-4d25-8b47-89ad95867bf1\") " pod="opendatahub/test-curl-7608-84c18e"
Sep 16 14:42:17.120649 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:17.120561    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-84c18e"
Sep 16 14:42:17.400891 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:17.400866    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7608-84c18e"]
Sep 16 14:42:17.403521 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:42:17.403495    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9568c301_f45f_4d25_8b47_89ad95867bf1.slice/crio-d5a773490f73aa18bae28dd0e5b3f680f57218edd1c4f67023c7b3477778ba03 WatchSource:0}: Error finding container d5a773490f73aa18bae28dd0e5b3f680f57218edd1c4f67023c7b3477778ba03: Status 404 returned error can't find the container with id d5a773490f73aa18bae28dd0e5b3f680f57218edd1c4f67023c7b3477778ba03
Sep 16 14:42:17.527308 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:17.527275    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7608-84c18e" event={"ID":"9568c301-f45f-4d25-8b47-89ad95867bf1","Type":"ContainerStarted","Data":"a52d7799ac06aac97ff5467e89a64a0f92bf74081cd0b566edda9e149ec8a4ec"}
Sep 16 14:42:17.527515 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:17.527319    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7608-84c18e" event={"ID":"9568c301-f45f-4d25-8b47-89ad95867bf1","Type":"ContainerStarted","Data":"d5a773490f73aa18bae28dd0e5b3f680f57218edd1c4f67023c7b3477778ba03"}
Sep 16 14:42:17.544235 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:17.544183    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7608-84c18e" podStartSLOduration=1.5441684900000001 podStartE2EDuration="1.54416849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:42:17.541709273 +0000 UTC m=+1686.145334305" watchObservedRunningTime="2026-09-16 14:42:17.54416849 +0000 UTC m=+1686.147793522"
Sep 16 14:42:18.159729 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:18.159692    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7608-84c18e"]
Sep 16 14:42:18.167878 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:18.167847    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7608-84c18e"]
Sep 16 14:42:18.343896 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:18.343866    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2539c0-rdkn8"]
Sep 16 14:42:18.344265 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:18.344252    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9568c301-f45f-4d25-8b47-89ad95867bf1" containerName="test-curl-7608-84c18e"
Sep 16 14:42:18.344316 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:18.344268    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="9568c301-f45f-4d25-8b47-89ad95867bf1" containerName="test-curl-7608-84c18e"
Sep 16 14:42:18.344352 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:18.344325    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="9568c301-f45f-4d25-8b47-89ad95867bf1" containerName="test-curl-7608-84c18e"
Sep 16 14:42:18.347597 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:18.347574    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2539c0-rdkn8"
Sep 16 14:42:18.351089 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:18.351069    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4jf\""
Sep 16 14:42:18.351235 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:18.351124    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:42:18.359937 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:18.359911    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2539c0-rdkn8"]
Sep 16 14:42:18.377026 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:18.376990    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b81f64-92f2-4d26-ab25-2104b4ca8d85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2539c0-rdkn8\" (UID: \"59b81f64-92f2-4d26-ab25-2104b4ca8d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2539c0-rdkn8"
Sep 16 14:42:18.478422 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:18.478305    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b81f64-92f2-4d26-ab25-2104b4ca8d85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2539c0-rdkn8\" (UID: \"59b81f64-92f2-4d26-ab25-2104b4ca8d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2539c0-rdkn8"
Sep 16 14:42:18.478944 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:18.478919    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b81f64-92f2-4d26-ab25-2104b4ca8d85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-2539c0-rdkn8\" (UID: \"59b81f64-92f2-4d26-ab25-2104b4ca8d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2539c0-rdkn8"
Sep 16 14:42:18.532895 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:42:18.532866    2587 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7608-84c18e" container="test-curl-7608-84c18e"
Sep 16 14:42:18.658721 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:18.658683    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2539c0-rdkn8"
Sep 16 14:42:18.939171 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:18.939148    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2539c0-rdkn8"]
Sep 16 14:42:18.941365 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:42:18.941337    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b81f64_92f2_4d26_ab25_2104b4ca8d85.slice/crio-fd81f82cb7c334d98716c9d3afe28020a5773758338cb8f9778e34656706b631 WatchSource:0}: Error finding container fd81f82cb7c334d98716c9d3afe28020a5773758338cb8f9778e34656706b631: Status 404 returned error can't find the container with id fd81f82cb7c334d98716c9d3afe28020a5773758338cb8f9778e34656706b631
Sep 16 14:42:19.539534 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:19.539443    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2539c0-rdkn8" event={"ID":"59b81f64-92f2-4d26-ab25-2104b4ca8d85","Type":"ContainerStarted","Data":"fc38a98d7f9e638a8a24c9a9c5c534a51f0aa698b72a98becfe33ceec1567c14"}
Sep 16 14:42:19.539534 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:19.539513    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2539c0-rdkn8" event={"ID":"59b81f64-92f2-4d26-ab25-2104b4ca8d85","Type":"ContainerStarted","Data":"fd81f82cb7c334d98716c9d3afe28020a5773758338cb8f9778e34656706b631"}
Sep 16 14:42:19.540051 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:19.539656    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7608-84c18e" podUID="9568c301-f45f-4d25-8b47-89ad95867bf1" containerName="test-curl-7608-84c18e" containerID="cri-o://a52d7799ac06aac97ff5467e89a64a0f92bf74081cd0b566edda9e149ec8a4ec" gracePeriod=2
Sep 16 14:42:19.557930 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:19.557879    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2539c0-rdkn8" podStartSLOduration=1.55786281 podStartE2EDuration="1.55786281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:42:19.556042042 +0000 UTC m=+1688.159667073" watchObservedRunningTime="2026-09-16 14:42:19.55786281 +0000 UTC m=+1688.161487843"
Sep 16 14:42:19.558331 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:42:19.558305    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7608-84c18e\" is forbidden: User \"system:node:ip-10-0-132-217.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-217.ec2.internal' and this object" podUID="9568c301-f45f-4d25-8b47-89ad95867bf1" pod="opendatahub/test-curl-7608-84c18e"
Sep 16 14:42:20.547472 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:20.547433    2587 generic.go:356] "Generic (PLEG): container finished" podID="59b81f64-92f2-4d26-ab25-2104b4ca8d85" containerID="fc38a98d7f9e638a8a24c9a9c5c534a51f0aa698b72a98becfe33ceec1567c14" exitCode=0
Sep 16 14:42:20.547844 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:20.547520    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2539c0-rdkn8" event={"ID":"59b81f64-92f2-4d26-ab25-2104b4ca8d85","Type":"ContainerDied","Data":"fc38a98d7f9e638a8a24c9a9c5c534a51f0aa698b72a98becfe33ceec1567c14"}
Sep 16 14:42:21.695266 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:21.695240    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2539c0-rdkn8"
Sep 16 14:42:21.734105 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:21.734078    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-84c18e"
Sep 16 14:42:21.810691 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:21.810603    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvhtd\" (UniqueName: \"kubernetes.io/projected/9568c301-f45f-4d25-8b47-89ad95867bf1-kube-api-access-zvhtd\") pod \"9568c301-f45f-4d25-8b47-89ad95867bf1\" (UID: \"9568c301-f45f-4d25-8b47-89ad95867bf1\") "
Sep 16 14:42:21.810691 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:21.810675    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b81f64-92f2-4d26-ab25-2104b4ca8d85-maas-api-service-ca\") pod \"59b81f64-92f2-4d26-ab25-2104b4ca8d85\" (UID: \"59b81f64-92f2-4d26-ab25-2104b4ca8d85\") "
Sep 16 14:42:21.811078 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:21.811050    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b81f64-92f2-4d26-ab25-2104b4ca8d85-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "59b81f64-92f2-4d26-ab25-2104b4ca8d85" (UID: "59b81f64-92f2-4d26-ab25-2104b4ca8d85"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:42:21.813071 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:21.813045    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9568c301-f45f-4d25-8b47-89ad95867bf1-kube-api-access-zvhtd" (OuterVolumeSpecName: "kube-api-access-zvhtd") pod "9568c301-f45f-4d25-8b47-89ad95867bf1" (UID: "9568c301-f45f-4d25-8b47-89ad95867bf1"). InnerVolumeSpecName "kube-api-access-zvhtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:42:21.905574 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:21.905534    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9568c301-f45f-4d25-8b47-89ad95867bf1" path="/var/lib/kubelet/pods/9568c301-f45f-4d25-8b47-89ad95867bf1/volumes"
Sep 16 14:42:21.911230 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:21.911202    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvhtd\" (UniqueName: \"kubernetes.io/projected/9568c301-f45f-4d25-8b47-89ad95867bf1-kube-api-access-zvhtd\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:42:21.911230 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:21.911228    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b81f64-92f2-4d26-ab25-2104b4ca8d85-maas-api-service-ca\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:42:22.559195 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:22.559165    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2539c0-rdkn8"
Sep 16 14:42:22.559370 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:22.559170    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2539c0-rdkn8" event={"ID":"59b81f64-92f2-4d26-ab25-2104b4ca8d85","Type":"ContainerDied","Data":"fd81f82cb7c334d98716c9d3afe28020a5773758338cb8f9778e34656706b631"}
Sep 16 14:42:22.559370 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:22.559295    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd81f82cb7c334d98716c9d3afe28020a5773758338cb8f9778e34656706b631"
Sep 16 14:42:22.560582 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:22.560558    2587 generic.go:356] "Generic (PLEG): container finished" podID="9568c301-f45f-4d25-8b47-89ad95867bf1" containerID="a52d7799ac06aac97ff5467e89a64a0f92bf74081cd0b566edda9e149ec8a4ec" exitCode=137
Sep 16 14:42:22.560685 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:22.560602    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-84c18e"
Sep 16 14:42:22.560685 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:22.560657    2587 scope.go:117] "RemoveContainer" containerID="a52d7799ac06aac97ff5467e89a64a0f92bf74081cd0b566edda9e149ec8a4ec"
Sep 16 14:42:22.569480 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:22.569457    2587 scope.go:117] "RemoveContainer" containerID="a52d7799ac06aac97ff5467e89a64a0f92bf74081cd0b566edda9e149ec8a4ec"
Sep 16 14:42:22.569782 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:42:22.569759    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a52d7799ac06aac97ff5467e89a64a0f92bf74081cd0b566edda9e149ec8a4ec\": container with ID starting with a52d7799ac06aac97ff5467e89a64a0f92bf74081cd0b566edda9e149ec8a4ec not found: ID does not exist" containerID="a52d7799ac06aac97ff5467e89a64a0f92bf74081cd0b566edda9e149ec8a4ec"
Sep 16 14:42:22.569890 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:22.569789    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a52d7799ac06aac97ff5467e89a64a0f92bf74081cd0b566edda9e149ec8a4ec"} err="failed to get container status \"a52d7799ac06aac97ff5467e89a64a0f92bf74081cd0b566edda9e149ec8a4ec\": rpc error: code = NotFound desc = could not find container \"a52d7799ac06aac97ff5467e89a64a0f92bf74081cd0b566edda9e149ec8a4ec\": container with ID starting with a52d7799ac06aac97ff5467e89a64a0f92bf74081cd0b566edda9e149ec8a4ec not found: ID does not exist"
Sep 16 14:42:28.869845 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:28.869809    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx"]
Sep 16 14:42:28.870290 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:28.870078    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx" podUID="ed2b8a7c-a772-40a9-b610-3828c5d3380c" containerName="payload-processing" containerID="cri-o://eed98af2615466373b1675397d3d9b7071afa88199e4bc0b78b946f04972f830" gracePeriod=30
Sep 16 14:42:28.876561 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:28.876534    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp"]
Sep 16 14:42:28.876792 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:28.876743    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp" podUID="fc51bfd4-1df1-481f-92d3-9e761495f65a" containerName="payload-pre-processing" containerID="cri-o://47b18e434d2c7c0d7530d13ddb772125a2e62532862b852d7fb414a665d73ac6" gracePeriod=30
Sep 16 14:42:29.035773 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.035740    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2539c0-rdkn8"]
Sep 16 14:42:29.038762 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.038733    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-2539c0-rdkn8"]
Sep 16 14:42:29.168171 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.168144    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx"
Sep 16 14:42:29.170749 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.170727    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp"
Sep 16 14:42:29.283104 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.283065    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed2b8a7c-a772-40a9-b610-3828c5d3380c-plugins-config-volume\") pod \"ed2b8a7c-a772-40a9-b610-3828c5d3380c\" (UID: \"ed2b8a7c-a772-40a9-b610-3828c5d3380c\") "
Sep 16 14:42:29.283281 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.283120    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc51bfd4-1df1-481f-92d3-9e761495f65a-plugins-config-volume\") pod \"fc51bfd4-1df1-481f-92d3-9e761495f65a\" (UID: \"fc51bfd4-1df1-481f-92d3-9e761495f65a\") "
Sep 16 14:42:29.283281 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.283210    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bngtj\" (UniqueName: \"kubernetes.io/projected/fc51bfd4-1df1-481f-92d3-9e761495f65a-kube-api-access-bngtj\") pod \"fc51bfd4-1df1-481f-92d3-9e761495f65a\" (UID: \"fc51bfd4-1df1-481f-92d3-9e761495f65a\") "
Sep 16 14:42:29.283281 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.283271    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5q6tx\" (UniqueName: \"kubernetes.io/projected/ed2b8a7c-a772-40a9-b610-3828c5d3380c-kube-api-access-5q6tx\") pod \"ed2b8a7c-a772-40a9-b610-3828c5d3380c\" (UID: \"ed2b8a7c-a772-40a9-b610-3828c5d3380c\") "
Sep 16 14:42:29.283575 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.283544    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed2b8a7c-a772-40a9-b610-3828c5d3380c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ed2b8a7c-a772-40a9-b610-3828c5d3380c" (UID: "ed2b8a7c-a772-40a9-b610-3828c5d3380c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:42:29.283714 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.283591    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc51bfd4-1df1-481f-92d3-9e761495f65a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fc51bfd4-1df1-481f-92d3-9e761495f65a" (UID: "fc51bfd4-1df1-481f-92d3-9e761495f65a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:42:29.285494 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.285468    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc51bfd4-1df1-481f-92d3-9e761495f65a-kube-api-access-bngtj" (OuterVolumeSpecName: "kube-api-access-bngtj") pod "fc51bfd4-1df1-481f-92d3-9e761495f65a" (UID: "fc51bfd4-1df1-481f-92d3-9e761495f65a"). InnerVolumeSpecName "kube-api-access-bngtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:42:29.285793 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.285767    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed2b8a7c-a772-40a9-b610-3828c5d3380c-kube-api-access-5q6tx" (OuterVolumeSpecName: "kube-api-access-5q6tx") pod "ed2b8a7c-a772-40a9-b610-3828c5d3380c" (UID: "ed2b8a7c-a772-40a9-b610-3828c5d3380c"). InnerVolumeSpecName "kube-api-access-5q6tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:42:29.384307 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.384277    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q6tx\" (UniqueName: \"kubernetes.io/projected/ed2b8a7c-a772-40a9-b610-3828c5d3380c-kube-api-access-5q6tx\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:42:29.384307 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.384307    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed2b8a7c-a772-40a9-b610-3828c5d3380c-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:42:29.384565 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.384321    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc51bfd4-1df1-481f-92d3-9e761495f65a-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:42:29.384565 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.384335    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bngtj\" (UniqueName: \"kubernetes.io/projected/fc51bfd4-1df1-481f-92d3-9e761495f65a-kube-api-access-bngtj\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:42:29.601628 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.601588    2587 generic.go:356] "Generic (PLEG): container finished" podID="fc51bfd4-1df1-481f-92d3-9e761495f65a" containerID="47b18e434d2c7c0d7530d13ddb772125a2e62532862b852d7fb414a665d73ac6" exitCode=0
Sep 16 14:42:29.601814 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.601638    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp"
Sep 16 14:42:29.601814 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.601676    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp" event={"ID":"fc51bfd4-1df1-481f-92d3-9e761495f65a","Type":"ContainerDied","Data":"47b18e434d2c7c0d7530d13ddb772125a2e62532862b852d7fb414a665d73ac6"}
Sep 16 14:42:29.601814 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.601735    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp" event={"ID":"fc51bfd4-1df1-481f-92d3-9e761495f65a","Type":"ContainerDied","Data":"9af7ecbe928a5f3b28ab626ed7c22e5d0b01ca8f32826f330f2a830a8380e27e"}
Sep 16 14:42:29.601814 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.601765    2587 scope.go:117] "RemoveContainer" containerID="47b18e434d2c7c0d7530d13ddb772125a2e62532862b852d7fb414a665d73ac6"
Sep 16 14:42:29.603109 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.603087    2587 generic.go:356] "Generic (PLEG): container finished" podID="ed2b8a7c-a772-40a9-b610-3828c5d3380c" containerID="eed98af2615466373b1675397d3d9b7071afa88199e4bc0b78b946f04972f830" exitCode=0
Sep 16 14:42:29.603191 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.603132    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx"
Sep 16 14:42:29.603191 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.603172    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx" event={"ID":"ed2b8a7c-a772-40a9-b610-3828c5d3380c","Type":"ContainerDied","Data":"eed98af2615466373b1675397d3d9b7071afa88199e4bc0b78b946f04972f830"}
Sep 16 14:42:29.603267 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.603225    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx" event={"ID":"ed2b8a7c-a772-40a9-b610-3828c5d3380c","Type":"ContainerDied","Data":"db05b2e020f2a045efff7037ece49f6c36549af393d23b2e7bed28adfd9a7338"}
Sep 16 14:42:29.612285 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.612270    2587 scope.go:117] "RemoveContainer" containerID="47b18e434d2c7c0d7530d13ddb772125a2e62532862b852d7fb414a665d73ac6"
Sep 16 14:42:29.612567 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:42:29.612548    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47b18e434d2c7c0d7530d13ddb772125a2e62532862b852d7fb414a665d73ac6\": container with ID starting with 47b18e434d2c7c0d7530d13ddb772125a2e62532862b852d7fb414a665d73ac6 not found: ID does not exist" containerID="47b18e434d2c7c0d7530d13ddb772125a2e62532862b852d7fb414a665d73ac6"
Sep 16 14:42:29.612656 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.612579    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47b18e434d2c7c0d7530d13ddb772125a2e62532862b852d7fb414a665d73ac6"} err="failed to get container status \"47b18e434d2c7c0d7530d13ddb772125a2e62532862b852d7fb414a665d73ac6\": rpc error: code = NotFound desc = could not find container \"47b18e434d2c7c0d7530d13ddb772125a2e62532862b852d7fb414a665d73ac6\": container with ID starting with 47b18e434d2c7c0d7530d13ddb772125a2e62532862b852d7fb414a665d73ac6 not found: ID does not exist"
Sep 16 14:42:29.612656 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.612602    2587 scope.go:117] "RemoveContainer" containerID="eed98af2615466373b1675397d3d9b7071afa88199e4bc0b78b946f04972f830"
Sep 16 14:42:29.620596 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.620581    2587 scope.go:117] "RemoveContainer" containerID="eed98af2615466373b1675397d3d9b7071afa88199e4bc0b78b946f04972f830"
Sep 16 14:42:29.620833 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:42:29.620817    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed98af2615466373b1675397d3d9b7071afa88199e4bc0b78b946f04972f830\": container with ID starting with eed98af2615466373b1675397d3d9b7071afa88199e4bc0b78b946f04972f830 not found: ID does not exist" containerID="eed98af2615466373b1675397d3d9b7071afa88199e4bc0b78b946f04972f830"
Sep 16 14:42:29.620882 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.620840    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed98af2615466373b1675397d3d9b7071afa88199e4bc0b78b946f04972f830"} err="failed to get container status \"eed98af2615466373b1675397d3d9b7071afa88199e4bc0b78b946f04972f830\": rpc error: code = NotFound desc = could not find container \"eed98af2615466373b1675397d3d9b7071afa88199e4bc0b78b946f04972f830\": container with ID starting with eed98af2615466373b1675397d3d9b7071afa88199e4bc0b78b946f04972f830 not found: ID does not exist"
Sep 16 14:42:29.628469 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.628445    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp"]
Sep 16 14:42:29.637275 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.635284    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2539c0-84b4946ff8-zffpp"]
Sep 16 14:42:29.647212 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.647190    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx"]
Sep 16 14:42:29.652908 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.652888    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2539c0-d4647b86c-7bprx"]
Sep 16 14:42:29.902347 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.902313    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59b81f64-92f2-4d26-ab25-2104b4ca8d85" path="/var/lib/kubelet/pods/59b81f64-92f2-4d26-ab25-2104b4ca8d85/volumes"
Sep 16 14:42:29.902759 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.902679    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed2b8a7c-a772-40a9-b610-3828c5d3380c" path="/var/lib/kubelet/pods/ed2b8a7c-a772-40a9-b610-3828c5d3380c/volumes"
Sep 16 14:42:29.902981 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:29.902968    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc51bfd4-1df1-481f-92d3-9e761495f65a" path="/var/lib/kubelet/pods/fc51bfd4-1df1-481f-92d3-9e761495f65a/volumes"
Sep 16 14:42:35.120536 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.120463    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"]
Sep 16 14:42:35.120867 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.120800    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc51bfd4-1df1-481f-92d3-9e761495f65a" containerName="payload-pre-processing"
Sep 16 14:42:35.120867 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.120811    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc51bfd4-1df1-481f-92d3-9e761495f65a" containerName="payload-pre-processing"
Sep 16 14:42:35.120867 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.120831    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59b81f64-92f2-4d26-ab25-2104b4ca8d85" containerName="revoke-keys"
Sep 16 14:42:35.120867 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.120837    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="59b81f64-92f2-4d26-ab25-2104b4ca8d85" containerName="revoke-keys"
Sep 16 14:42:35.120867 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.120844    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed2b8a7c-a772-40a9-b610-3828c5d3380c" containerName="payload-processing"
Sep 16 14:42:35.120867 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.120849    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed2b8a7c-a772-40a9-b610-3828c5d3380c" containerName="payload-processing"
Sep 16 14:42:35.121069 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.120924    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed2b8a7c-a772-40a9-b610-3828c5d3380c" containerName="payload-processing"
Sep 16 14:42:35.121069 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.120933    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="59b81f64-92f2-4d26-ab25-2104b4ca8d85" containerName="revoke-keys"
Sep 16 14:42:35.121069 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.120941    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc51bfd4-1df1-481f-92d3-9e761495f65a" containerName="payload-pre-processing"
Sep 16 14:42:35.125182 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.125164    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"
Sep 16 14:42:35.129027 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.129003    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-9f88e1\"/\"default-dockercfg-lt7qm\""
Sep 16 14:42:35.129170 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.129033    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-9f88e1\"/\"openshift-service-ca.crt\""
Sep 16 14:42:35.129170 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.129013    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-9f88e1\"/\"kube-root-ca.crt\""
Sep 16 14:42:35.129170 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.129013    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-9f88e1\"/\"sub-test-model-9f88e1-kserve-self-signed-certs\""
Sep 16 14:42:35.134838 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.134820    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"]
Sep 16 14:42:35.232894 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.232868    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nfhn\" (UniqueName: \"kubernetes.io/projected/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-kube-api-access-8nfhn\") pod \"sub-test-model-9f88e1-kserve-8546bd845c-sbm4w\" (UID: \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\") " pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"
Sep 16 14:42:35.233036 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.232920    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-model-cache\") pod \"sub-test-model-9f88e1-kserve-8546bd845c-sbm4w\" (UID: \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\") " pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"
Sep 16 14:42:35.233036 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.232995    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-tmp-dir\") pod \"sub-test-model-9f88e1-kserve-8546bd845c-sbm4w\" (UID: \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\") " pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"
Sep 16 14:42:35.233155 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.233036    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-tls-certs\") pod \"sub-test-model-9f88e1-kserve-8546bd845c-sbm4w\" (UID: \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\") " pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"
Sep 16 14:42:35.233155 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.233067    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-dshm\") pod \"sub-test-model-9f88e1-kserve-8546bd845c-sbm4w\" (UID: \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\") " pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"
Sep 16 14:42:35.233155 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.233095    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-home\") pod \"sub-test-model-9f88e1-kserve-8546bd845c-sbm4w\" (UID: \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\") " pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"
Sep 16 14:42:35.333724 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.333693    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-tmp-dir\") pod \"sub-test-model-9f88e1-kserve-8546bd845c-sbm4w\" (UID: \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\") " pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"
Sep 16 14:42:35.333724 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.333725    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-tls-certs\") pod \"sub-test-model-9f88e1-kserve-8546bd845c-sbm4w\" (UID: \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\") " pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"
Sep 16 14:42:35.333943 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.333744    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-dshm\") pod \"sub-test-model-9f88e1-kserve-8546bd845c-sbm4w\" (UID: \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\") " pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"
Sep 16 14:42:35.333943 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.333763    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-home\") pod \"sub-test-model-9f88e1-kserve-8546bd845c-sbm4w\" (UID: \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\") " pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"
Sep 16 14:42:35.333943 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.333800    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nfhn\" (UniqueName: \"kubernetes.io/projected/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-kube-api-access-8nfhn\") pod \"sub-test-model-9f88e1-kserve-8546bd845c-sbm4w\" (UID: \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\") " pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"
Sep 16 14:42:35.333943 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.333834    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-model-cache\") pod \"sub-test-model-9f88e1-kserve-8546bd845c-sbm4w\" (UID: \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\") " pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"
Sep 16 14:42:35.334137 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.334105    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-tmp-dir\") pod \"sub-test-model-9f88e1-kserve-8546bd845c-sbm4w\" (UID: \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\") " pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"
Sep 16 14:42:35.334200 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.334139    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-home\") pod \"sub-test-model-9f88e1-kserve-8546bd845c-sbm4w\" (UID: \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\") " pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"
Sep 16 14:42:35.334200 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.334178    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-model-cache\") pod \"sub-test-model-9f88e1-kserve-8546bd845c-sbm4w\" (UID: \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\") " pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"
Sep 16 14:42:35.336063 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.336036    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-dshm\") pod \"sub-test-model-9f88e1-kserve-8546bd845c-sbm4w\" (UID: \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\") " pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"
Sep 16 14:42:35.336384 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.336364    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-tls-certs\") pod \"sub-test-model-9f88e1-kserve-8546bd845c-sbm4w\" (UID: \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\") " pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"
Sep 16 14:42:35.342883 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.342861    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nfhn\" (UniqueName: \"kubernetes.io/projected/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-kube-api-access-8nfhn\") pod \"sub-test-model-9f88e1-kserve-8546bd845c-sbm4w\" (UID: \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\") " pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"
Sep 16 14:42:35.435083 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.435057    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"
Sep 16 14:42:35.710370 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:35.710347    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"]
Sep 16 14:42:35.712239 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:42:35.712213    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb5abd7b_0d13_48fd_9e8d_f6b4c4f3cf42.slice/crio-2ee3fe639ac076f72f652f716a4f5704938ee4ad021ae496defd2270e300fafd WatchSource:0}: Error finding container 2ee3fe639ac076f72f652f716a4f5704938ee4ad021ae496defd2270e300fafd: Status 404 returned error can't find the container with id 2ee3fe639ac076f72f652f716a4f5704938ee4ad021ae496defd2270e300fafd
Sep 16 14:42:36.646802 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:36.646760    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w" event={"ID":"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42","Type":"ContainerStarted","Data":"e223f8c8f3e6aac90570cc0319b485ea13a80a79ea26527a0a80d9d6767b84d9"}
Sep 16 14:42:36.647160 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:36.646812    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w" event={"ID":"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42","Type":"ContainerStarted","Data":"2ee3fe639ac076f72f652f716a4f5704938ee4ad021ae496defd2270e300fafd"}
Sep 16 14:42:36.665213 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:36.665160    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w" podStartSLOduration=1.665144073 podStartE2EDuration="1.665144073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:42:36.664244113 +0000 UTC m=+1705.267869146" watchObservedRunningTime="2026-09-16 14:42:36.665144073 +0000 UTC m=+1705.268769106"
Sep 16 14:42:45.435813 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:45.435775    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"
Sep 16 14:42:45.436279 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:45.435828    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"
Sep 16 14:42:45.448684 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:45.448654    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"
Sep 16 14:42:45.703270 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:42:45.703195    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"
Sep 16 14:43:44.815335 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:44.815302    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7608-18c4e0"]
Sep 16 14:43:44.818267 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:44.818250    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-18c4e0"
Sep 16 14:43:44.820661 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:44.820642    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gn749\""
Sep 16 14:43:44.825633 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:44.825611    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7608-18c4e0"]
Sep 16 14:43:44.846102 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:44.846075    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v45q6\" (UniqueName: \"kubernetes.io/projected/ba14d9b9-d887-44c3-9824-5ac92f509a15-kube-api-access-v45q6\") pod \"test-curl-7608-18c4e0\" (UID: \"ba14d9b9-d887-44c3-9824-5ac92f509a15\") " pod="opendatahub/test-curl-7608-18c4e0"
Sep 16 14:43:44.947351 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:44.947315    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v45q6\" (UniqueName: \"kubernetes.io/projected/ba14d9b9-d887-44c3-9824-5ac92f509a15-kube-api-access-v45q6\") pod \"test-curl-7608-18c4e0\" (UID: \"ba14d9b9-d887-44c3-9824-5ac92f509a15\") " pod="opendatahub/test-curl-7608-18c4e0"
Sep 16 14:43:44.955828 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:44.955796    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v45q6\" (UniqueName: \"kubernetes.io/projected/ba14d9b9-d887-44c3-9824-5ac92f509a15-kube-api-access-v45q6\") pod \"test-curl-7608-18c4e0\" (UID: \"ba14d9b9-d887-44c3-9824-5ac92f509a15\") " pod="opendatahub/test-curl-7608-18c4e0"
Sep 16 14:43:45.128379 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:45.128297    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-18c4e0"
Sep 16 14:43:45.398810 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:45.398785    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7608-18c4e0"]
Sep 16 14:43:45.401211 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:43:45.401183    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba14d9b9_d887_44c3_9824_5ac92f509a15.slice/crio-39dae781f874ed6c7311108ca3801738e082d4dc82107ae988acde48b36dccb8 WatchSource:0}: Error finding container 39dae781f874ed6c7311108ca3801738e082d4dc82107ae988acde48b36dccb8: Status 404 returned error can't find the container with id 39dae781f874ed6c7311108ca3801738e082d4dc82107ae988acde48b36dccb8
Sep 16 14:43:46.004929 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:46.004887    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7608-18c4e0" event={"ID":"ba14d9b9-d887-44c3-9824-5ac92f509a15","Type":"ContainerStarted","Data":"5da4186bb4b48ce559fe66000ba0d7943a71817f7da68fe636c1fba1b5b891f6"}
Sep 16 14:43:46.004929 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:46.004940    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7608-18c4e0" event={"ID":"ba14d9b9-d887-44c3-9824-5ac92f509a15","Type":"ContainerStarted","Data":"39dae781f874ed6c7311108ca3801738e082d4dc82107ae988acde48b36dccb8"}
Sep 16 14:43:46.022402 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:46.022353    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7608-18c4e0" podStartSLOduration=2.022336361 podStartE2EDuration="2.022336361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:43:46.021991148 +0000 UTC m=+1774.625616181" watchObservedRunningTime="2026-09-16 14:43:46.022336361 +0000 UTC m=+1774.625961394"
Sep 16 14:43:46.697019 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:46.696990    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7608-18c4e0"]
Sep 16 14:43:46.700965 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:46.700938    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7608-18c4e0"]
Sep 16 14:43:46.827563 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:46.827485    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7608-5aa07d"]
Sep 16 14:43:46.827878 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:46.827865    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba14d9b9-d887-44c3-9824-5ac92f509a15" containerName="test-curl-7608-18c4e0"
Sep 16 14:43:46.827921 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:46.827880    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba14d9b9-d887-44c3-9824-5ac92f509a15" containerName="test-curl-7608-18c4e0"
Sep 16 14:43:46.827954 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:46.827937    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba14d9b9-d887-44c3-9824-5ac92f509a15" containerName="test-curl-7608-18c4e0"
Sep 16 14:43:46.830955 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:46.830939    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-5aa07d"
Sep 16 14:43:46.847531 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:46.847499    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7608-5aa07d"]
Sep 16 14:43:46.861147 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:46.861122    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qht8\" (UniqueName: \"kubernetes.io/projected/ab151e63-54dc-42c8-bd4d-39c9e5edadf1-kube-api-access-4qht8\") pod \"test-curl-7608-5aa07d\" (UID: \"ab151e63-54dc-42c8-bd4d-39c9e5edadf1\") " pod="opendatahub/test-curl-7608-5aa07d"
Sep 16 14:43:46.962060 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:46.962026    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qht8\" (UniqueName: \"kubernetes.io/projected/ab151e63-54dc-42c8-bd4d-39c9e5edadf1-kube-api-access-4qht8\") pod \"test-curl-7608-5aa07d\" (UID: \"ab151e63-54dc-42c8-bd4d-39c9e5edadf1\") " pod="opendatahub/test-curl-7608-5aa07d"
Sep 16 14:43:46.970692 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:46.970663    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qht8\" (UniqueName: \"kubernetes.io/projected/ab151e63-54dc-42c8-bd4d-39c9e5edadf1-kube-api-access-4qht8\") pod \"test-curl-7608-5aa07d\" (UID: \"ab151e63-54dc-42c8-bd4d-39c9e5edadf1\") " pod="opendatahub/test-curl-7608-5aa07d"
Sep 16 14:43:47.009932 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:43:47.009908    2587 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7608-18c4e0" container="test-curl-7608-18c4e0"
Sep 16 14:43:47.143118 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:47.143036    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-5aa07d"
Sep 16 14:43:47.420430 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:43:47.420381    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab151e63_54dc_42c8_bd4d_39c9e5edadf1.slice/crio-765078a0697c7d8f8c99cc2fb4792965378bf3762a18bdceb84f8870f4fa9eab WatchSource:0}: Error finding container 765078a0697c7d8f8c99cc2fb4792965378bf3762a18bdceb84f8870f4fa9eab: Status 404 returned error can't find the container with id 765078a0697c7d8f8c99cc2fb4792965378bf3762a18bdceb84f8870f4fa9eab
Sep 16 14:43:47.420995 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:47.420934    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7608-5aa07d"]
Sep 16 14:43:48.016754 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:48.016713    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7608-5aa07d" event={"ID":"ab151e63-54dc-42c8-bd4d-39c9e5edadf1","Type":"ContainerStarted","Data":"eb2940c4389f1e7c24adda3f62dd6d957459c21e02835104a0bdc9e80e293b85"}
Sep 16 14:43:48.016754 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:48.016761    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7608-5aa07d" event={"ID":"ab151e63-54dc-42c8-bd4d-39c9e5edadf1","Type":"ContainerStarted","Data":"765078a0697c7d8f8c99cc2fb4792965378bf3762a18bdceb84f8870f4fa9eab"}
Sep 16 14:43:48.017229 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:48.016784    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7608-18c4e0" podUID="ba14d9b9-d887-44c3-9824-5ac92f509a15" containerName="test-curl-7608-18c4e0" containerID="cri-o://5da4186bb4b48ce559fe66000ba0d7943a71817f7da68fe636c1fba1b5b891f6" gracePeriod=2
Sep 16 14:43:48.035042 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:48.034998    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7608-5aa07d" podStartSLOduration=2.034983046 podStartE2EDuration="2.034983046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:43:48.034108147 +0000 UTC m=+1776.637733179" watchObservedRunningTime="2026-09-16 14:43:48.034983046 +0000 UTC m=+1776.638608077"
Sep 16 14:43:48.713522 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:48.713445    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7608-5aa07d"]
Sep 16 14:43:48.718589 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:48.718562    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7608-5aa07d"]
Sep 16 14:43:49.022233 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:43:49.022139    2587 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7608-5aa07d" container="test-curl-7608-5aa07d"
Sep 16 14:43:50.027424 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:50.027371    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7608-5aa07d" podUID="ab151e63-54dc-42c8-bd4d-39c9e5edadf1" containerName="test-curl-7608-5aa07d" containerID="cri-o://eb2940c4389f1e7c24adda3f62dd6d957459c21e02835104a0bdc9e80e293b85" gracePeriod=2
Sep 16 14:43:50.188111 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:50.188086    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-18c4e0"
Sep 16 14:43:50.291950 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:50.291867    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v45q6\" (UniqueName: \"kubernetes.io/projected/ba14d9b9-d887-44c3-9824-5ac92f509a15-kube-api-access-v45q6\") pod \"ba14d9b9-d887-44c3-9824-5ac92f509a15\" (UID: \"ba14d9b9-d887-44c3-9824-5ac92f509a15\") "
Sep 16 14:43:50.294162 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:50.294129    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba14d9b9-d887-44c3-9824-5ac92f509a15-kube-api-access-v45q6" (OuterVolumeSpecName: "kube-api-access-v45q6") pod "ba14d9b9-d887-44c3-9824-5ac92f509a15" (UID: "ba14d9b9-d887-44c3-9824-5ac92f509a15"). InnerVolumeSpecName "kube-api-access-v45q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:43:50.392840 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:50.392798    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v45q6\" (UniqueName: \"kubernetes.io/projected/ba14d9b9-d887-44c3-9824-5ac92f509a15-kube-api-access-v45q6\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:43:50.937714 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:50.937681    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"]
Sep 16 14:43:51.033590 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:51.033546    2587 generic.go:356] "Generic (PLEG): container finished" podID="ba14d9b9-d887-44c3-9824-5ac92f509a15" containerID="5da4186bb4b48ce559fe66000ba0d7943a71817f7da68fe636c1fba1b5b891f6" exitCode=137
Sep 16 14:43:51.033994 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:51.033621    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-18c4e0"
Sep 16 14:43:51.033994 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:51.033651    2587 scope.go:117] "RemoveContainer" containerID="5da4186bb4b48ce559fe66000ba0d7943a71817f7da68fe636c1fba1b5b891f6"
Sep 16 14:43:51.043978 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:51.043957    2587 scope.go:117] "RemoveContainer" containerID="5da4186bb4b48ce559fe66000ba0d7943a71817f7da68fe636c1fba1b5b891f6"
Sep 16 14:43:51.044284 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:43:51.044269    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5da4186bb4b48ce559fe66000ba0d7943a71817f7da68fe636c1fba1b5b891f6\": container with ID starting with 5da4186bb4b48ce559fe66000ba0d7943a71817f7da68fe636c1fba1b5b891f6 not found: ID does not exist" containerID="5da4186bb4b48ce559fe66000ba0d7943a71817f7da68fe636c1fba1b5b891f6"
Sep 16 14:43:51.044355 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:51.044297    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da4186bb4b48ce559fe66000ba0d7943a71817f7da68fe636c1fba1b5b891f6"} err="failed to get container status \"5da4186bb4b48ce559fe66000ba0d7943a71817f7da68fe636c1fba1b5b891f6\": rpc error: code = NotFound desc = could not find container \"5da4186bb4b48ce559fe66000ba0d7943a71817f7da68fe636c1fba1b5b891f6\": container with ID starting with 5da4186bb4b48ce559fe66000ba0d7943a71817f7da68fe636c1fba1b5b891f6 not found: ID does not exist"
Sep 16 14:43:51.044355 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:43:51.044264    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7608-18c4e0\" is forbidden: User \"system:node:ip-10-0-132-217.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-217.ec2.internal' and this object" podUID="ba14d9b9-d887-44c3-9824-5ac92f509a15" pod="opendatahub/test-curl-7608-18c4e0"
Sep 16 14:43:51.903227 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:43:51.903122    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7608-18c4e0\" is forbidden: User \"system:node:ip-10-0-132-217.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-217.ec2.internal' and this object" podUID="ba14d9b9-d887-44c3-9824-5ac92f509a15" pod="opendatahub/test-curl-7608-18c4e0"
Sep 16 14:43:51.904863 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:51.904837    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba14d9b9-d887-44c3-9824-5ac92f509a15" path="/var/lib/kubelet/pods/ba14d9b9-d887-44c3-9824-5ac92f509a15/volumes"
Sep 16 14:43:52.208041 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:52.208015    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-5aa07d"
Sep 16 14:43:52.308724 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:52.308692    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qht8\" (UniqueName: \"kubernetes.io/projected/ab151e63-54dc-42c8-bd4d-39c9e5edadf1-kube-api-access-4qht8\") pod \"ab151e63-54dc-42c8-bd4d-39c9e5edadf1\" (UID: \"ab151e63-54dc-42c8-bd4d-39c9e5edadf1\") "
Sep 16 14:43:52.311071 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:52.311046    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab151e63-54dc-42c8-bd4d-39c9e5edadf1-kube-api-access-4qht8" (OuterVolumeSpecName: "kube-api-access-4qht8") pod "ab151e63-54dc-42c8-bd4d-39c9e5edadf1" (UID: "ab151e63-54dc-42c8-bd4d-39c9e5edadf1"). InnerVolumeSpecName "kube-api-access-4qht8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:43:52.409933 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:52.409909    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qht8\" (UniqueName: \"kubernetes.io/projected/ab151e63-54dc-42c8-bd4d-39c9e5edadf1-kube-api-access-4qht8\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:43:53.048002 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:53.047970    2587 generic.go:356] "Generic (PLEG): container finished" podID="ab151e63-54dc-42c8-bd4d-39c9e5edadf1" containerID="eb2940c4389f1e7c24adda3f62dd6d957459c21e02835104a0bdc9e80e293b85" exitCode=137
Sep 16 14:43:53.048177 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:53.048011    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-5aa07d"
Sep 16 14:43:53.048177 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:53.048032    2587 scope.go:117] "RemoveContainer" containerID="eb2940c4389f1e7c24adda3f62dd6d957459c21e02835104a0bdc9e80e293b85"
Sep 16 14:43:53.057968 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:43:53.057927    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7608-5aa07d\" is forbidden: User \"system:node:ip-10-0-132-217.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-217.ec2.internal' and this object" podUID="ab151e63-54dc-42c8-bd4d-39c9e5edadf1" pod="opendatahub/test-curl-7608-5aa07d"
Sep 16 14:43:53.059948 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:53.059927    2587 scope.go:117] "RemoveContainer" containerID="eb2940c4389f1e7c24adda3f62dd6d957459c21e02835104a0bdc9e80e293b85"
Sep 16 14:43:53.060244 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:43:53.060221    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb2940c4389f1e7c24adda3f62dd6d957459c21e02835104a0bdc9e80e293b85\": container with ID starting with eb2940c4389f1e7c24adda3f62dd6d957459c21e02835104a0bdc9e80e293b85 not found: ID does not exist" containerID="eb2940c4389f1e7c24adda3f62dd6d957459c21e02835104a0bdc9e80e293b85"
Sep 16 14:43:53.060301 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:53.060250    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb2940c4389f1e7c24adda3f62dd6d957459c21e02835104a0bdc9e80e293b85"} err="failed to get container status \"eb2940c4389f1e7c24adda3f62dd6d957459c21e02835104a0bdc9e80e293b85\": rpc error: code = NotFound desc = could not find container \"eb2940c4389f1e7c24adda3f62dd6d957459c21e02835104a0bdc9e80e293b85\": container with ID starting with eb2940c4389f1e7c24adda3f62dd6d957459c21e02835104a0bdc9e80e293b85 not found: ID does not exist"
Sep 16 14:43:53.903341 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:43:53.903300    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab151e63-54dc-42c8-bd4d-39c9e5edadf1" path="/var/lib/kubelet/pods/ab151e63-54dc-42c8-bd4d-39c9e5edadf1/volumes"
Sep 16 14:44:05.958275 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:05.958146    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w" podUID="cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42" containerName="main" containerID="cri-o://e223f8c8f3e6aac90570cc0319b485ea13a80a79ea26527a0a80d9d6767b84d9" gracePeriod=45
Sep 16 14:44:06.121018 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:06.120977    2587 generic.go:356] "Generic (PLEG): container finished" podID="cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42" containerID="e223f8c8f3e6aac90570cc0319b485ea13a80a79ea26527a0a80d9d6767b84d9" exitCode=0
Sep 16 14:44:06.121190 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:06.121047    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w" event={"ID":"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42","Type":"ContainerDied","Data":"e223f8c8f3e6aac90570cc0319b485ea13a80a79ea26527a0a80d9d6767b84d9"}
Sep 16 14:44:06.219317 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:06.219255    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"
Sep 16 14:44:06.331562 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:06.331526    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-tmp-dir\") pod \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\" (UID: \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\") "
Sep 16 14:44:06.331771 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:06.331617    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-home\") pod \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\" (UID: \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\") "
Sep 16 14:44:06.331771 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:06.331642    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-dshm\") pod \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\" (UID: \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\") "
Sep 16 14:44:06.331771 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:06.331667    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nfhn\" (UniqueName: \"kubernetes.io/projected/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-kube-api-access-8nfhn\") pod \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\" (UID: \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\") "
Sep 16 14:44:06.331771 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:06.331691    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-tls-certs\") pod \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\" (UID: \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\") "
Sep 16 14:44:06.331771 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:06.331748    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-model-cache\") pod \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\" (UID: \"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42\") "
Sep 16 14:44:06.332018 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:06.331882    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-home" (OuterVolumeSpecName: "home") pod "cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42" (UID: "cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:44:06.332018 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:06.331898    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42" (UID: "cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:44:06.332132 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:06.332079    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-model-cache" (OuterVolumeSpecName: "model-cache") pod "cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42" (UID: "cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:44:06.332191 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:06.332140    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-model-cache\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:44:06.332191 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:06.332175    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-tmp-dir\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:44:06.332191 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:06.332184    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-home\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:44:06.334020 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:06.333989    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42" (UID: "cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:44:06.334020 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:06.334009    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-dshm" (OuterVolumeSpecName: "dshm") pod "cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42" (UID: "cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:44:06.334159 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:06.334022    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-kube-api-access-8nfhn" (OuterVolumeSpecName: "kube-api-access-8nfhn") pod "cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42" (UID: "cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42"). InnerVolumeSpecName "kube-api-access-8nfhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:44:06.432766 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:06.432731    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-dshm\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:44:06.432766 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:06.432760    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nfhn\" (UniqueName: \"kubernetes.io/projected/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-kube-api-access-8nfhn\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:44:06.432766 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:06.432770    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42-tls-certs\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:44:07.127951 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:07.127917    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w" event={"ID":"cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42","Type":"ContainerDied","Data":"2ee3fe639ac076f72f652f716a4f5704938ee4ad021ae496defd2270e300fafd"}
Sep 16 14:44:07.127951 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:07.127947    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"
Sep 16 14:44:07.128520 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:07.127975    2587 scope.go:117] "RemoveContainer" containerID="e223f8c8f3e6aac90570cc0319b485ea13a80a79ea26527a0a80d9d6767b84d9"
Sep 16 14:44:07.150882 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:07.150851    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"]
Sep 16 14:44:07.156891 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:07.156862    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9f88e1/sub-test-model-9f88e1-kserve-8546bd845c-sbm4w"]
Sep 16 14:44:07.903251 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:07.903217    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42" path="/var/lib/kubelet/pods/cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42/volumes"
Sep 16 14:44:12.285508 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:12.285477    2587 scope.go:117] "RemoveContainer" containerID="564d951e0312c4b09f914a517e45adc833fb156c7f120a20b02f7f064b743830"
Sep 16 14:44:27.325087 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.325056    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"]
Sep 16 14:44:27.325631 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.325434    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42" containerName="main"
Sep 16 14:44:27.325631 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.325451    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42" containerName="main"
Sep 16 14:44:27.325631 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.325463    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab151e63-54dc-42c8-bd4d-39c9e5edadf1" containerName="test-curl-7608-5aa07d"
Sep 16 14:44:27.325631 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.325469    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab151e63-54dc-42c8-bd4d-39c9e5edadf1" containerName="test-curl-7608-5aa07d"
Sep 16 14:44:27.325631 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.325538    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab151e63-54dc-42c8-bd4d-39c9e5edadf1" containerName="test-curl-7608-5aa07d"
Sep 16 14:44:27.325631 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.325548    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb5abd7b-0d13-48fd-9e8d-f6b4c4f3cf42" containerName="main"
Sep 16 14:44:27.328593 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.328574    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"
Sep 16 14:44:27.339305 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.339275    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-055ce3\"/\"rate-test-model-055ce3-kserve-self-signed-certs\""
Sep 16 14:44:27.339900 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.339333    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-055ce3\"/\"openshift-service-ca.crt\""
Sep 16 14:44:27.340026 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.339357    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-055ce3\"/\"default-dockercfg-v45lm\""
Sep 16 14:44:27.340094 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.339444    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-055ce3\"/\"kube-root-ca.crt\""
Sep 16 14:44:27.341870 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.341829    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"]
Sep 16 14:44:27.417440 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.417381    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65v4z\" (UniqueName: \"kubernetes.io/projected/f1706bc4-9e9b-4767-84b2-d8e49111e048-kube-api-access-65v4z\") pod \"rate-test-model-055ce3-kserve-656fc44487-ljb99\" (UID: \"f1706bc4-9e9b-4767-84b2-d8e49111e048\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"
Sep 16 14:44:27.417597 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.417462    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1706bc4-9e9b-4767-84b2-d8e49111e048-model-cache\") pod \"rate-test-model-055ce3-kserve-656fc44487-ljb99\" (UID: \"f1706bc4-9e9b-4767-84b2-d8e49111e048\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"
Sep 16 14:44:27.417597 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.417491    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1706bc4-9e9b-4767-84b2-d8e49111e048-dshm\") pod \"rate-test-model-055ce3-kserve-656fc44487-ljb99\" (UID: \"f1706bc4-9e9b-4767-84b2-d8e49111e048\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"
Sep 16 14:44:27.417597 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.417513    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1706bc4-9e9b-4767-84b2-d8e49111e048-tmp-dir\") pod \"rate-test-model-055ce3-kserve-656fc44487-ljb99\" (UID: \"f1706bc4-9e9b-4767-84b2-d8e49111e048\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"
Sep 16 14:44:27.417597 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.417530    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1706bc4-9e9b-4767-84b2-d8e49111e048-tls-certs\") pod \"rate-test-model-055ce3-kserve-656fc44487-ljb99\" (UID: \"f1706bc4-9e9b-4767-84b2-d8e49111e048\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"
Sep 16 14:44:27.417597 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.417569    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1706bc4-9e9b-4767-84b2-d8e49111e048-home\") pod \"rate-test-model-055ce3-kserve-656fc44487-ljb99\" (UID: \"f1706bc4-9e9b-4767-84b2-d8e49111e048\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"
Sep 16 14:44:27.518367 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.518333    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65v4z\" (UniqueName: \"kubernetes.io/projected/f1706bc4-9e9b-4767-84b2-d8e49111e048-kube-api-access-65v4z\") pod \"rate-test-model-055ce3-kserve-656fc44487-ljb99\" (UID: \"f1706bc4-9e9b-4767-84b2-d8e49111e048\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"
Sep 16 14:44:27.518561 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.518415    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1706bc4-9e9b-4767-84b2-d8e49111e048-model-cache\") pod \"rate-test-model-055ce3-kserve-656fc44487-ljb99\" (UID: \"f1706bc4-9e9b-4767-84b2-d8e49111e048\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"
Sep 16 14:44:27.518561 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.518454    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1706bc4-9e9b-4767-84b2-d8e49111e048-dshm\") pod \"rate-test-model-055ce3-kserve-656fc44487-ljb99\" (UID: \"f1706bc4-9e9b-4767-84b2-d8e49111e048\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"
Sep 16 14:44:27.518561 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.518484    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1706bc4-9e9b-4767-84b2-d8e49111e048-tmp-dir\") pod \"rate-test-model-055ce3-kserve-656fc44487-ljb99\" (UID: \"f1706bc4-9e9b-4767-84b2-d8e49111e048\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"
Sep 16 14:44:27.518561 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.518508    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1706bc4-9e9b-4767-84b2-d8e49111e048-tls-certs\") pod \"rate-test-model-055ce3-kserve-656fc44487-ljb99\" (UID: \"f1706bc4-9e9b-4767-84b2-d8e49111e048\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"
Sep 16 14:44:27.518561 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.518537    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1706bc4-9e9b-4767-84b2-d8e49111e048-home\") pod \"rate-test-model-055ce3-kserve-656fc44487-ljb99\" (UID: \"f1706bc4-9e9b-4767-84b2-d8e49111e048\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"
Sep 16 14:44:27.518870 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.518835    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1706bc4-9e9b-4767-84b2-d8e49111e048-model-cache\") pod \"rate-test-model-055ce3-kserve-656fc44487-ljb99\" (UID: \"f1706bc4-9e9b-4767-84b2-d8e49111e048\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"
Sep 16 14:44:27.518870 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.518851    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1706bc4-9e9b-4767-84b2-d8e49111e048-tmp-dir\") pod \"rate-test-model-055ce3-kserve-656fc44487-ljb99\" (UID: \"f1706bc4-9e9b-4767-84b2-d8e49111e048\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"
Sep 16 14:44:27.519031 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.518920    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1706bc4-9e9b-4767-84b2-d8e49111e048-home\") pod \"rate-test-model-055ce3-kserve-656fc44487-ljb99\" (UID: \"f1706bc4-9e9b-4767-84b2-d8e49111e048\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"
Sep 16 14:44:27.520941 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.520921    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1706bc4-9e9b-4767-84b2-d8e49111e048-dshm\") pod \"rate-test-model-055ce3-kserve-656fc44487-ljb99\" (UID: \"f1706bc4-9e9b-4767-84b2-d8e49111e048\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"
Sep 16 14:44:27.521130 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.521113    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1706bc4-9e9b-4767-84b2-d8e49111e048-tls-certs\") pod \"rate-test-model-055ce3-kserve-656fc44487-ljb99\" (UID: \"f1706bc4-9e9b-4767-84b2-d8e49111e048\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"
Sep 16 14:44:27.528231 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.528209    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65v4z\" (UniqueName: \"kubernetes.io/projected/f1706bc4-9e9b-4767-84b2-d8e49111e048-kube-api-access-65v4z\") pod \"rate-test-model-055ce3-kserve-656fc44487-ljb99\" (UID: \"f1706bc4-9e9b-4767-84b2-d8e49111e048\") " pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"
Sep 16 14:44:27.647187 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.647160    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"
Sep 16 14:44:27.919942 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:27.919918    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"]
Sep 16 14:44:27.922018 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:44:27.921987    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1706bc4_9e9b_4767_84b2_d8e49111e048.slice/crio-65ee2e83255aec46c68dd8d140af7c8223215570f8ef0d4e1506ff8242e6bbb2 WatchSource:0}: Error finding container 65ee2e83255aec46c68dd8d140af7c8223215570f8ef0d4e1506ff8242e6bbb2: Status 404 returned error can't find the container with id 65ee2e83255aec46c68dd8d140af7c8223215570f8ef0d4e1506ff8242e6bbb2
Sep 16 14:44:28.230382 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:28.230294    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99" event={"ID":"f1706bc4-9e9b-4767-84b2-d8e49111e048","Type":"ContainerStarted","Data":"2034a38fecb1737b8e540f82e0c8a665fff7ac7e8b7f6190e50b4f67b38f1d71"}
Sep 16 14:44:28.230382 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:28.230344    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99" event={"ID":"f1706bc4-9e9b-4767-84b2-d8e49111e048","Type":"ContainerStarted","Data":"65ee2e83255aec46c68dd8d140af7c8223215570f8ef0d4e1506ff8242e6bbb2"}
Sep 16 14:44:28.249853 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:28.249800    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99" podStartSLOduration=1.249784352 podStartE2EDuration="1.249784352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:44:28.247892473 +0000 UTC m=+1816.851517506" watchObservedRunningTime="2026-09-16 14:44:28.249784352 +0000 UTC m=+1816.853409384"
Sep 16 14:44:37.648229 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:37.648191    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"
Sep 16 14:44:37.648647 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:37.648452    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"
Sep 16 14:44:37.661309 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:37.661286    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"
Sep 16 14:44:38.292682 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:44:38.292653    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"
Sep 16 14:45:00.134802 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:00.134764    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826165-nh6tj"]
Sep 16 14:45:00.138701 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:00.138683    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-055ce3-29826165-6rc8k"]
Sep 16 14:45:00.138820 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:00.138804    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826165-nh6tj"
Sep 16 14:45:00.141201 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:00.141184    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4jf\""
Sep 16 14:45:00.143034 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:00.143004    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-055ce3-29826165-6rc8k"
Sep 16 14:45:00.145724 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:00.145699    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826165-nh6tj"]
Sep 16 14:45:00.148712 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:00.148665    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-055ce3-29826165-6rc8k"]
Sep 16 14:45:00.153142 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:00.153122    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826165-nh6tj"
Sep 16 14:45:00.159974 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:00.159953    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-055ce3-29826165-6rc8k"
Sep 16 14:45:00.430595 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:00.430566    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826165-nh6tj"]
Sep 16 14:45:00.433481 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:00.433455    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-055ce3-29826165-6rc8k"]
Sep 16 14:45:00.434288 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:45:00.434153    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e3982e_e6c8_4398_a095_45cb4be13377.slice/crio-6381eb8f630afa4a7f3d0b55c6cef9b999cb4c397f5087b553ce5795230afa3a WatchSource:0}: Error finding container 6381eb8f630afa4a7f3d0b55c6cef9b999cb4c397f5087b553ce5795230afa3a: Status 404 returned error can't find the container with id 6381eb8f630afa4a7f3d0b55c6cef9b999cb4c397f5087b553ce5795230afa3a
Sep 16 14:45:00.436187 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:45:00.436133    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc8dea42_c4ae_487b_8889_6797bbf48675.slice/crio-48f8a1d6434a0ce455a0f9702ff13f5750c507c4ed888b3fc27ad74a57fe7256 WatchSource:0}: Error finding container 48f8a1d6434a0ce455a0f9702ff13f5750c507c4ed888b3fc27ad74a57fe7256: Status 404 returned error can't find the container with id 48f8a1d6434a0ce455a0f9702ff13f5750c507c4ed888b3fc27ad74a57fe7256
Sep 16 14:45:01.394842 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:01.394810    2587 generic.go:356] "Generic (PLEG): container finished" podID="f3e3982e-e6c8-4398-a095-45cb4be13377" containerID="cd37ed88d70238d5c4d23eaceab79f0005065e183b062f9b9167efc6b60ad681" exitCode=0
Sep 16 14:45:01.395215 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:01.394856    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826165-nh6tj" event={"ID":"f3e3982e-e6c8-4398-a095-45cb4be13377","Type":"ContainerDied","Data":"cd37ed88d70238d5c4d23eaceab79f0005065e183b062f9b9167efc6b60ad681"}
Sep 16 14:45:01.395215 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:01.394917    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826165-nh6tj" event={"ID":"f3e3982e-e6c8-4398-a095-45cb4be13377","Type":"ContainerStarted","Data":"6381eb8f630afa4a7f3d0b55c6cef9b999cb4c397f5087b553ce5795230afa3a"}
Sep 16 14:45:01.396523 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:01.396503    2587 generic.go:356] "Generic (PLEG): container finished" podID="bc8dea42-c4ae-487b-8889-6797bbf48675" containerID="db04ef57a2e383c4ff247719adfea8fdbdce4a660fd97f2c9674484afe9d75b6" exitCode=0
Sep 16 14:45:01.396629 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:01.396533    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-055ce3-29826165-6rc8k" event={"ID":"bc8dea42-c4ae-487b-8889-6797bbf48675","Type":"ContainerDied","Data":"db04ef57a2e383c4ff247719adfea8fdbdce4a660fd97f2c9674484afe9d75b6"}
Sep 16 14:45:01.396629 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:01.396567    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-055ce3-29826165-6rc8k" event={"ID":"bc8dea42-c4ae-487b-8889-6797bbf48675","Type":"ContainerStarted","Data":"48f8a1d6434a0ce455a0f9702ff13f5750c507c4ed888b3fc27ad74a57fe7256"}
Sep 16 14:45:02.526406 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:02.526370    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-055ce3-29826165-6rc8k"
Sep 16 14:45:02.555806 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:02.555781    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826165-nh6tj"
Sep 16 14:45:03.407460 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:03.407423    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826165-nh6tj" event={"ID":"f3e3982e-e6c8-4398-a095-45cb4be13377","Type":"ContainerDied","Data":"6381eb8f630afa4a7f3d0b55c6cef9b999cb4c397f5087b553ce5795230afa3a"}
Sep 16 14:45:03.407460 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:03.407446    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826165-nh6tj"
Sep 16 14:45:03.407728 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:03.407479    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6381eb8f630afa4a7f3d0b55c6cef9b999cb4c397f5087b553ce5795230afa3a"
Sep 16 14:45:03.408857 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:03.408834    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-055ce3-29826165-6rc8k" event={"ID":"bc8dea42-c4ae-487b-8889-6797bbf48675","Type":"ContainerDied","Data":"48f8a1d6434a0ce455a0f9702ff13f5750c507c4ed888b3fc27ad74a57fe7256"}
Sep 16 14:45:03.408857 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:03.408858    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-055ce3-29826165-6rc8k"
Sep 16 14:45:03.409063 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:03.408868    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48f8a1d6434a0ce455a0f9702ff13f5750c507c4ed888b3fc27ad74a57fe7256"
Sep 16 14:45:12.128263 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:12.128228    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"]
Sep 16 14:45:18.664480 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:18.664441    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn"]
Sep 16 14:45:18.664959 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:18.664942    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc8dea42-c4ae-487b-8889-6797bbf48675" containerName="cleanup"
Sep 16 14:45:18.665014 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:18.664961    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc8dea42-c4ae-487b-8889-6797bbf48675" containerName="cleanup"
Sep 16 14:45:18.665014 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:18.664976    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3e3982e-e6c8-4398-a095-45cb4be13377" containerName="cleanup"
Sep 16 14:45:18.665014 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:18.664983    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3e3982e-e6c8-4398-a095-45cb4be13377" containerName="cleanup"
Sep 16 14:45:18.665119 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:18.665049    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3e3982e-e6c8-4398-a095-45cb4be13377" containerName="cleanup"
Sep 16 14:45:18.665119 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:18.665061    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc8dea42-c4ae-487b-8889-6797bbf48675" containerName="cleanup"
Sep 16 14:45:18.668173 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:18.668155    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn"
Sep 16 14:45:18.671706 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:18.671685    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a70ddb\""
Sep 16 14:45:18.682347 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:18.682325    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn"]
Sep 16 14:45:18.766161 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:18.766134    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzgjs\" (UniqueName: \"kubernetes.io/projected/b5b73e81-38bb-40e0-9c1b-6298fbc42720-kube-api-access-hzgjs\") pod \"payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn\" (UID: \"b5b73e81-38bb-40e0-9c1b-6298fbc42720\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn"
Sep 16 14:45:18.766301 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:18.766200    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b73e81-38bb-40e0-9c1b-6298fbc42720-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn\" (UID: \"b5b73e81-38bb-40e0-9c1b-6298fbc42720\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn"
Sep 16 14:45:18.867468 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:18.867442    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b73e81-38bb-40e0-9c1b-6298fbc42720-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn\" (UID: \"b5b73e81-38bb-40e0-9c1b-6298fbc42720\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn"
Sep 16 14:45:18.867619 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:18.867521    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzgjs\" (UniqueName: \"kubernetes.io/projected/b5b73e81-38bb-40e0-9c1b-6298fbc42720-kube-api-access-hzgjs\") pod \"payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn\" (UID: \"b5b73e81-38bb-40e0-9c1b-6298fbc42720\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn"
Sep 16 14:45:18.868328 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:18.868305    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b73e81-38bb-40e0-9c1b-6298fbc42720-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn\" (UID: \"b5b73e81-38bb-40e0-9c1b-6298fbc42720\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn"
Sep 16 14:45:18.877134 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:18.877103    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzgjs\" (UniqueName: \"kubernetes.io/projected/b5b73e81-38bb-40e0-9c1b-6298fbc42720-kube-api-access-hzgjs\") pod \"payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn\" (UID: \"b5b73e81-38bb-40e0-9c1b-6298fbc42720\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn"
Sep 16 14:45:18.977892 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:18.977828    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn"
Sep 16 14:45:19.255315 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:19.255227    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn"]
Sep 16 14:45:19.264765 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:45:19.264732    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b73e81_38bb_40e0_9c1b_6298fbc42720.slice/crio-e6dbe04821706d69d4f299d9edf7c7840fcc66d388a724d0598de64bb60ef522 WatchSource:0}: Error finding container e6dbe04821706d69d4f299d9edf7c7840fcc66d388a724d0598de64bb60ef522: Status 404 returned error can't find the container with id e6dbe04821706d69d4f299d9edf7c7840fcc66d388a724d0598de64bb60ef522
Sep 16 14:45:19.492550 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:19.492508    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn" event={"ID":"b5b73e81-38bb-40e0-9c1b-6298fbc42720","Type":"ContainerStarted","Data":"5926ecb724f8ac86bbe3000f17a3c21eb46ee099a81158d5e2f6d0579e0aeb21"}
Sep 16 14:45:19.492718 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:19.492575    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn" event={"ID":"b5b73e81-38bb-40e0-9c1b-6298fbc42720","Type":"ContainerStarted","Data":"e6dbe04821706d69d4f299d9edf7c7840fcc66d388a724d0598de64bb60ef522"}
Sep 16 14:45:19.492718 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:19.492627    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn"
Sep 16 14:45:19.511740 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:19.511643    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn" podStartSLOduration=1.511627939 podStartE2EDuration="1.511627939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:45:19.511355558 +0000 UTC m=+1868.114980591" watchObservedRunningTime="2026-09-16 14:45:19.511627939 +0000 UTC m=+1868.115252970"
Sep 16 14:45:27.147789 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.147719    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99" podUID="f1706bc4-9e9b-4767-84b2-d8e49111e048" containerName="main" containerID="cri-o://2034a38fecb1737b8e540f82e0c8a665fff7ac7e8b7f6190e50b4f67b38f1d71" gracePeriod=45
Sep 16 14:45:27.403868 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.403806    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"
Sep 16 14:45:27.537078 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.537045    2587 generic.go:356] "Generic (PLEG): container finished" podID="f1706bc4-9e9b-4767-84b2-d8e49111e048" containerID="2034a38fecb1737b8e540f82e0c8a665fff7ac7e8b7f6190e50b4f67b38f1d71" exitCode=0
Sep 16 14:45:27.537253 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.537084    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"
Sep 16 14:45:27.537253 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.537136    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99" event={"ID":"f1706bc4-9e9b-4767-84b2-d8e49111e048","Type":"ContainerDied","Data":"2034a38fecb1737b8e540f82e0c8a665fff7ac7e8b7f6190e50b4f67b38f1d71"}
Sep 16 14:45:27.537253 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.537173    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99" event={"ID":"f1706bc4-9e9b-4767-84b2-d8e49111e048","Type":"ContainerDied","Data":"65ee2e83255aec46c68dd8d140af7c8223215570f8ef0d4e1506ff8242e6bbb2"}
Sep 16 14:45:27.537253 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.537190    2587 scope.go:117] "RemoveContainer" containerID="2034a38fecb1737b8e540f82e0c8a665fff7ac7e8b7f6190e50b4f67b38f1d71"
Sep 16 14:45:27.547853 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.547833    2587 scope.go:117] "RemoveContainer" containerID="2034a38fecb1737b8e540f82e0c8a665fff7ac7e8b7f6190e50b4f67b38f1d71"
Sep 16 14:45:27.548014 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.547990    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1706bc4-9e9b-4767-84b2-d8e49111e048-home\") pod \"f1706bc4-9e9b-4767-84b2-d8e49111e048\" (UID: \"f1706bc4-9e9b-4767-84b2-d8e49111e048\") "
Sep 16 14:45:27.548083 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.548044    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1706bc4-9e9b-4767-84b2-d8e49111e048-dshm\") pod \"f1706bc4-9e9b-4767-84b2-d8e49111e048\" (UID: \"f1706bc4-9e9b-4767-84b2-d8e49111e048\") "
Sep 16 14:45:27.548083 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.548078    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1706bc4-9e9b-4767-84b2-d8e49111e048-tmp-dir\") pod \"f1706bc4-9e9b-4767-84b2-d8e49111e048\" (UID: \"f1706bc4-9e9b-4767-84b2-d8e49111e048\") "
Sep 16 14:45:27.548182 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.548107    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65v4z\" (UniqueName: \"kubernetes.io/projected/f1706bc4-9e9b-4767-84b2-d8e49111e048-kube-api-access-65v4z\") pod \"f1706bc4-9e9b-4767-84b2-d8e49111e048\" (UID: \"f1706bc4-9e9b-4767-84b2-d8e49111e048\") "
Sep 16 14:45:27.548182 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.548172    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1706bc4-9e9b-4767-84b2-d8e49111e048-tls-certs\") pod \"f1706bc4-9e9b-4767-84b2-d8e49111e048\" (UID: \"f1706bc4-9e9b-4767-84b2-d8e49111e048\") "
Sep 16 14:45:27.548289 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.548236    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1706bc4-9e9b-4767-84b2-d8e49111e048-model-cache\") pod \"f1706bc4-9e9b-4767-84b2-d8e49111e048\" (UID: \"f1706bc4-9e9b-4767-84b2-d8e49111e048\") "
Sep 16 14:45:27.548347 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.548326    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1706bc4-9e9b-4767-84b2-d8e49111e048-home" (OuterVolumeSpecName: "home") pod "f1706bc4-9e9b-4767-84b2-d8e49111e048" (UID: "f1706bc4-9e9b-4767-84b2-d8e49111e048"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:45:27.548433 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.548341    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1706bc4-9e9b-4767-84b2-d8e49111e048-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f1706bc4-9e9b-4767-84b2-d8e49111e048" (UID: "f1706bc4-9e9b-4767-84b2-d8e49111e048"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:45:27.548433 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:45:27.548377    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2034a38fecb1737b8e540f82e0c8a665fff7ac7e8b7f6190e50b4f67b38f1d71\": container with ID starting with 2034a38fecb1737b8e540f82e0c8a665fff7ac7e8b7f6190e50b4f67b38f1d71 not found: ID does not exist" containerID="2034a38fecb1737b8e540f82e0c8a665fff7ac7e8b7f6190e50b4f67b38f1d71"
Sep 16 14:45:27.548528 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.548434    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2034a38fecb1737b8e540f82e0c8a665fff7ac7e8b7f6190e50b4f67b38f1d71"} err="failed to get container status \"2034a38fecb1737b8e540f82e0c8a665fff7ac7e8b7f6190e50b4f67b38f1d71\": rpc error: code = NotFound desc = could not find container \"2034a38fecb1737b8e540f82e0c8a665fff7ac7e8b7f6190e50b4f67b38f1d71\": container with ID starting with 2034a38fecb1737b8e540f82e0c8a665fff7ac7e8b7f6190e50b4f67b38f1d71 not found: ID does not exist"
Sep 16 14:45:27.548599 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.548554    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1706bc4-9e9b-4767-84b2-d8e49111e048-model-cache" (OuterVolumeSpecName: "model-cache") pod "f1706bc4-9e9b-4767-84b2-d8e49111e048" (UID: "f1706bc4-9e9b-4767-84b2-d8e49111e048"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:45:27.548655 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.548632    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1706bc4-9e9b-4767-84b2-d8e49111e048-model-cache\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:45:27.548655 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.548646    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1706bc4-9e9b-4767-84b2-d8e49111e048-home\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:45:27.548655 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.548654    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1706bc4-9e9b-4767-84b2-d8e49111e048-tmp-dir\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:45:27.550359 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.550334    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1706bc4-9e9b-4767-84b2-d8e49111e048-dshm" (OuterVolumeSpecName: "dshm") pod "f1706bc4-9e9b-4767-84b2-d8e49111e048" (UID: "f1706bc4-9e9b-4767-84b2-d8e49111e048"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:45:27.550482 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.550408    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1706bc4-9e9b-4767-84b2-d8e49111e048-kube-api-access-65v4z" (OuterVolumeSpecName: "kube-api-access-65v4z") pod "f1706bc4-9e9b-4767-84b2-d8e49111e048" (UID: "f1706bc4-9e9b-4767-84b2-d8e49111e048"). InnerVolumeSpecName "kube-api-access-65v4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:45:27.550482 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.550436    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1706bc4-9e9b-4767-84b2-d8e49111e048-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f1706bc4-9e9b-4767-84b2-d8e49111e048" (UID: "f1706bc4-9e9b-4767-84b2-d8e49111e048"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:45:27.649725 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.649702    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1706bc4-9e9b-4767-84b2-d8e49111e048-dshm\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:45:27.649725 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.649723    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65v4z\" (UniqueName: \"kubernetes.io/projected/f1706bc4-9e9b-4767-84b2-d8e49111e048-kube-api-access-65v4z\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:45:27.649870 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.649734    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1706bc4-9e9b-4767-84b2-d8e49111e048-tls-certs\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:45:27.859139 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.859105    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"]
Sep 16 14:45:27.860820 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.860799    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-055ce3/rate-test-model-055ce3-kserve-656fc44487-ljb99"]
Sep 16 14:45:27.903296 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:27.903263    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1706bc4-9e9b-4767-84b2-d8e49111e048" path="/var/lib/kubelet/pods/f1706bc4-9e9b-4767-84b2-d8e49111e048/volumes"
Sep 16 14:45:30.501040 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:30.501011    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn"
Sep 16 14:45:48.294133 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:48.294059    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn"]
Sep 16 14:45:48.294627 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:48.294502    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1706bc4-9e9b-4767-84b2-d8e49111e048" containerName="main"
Sep 16 14:45:48.294627 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:48.294517    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1706bc4-9e9b-4767-84b2-d8e49111e048" containerName="main"
Sep 16 14:45:48.294627 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:48.294589    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1706bc4-9e9b-4767-84b2-d8e49111e048" containerName="main"
Sep 16 14:45:48.297762 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:48.297744    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn"
Sep 16 14:45:48.300768 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:48.300738    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3eaa3c\""
Sep 16 14:45:48.301018 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:48.301002    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3eaa3c-dockercfg-zfd4p\""
Sep 16 14:45:48.329862 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:48.329834    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pkll\" (UniqueName: \"kubernetes.io/projected/2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5-kube-api-access-9pkll\") pod \"payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn\" (UID: \"2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn"
Sep 16 14:45:48.330004 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:48.329878    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn\" (UID: \"2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn"
Sep 16 14:45:48.353352 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:48.353318    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b"]
Sep 16 14:45:48.356664 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:48.356648    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b"
Sep 16 14:45:48.367102 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:48.367076    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn"]
Sep 16 14:45:48.377427 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:48.377377    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b"]
Sep 16 14:45:48.430929 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:48.430889    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn\" (UID: \"2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn"
Sep 16 14:45:48.431101 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:48.430944    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r6g8\" (UniqueName: \"kubernetes.io/projected/eda20022-ec75-4082-b4df-c9f878c328f4-kube-api-access-4r6g8\") pod \"payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b\" (UID: \"eda20022-ec75-4082-b4df-c9f878c328f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b"
Sep 16 14:45:48.431101 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:48.430979    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eda20022-ec75-4082-b4df-c9f878c328f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b\" (UID: \"eda20022-ec75-4082-b4df-c9f878c328f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b"
Sep 16 14:45:48.431101 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:48.431014    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pkll\" (UniqueName: \"kubernetes.io/projected/2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5-kube-api-access-9pkll\") pod \"payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn\" (UID: \"2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn"
Sep 16 14:45:48.431704 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:48.431682    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn\" (UID: \"2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn"
Sep 16 14:45:48.446088 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:48.446060    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pkll\" (UniqueName: \"kubernetes.io/projected/2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5-kube-api-access-9pkll\") pod \"payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn\" (UID: \"2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn"
Sep 16 14:45:48.531897 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:48.531864    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r6g8\" (UniqueName: \"kubernetes.io/projected/eda20022-ec75-4082-b4df-c9f878c328f4-kube-api-access-4r6g8\") pod \"payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b\" (UID: \"eda20022-ec75-4082-b4df-c9f878c328f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b"
Sep 16 14:45:48.532068 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:48.531911    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eda20022-ec75-4082-b4df-c9f878c328f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b\" (UID: \"eda20022-ec75-4082-b4df-c9f878c328f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b"
Sep 16 14:45:48.532575 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:48.532556    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eda20022-ec75-4082-b4df-c9f878c328f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b\" (UID: \"eda20022-ec75-4082-b4df-c9f878c328f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b"
Sep 16 14:45:48.541848 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:48.541819    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r6g8\" (UniqueName: \"kubernetes.io/projected/eda20022-ec75-4082-b4df-c9f878c328f4-kube-api-access-4r6g8\") pod \"payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b\" (UID: \"eda20022-ec75-4082-b4df-c9f878c328f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b"
Sep 16 14:45:48.607963 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:48.607861    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn"
Sep 16 14:45:48.670044 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:48.670013    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b"
Sep 16 14:45:48.890438 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:48.890411    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn"]
Sep 16 14:45:48.892335 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:45:48.892302    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ee1bd44_79e7_41d6_ab3e_8b1f8edf02c5.slice/crio-d4eed3ed7e3ad366817a66a802c6af0be82d5b9fb9b15f17cb9af93ef1e84107 WatchSource:0}: Error finding container d4eed3ed7e3ad366817a66a802c6af0be82d5b9fb9b15f17cb9af93ef1e84107: Status 404 returned error can't find the container with id d4eed3ed7e3ad366817a66a802c6af0be82d5b9fb9b15f17cb9af93ef1e84107
Sep 16 14:45:48.958662 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:48.958631    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b"]
Sep 16 14:45:48.959222 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:45:48.959198    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda20022_ec75_4082_b4df_c9f878c328f4.slice/crio-92ea26b137581efd2ff8d0a508f8270503ed4d4c31eb246dca9c54f2ed27693f WatchSource:0}: Error finding container 92ea26b137581efd2ff8d0a508f8270503ed4d4c31eb246dca9c54f2ed27693f: Status 404 returned error can't find the container with id 92ea26b137581efd2ff8d0a508f8270503ed4d4c31eb246dca9c54f2ed27693f
Sep 16 14:45:49.671238 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:49.671197    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b" event={"ID":"eda20022-ec75-4082-b4df-c9f878c328f4","Type":"ContainerStarted","Data":"c2312ee46693bd6d58ef65d2b3c01f1856f498a062978cdeedf7e2b00e20c42a"}
Sep 16 14:45:49.671687 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:49.671263    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b" event={"ID":"eda20022-ec75-4082-b4df-c9f878c328f4","Type":"ContainerStarted","Data":"92ea26b137581efd2ff8d0a508f8270503ed4d4c31eb246dca9c54f2ed27693f"}
Sep 16 14:45:49.671687 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:49.671376    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b"
Sep 16 14:45:49.672894 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:49.672871    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn" event={"ID":"2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5","Type":"ContainerStarted","Data":"e137ce917940e59ad3584b134a33715e4ac13a93bc63a843c88796991e07b247"}
Sep 16 14:45:49.673017 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:49.672912    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn" event={"ID":"2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5","Type":"ContainerStarted","Data":"d4eed3ed7e3ad366817a66a802c6af0be82d5b9fb9b15f17cb9af93ef1e84107"}
Sep 16 14:45:49.673017 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:49.672959    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn"
Sep 16 14:45:49.701304 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:49.701251    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b" podStartSLOduration=1.7012392570000001 podStartE2EDuration="1.701239257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:45:49.699755535 +0000 UTC m=+1898.303380566" watchObservedRunningTime="2026-09-16 14:45:49.701239257 +0000 UTC m=+1898.304864288"
Sep 16 14:45:49.731580 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:45:49.725502    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn" podStartSLOduration=1.725479387 podStartE2EDuration="1.725479387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:45:49.723614612 +0000 UTC m=+1898.327239645" watchObservedRunningTime="2026-09-16 14:45:49.725479387 +0000 UTC m=+1898.329104421"
Sep 16 14:46:00.680963 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:00.680930    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn"
Sep 16 14:46:00.681448 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:00.680999    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b"
Sep 16 14:46:12.326217 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:12.326183    2587 scope.go:117] "RemoveContainer" containerID="39e858dc5f29b964dc80feeaf30e795159b22501d463c96a6f08ed9f059fceb0"
Sep 16 14:46:15.467640 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:15.467613    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b_eda20022-ec75-4082-b4df-c9f878c328f4/payload-processing/0.log"
Sep 16 14:46:26.520121 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:26.520085    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"]
Sep 16 14:46:26.523834 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:26.523817    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"
Sep 16 14:46:26.526440 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:26.526419    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a70ddb\"/\"openshift-service-ca.crt\""
Sep 16 14:46:26.527441 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:26.527417    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a70ddb\"/\"ipp-route-a70ddb-kserve-self-signed-certs\""
Sep 16 14:46:26.527582 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:26.527443    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a70ddb\"/\"default-dockercfg-nsdxb\""
Sep 16 14:46:26.527582 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:26.527491    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a70ddb\"/\"kube-root-ca.crt\""
Sep 16 14:46:26.535101 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:26.535080    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"]
Sep 16 14:46:26.568574 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:26.568548    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e66be06-03c1-447b-ab9f-68c2f59868a2-dshm\") pod \"ipp-route-a70ddb-kserve-5df89dc579-zw6kj\" (UID: \"2e66be06-03c1-447b-ab9f-68c2f59868a2\") " pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"
Sep 16 14:46:26.568824 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:26.568797    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e66be06-03c1-447b-ab9f-68c2f59868a2-tls-certs\") pod \"ipp-route-a70ddb-kserve-5df89dc579-zw6kj\" (UID: \"2e66be06-03c1-447b-ab9f-68c2f59868a2\") " pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"
Sep 16 14:46:26.568947 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:26.568905    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e66be06-03c1-447b-ab9f-68c2f59868a2-home\") pod \"ipp-route-a70ddb-kserve-5df89dc579-zw6kj\" (UID: \"2e66be06-03c1-447b-ab9f-68c2f59868a2\") " pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"
Sep 16 14:46:26.569018 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:26.568938    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e66be06-03c1-447b-ab9f-68c2f59868a2-tmp-dir\") pod \"ipp-route-a70ddb-kserve-5df89dc579-zw6kj\" (UID: \"2e66be06-03c1-447b-ab9f-68c2f59868a2\") " pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"
Sep 16 14:46:26.569084 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:26.569014    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rwtt\" (UniqueName: \"kubernetes.io/projected/2e66be06-03c1-447b-ab9f-68c2f59868a2-kube-api-access-2rwtt\") pod \"ipp-route-a70ddb-kserve-5df89dc579-zw6kj\" (UID: \"2e66be06-03c1-447b-ab9f-68c2f59868a2\") " pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"
Sep 16 14:46:26.569084 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:26.569063    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e66be06-03c1-447b-ab9f-68c2f59868a2-model-cache\") pod \"ipp-route-a70ddb-kserve-5df89dc579-zw6kj\" (UID: \"2e66be06-03c1-447b-ab9f-68c2f59868a2\") " pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"
Sep 16 14:46:26.670017 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:26.669989    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e66be06-03c1-447b-ab9f-68c2f59868a2-home\") pod \"ipp-route-a70ddb-kserve-5df89dc579-zw6kj\" (UID: \"2e66be06-03c1-447b-ab9f-68c2f59868a2\") " pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"
Sep 16 14:46:26.670017 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:26.670018    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e66be06-03c1-447b-ab9f-68c2f59868a2-tmp-dir\") pod \"ipp-route-a70ddb-kserve-5df89dc579-zw6kj\" (UID: \"2e66be06-03c1-447b-ab9f-68c2f59868a2\") " pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"
Sep 16 14:46:26.670222 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:26.670046    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rwtt\" (UniqueName: \"kubernetes.io/projected/2e66be06-03c1-447b-ab9f-68c2f59868a2-kube-api-access-2rwtt\") pod \"ipp-route-a70ddb-kserve-5df89dc579-zw6kj\" (UID: \"2e66be06-03c1-447b-ab9f-68c2f59868a2\") " pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"
Sep 16 14:46:26.670222 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:26.670072    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e66be06-03c1-447b-ab9f-68c2f59868a2-model-cache\") pod \"ipp-route-a70ddb-kserve-5df89dc579-zw6kj\" (UID: \"2e66be06-03c1-447b-ab9f-68c2f59868a2\") " pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"
Sep 16 14:46:26.670222 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:26.670197    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e66be06-03c1-447b-ab9f-68c2f59868a2-dshm\") pod \"ipp-route-a70ddb-kserve-5df89dc579-zw6kj\" (UID: \"2e66be06-03c1-447b-ab9f-68c2f59868a2\") " pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"
Sep 16 14:46:26.670368 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:26.670230    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e66be06-03c1-447b-ab9f-68c2f59868a2-tls-certs\") pod \"ipp-route-a70ddb-kserve-5df89dc579-zw6kj\" (UID: \"2e66be06-03c1-447b-ab9f-68c2f59868a2\") " pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"
Sep 16 14:46:26.670464 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:26.670431    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e66be06-03c1-447b-ab9f-68c2f59868a2-model-cache\") pod \"ipp-route-a70ddb-kserve-5df89dc579-zw6kj\" (UID: \"2e66be06-03c1-447b-ab9f-68c2f59868a2\") " pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"
Sep 16 14:46:26.670516 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:26.670485    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e66be06-03c1-447b-ab9f-68c2f59868a2-tmp-dir\") pod \"ipp-route-a70ddb-kserve-5df89dc579-zw6kj\" (UID: \"2e66be06-03c1-447b-ab9f-68c2f59868a2\") " pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"
Sep 16 14:46:26.670552 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:26.670510    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e66be06-03c1-447b-ab9f-68c2f59868a2-home\") pod \"ipp-route-a70ddb-kserve-5df89dc579-zw6kj\" (UID: \"2e66be06-03c1-447b-ab9f-68c2f59868a2\") " pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"
Sep 16 14:46:26.672634 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:26.672609    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e66be06-03c1-447b-ab9f-68c2f59868a2-dshm\") pod \"ipp-route-a70ddb-kserve-5df89dc579-zw6kj\" (UID: \"2e66be06-03c1-447b-ab9f-68c2f59868a2\") " pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"
Sep 16 14:46:26.672818 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:26.672798    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e66be06-03c1-447b-ab9f-68c2f59868a2-tls-certs\") pod \"ipp-route-a70ddb-kserve-5df89dc579-zw6kj\" (UID: \"2e66be06-03c1-447b-ab9f-68c2f59868a2\") " pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"
Sep 16 14:46:26.678833 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:26.678812    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rwtt\" (UniqueName: \"kubernetes.io/projected/2e66be06-03c1-447b-ab9f-68c2f59868a2-kube-api-access-2rwtt\") pod \"ipp-route-a70ddb-kserve-5df89dc579-zw6kj\" (UID: \"2e66be06-03c1-447b-ab9f-68c2f59868a2\") " pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"
Sep 16 14:46:26.832656 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:26.832576    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"
Sep 16 14:46:27.105591 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:27.105565    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"]
Sep 16 14:46:27.107282 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:46:27.107257    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e66be06_03c1_447b_ab9f_68c2f59868a2.slice/crio-9cc8ae55bb1181579b23cf8e218df1e6939da5f02b3197c66aa10306320a1de1 WatchSource:0}: Error finding container 9cc8ae55bb1181579b23cf8e218df1e6939da5f02b3197c66aa10306320a1de1: Status 404 returned error can't find the container with id 9cc8ae55bb1181579b23cf8e218df1e6939da5f02b3197c66aa10306320a1de1
Sep 16 14:46:27.868093 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:27.868051    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj" event={"ID":"2e66be06-03c1-447b-ab9f-68c2f59868a2","Type":"ContainerStarted","Data":"09d21e89304c549a9861f0b5933d21bc708744632ee4af8cd8b27d2827c6743f"}
Sep 16 14:46:27.868501 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:27.868104    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj" event={"ID":"2e66be06-03c1-447b-ab9f-68c2f59868a2","Type":"ContainerStarted","Data":"9cc8ae55bb1181579b23cf8e218df1e6939da5f02b3197c66aa10306320a1de1"}
Sep 16 14:46:27.887345 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:27.887293    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj" podStartSLOduration=1.887279291 podStartE2EDuration="1.887279291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:46:27.884684939 +0000 UTC m=+1936.488309969" watchObservedRunningTime="2026-09-16 14:46:27.887279291 +0000 UTC m=+1936.490904323"
Sep 16 14:46:36.833486 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:36.833449    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"
Sep 16 14:46:36.833486 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:36.833493    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"
Sep 16 14:46:36.846162 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:36.846128    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"
Sep 16 14:46:36.930982 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:36.930954    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"
Sep 16 14:46:51.317066 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:51.317030    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn_b5b73e81-38bb-40e0-9c1b-6298fbc42720/payload-processing/0.log"
Sep 16 14:46:51.591907 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:46:51.591833    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b_eda20022-ec75-4082-b4df-c9f878c328f4/payload-processing/0.log"
Sep 16 14:47:12.353044 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:12.353012    2587 scope.go:117] "RemoveContainer" containerID="dc938f882afa65bd67023abc8896b344107ab5a0ae310d467b5bbfeed22dfb78"
Sep 16 14:47:26.480029 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:26.479993    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a26a50-mqrnc"]
Sep 16 14:47:26.483523 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:26.483503    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a26a50-mqrnc"
Sep 16 14:47:26.486027 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:26.486003    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4jf\""
Sep 16 14:47:26.486137 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:26.486012    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:47:26.491213 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:26.491189    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a26a50-mqrnc"]
Sep 16 14:47:26.613831 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:26.613794    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ddcea1-a8b9-4142-872f-e78456e49550-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-a26a50-mqrnc\" (UID: \"b0ddcea1-a8b9-4142-872f-e78456e49550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a26a50-mqrnc"
Sep 16 14:47:26.714931 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:26.714903    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ddcea1-a8b9-4142-872f-e78456e49550-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-a26a50-mqrnc\" (UID: \"b0ddcea1-a8b9-4142-872f-e78456e49550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a26a50-mqrnc"
Sep 16 14:47:26.715521 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:26.715501    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ddcea1-a8b9-4142-872f-e78456e49550-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-a26a50-mqrnc\" (UID: \"b0ddcea1-a8b9-4142-872f-e78456e49550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a26a50-mqrnc"
Sep 16 14:47:26.795605 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:26.795531    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a26a50-mqrnc"
Sep 16 14:47:27.074624 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:27.074532    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a26a50-mqrnc"]
Sep 16 14:47:27.081769 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:47:27.081736    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ddcea1_a8b9_4142_872f_e78456e49550.slice/crio-e1ce9c9aefaafe1c682c10816ded73ea6aa35fc7c152474f9adcdc9e0f3aa8af WatchSource:0}: Error finding container e1ce9c9aefaafe1c682c10816ded73ea6aa35fc7c152474f9adcdc9e0f3aa8af: Status 404 returned error can't find the container with id e1ce9c9aefaafe1c682c10816ded73ea6aa35fc7c152474f9adcdc9e0f3aa8af
Sep 16 14:47:27.173164 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:27.173136    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a26a50-mqrnc" event={"ID":"b0ddcea1-a8b9-4142-872f-e78456e49550","Type":"ContainerStarted","Data":"e1ce9c9aefaafe1c682c10816ded73ea6aa35fc7c152474f9adcdc9e0f3aa8af"}
Sep 16 14:47:28.179806 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:28.179763    2587 generic.go:356] "Generic (PLEG): container finished" podID="b0ddcea1-a8b9-4142-872f-e78456e49550" containerID="82d4d82ebf966aff2ce8986de6d896d2668209ca2ea8c16eb94e0b62330e9996" exitCode=0
Sep 16 14:47:28.180172 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:28.179826    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a26a50-mqrnc" event={"ID":"b0ddcea1-a8b9-4142-872f-e78456e49550","Type":"ContainerDied","Data":"82d4d82ebf966aff2ce8986de6d896d2668209ca2ea8c16eb94e0b62330e9996"}
Sep 16 14:47:29.325258 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:29.325233    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a26a50-mqrnc"
Sep 16 14:47:29.440605 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:29.440512    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ddcea1-a8b9-4142-872f-e78456e49550-maas-api-service-ca\") pod \"b0ddcea1-a8b9-4142-872f-e78456e49550\" (UID: \"b0ddcea1-a8b9-4142-872f-e78456e49550\") "
Sep 16 14:47:29.440868 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:29.440847    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ddcea1-a8b9-4142-872f-e78456e49550-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b0ddcea1-a8b9-4142-872f-e78456e49550" (UID: "b0ddcea1-a8b9-4142-872f-e78456e49550"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:47:29.541748 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:29.541720    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ddcea1-a8b9-4142-872f-e78456e49550-maas-api-service-ca\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:47:30.192313 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:30.192235    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a26a50-mqrnc" event={"ID":"b0ddcea1-a8b9-4142-872f-e78456e49550","Type":"ContainerDied","Data":"e1ce9c9aefaafe1c682c10816ded73ea6aa35fc7c152474f9adcdc9e0f3aa8af"}
Sep 16 14:47:30.192313 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:30.192263    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a26a50-mqrnc"
Sep 16 14:47:30.192313 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:30.192286    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1ce9c9aefaafe1c682c10816ded73ea6aa35fc7c152474f9adcdc9e0f3aa8af"
Sep 16 14:47:37.502344 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:37.502298    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a26a50-mqrnc"]
Sep 16 14:47:37.505383 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:37.505356    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a26a50-mqrnc"]
Sep 16 14:47:37.903724 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:37.903691    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0ddcea1-a8b9-4142-872f-e78456e49550" path="/var/lib/kubelet/pods/b0ddcea1-a8b9-4142-872f-e78456e49550/volumes"
Sep 16 14:47:47.945046 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:47.945008    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a70ddb-j698z"]
Sep 16 14:47:47.945564 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:47.945534    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0ddcea1-a8b9-4142-872f-e78456e49550" containerName="revoke-keys"
Sep 16 14:47:47.945564 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:47.945550    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0ddcea1-a8b9-4142-872f-e78456e49550" containerName="revoke-keys"
Sep 16 14:47:47.945653 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:47.945630    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0ddcea1-a8b9-4142-872f-e78456e49550" containerName="revoke-keys"
Sep 16 14:47:47.948737 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:47.948720    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a70ddb-j698z"
Sep 16 14:47:47.951593 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:47.951571    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:47:47.951718 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:47.951639    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4jf\""
Sep 16 14:47:47.958888 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:47.958862    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a70ddb-j698z"]
Sep 16 14:47:48.100892 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:48.100851    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f373164b-0b7d-44fe-9e16-ec02c70f59df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a70ddb-j698z\" (UID: \"f373164b-0b7d-44fe-9e16-ec02c70f59df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a70ddb-j698z"
Sep 16 14:47:48.201770 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:48.201685    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f373164b-0b7d-44fe-9e16-ec02c70f59df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a70ddb-j698z\" (UID: \"f373164b-0b7d-44fe-9e16-ec02c70f59df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a70ddb-j698z"
Sep 16 14:47:48.202368 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:48.202340    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f373164b-0b7d-44fe-9e16-ec02c70f59df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a70ddb-j698z\" (UID: \"f373164b-0b7d-44fe-9e16-ec02c70f59df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a70ddb-j698z"
Sep 16 14:47:48.259822 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:48.259788    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a70ddb-j698z"
Sep 16 14:47:48.547441 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:48.547407    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a70ddb-j698z"]
Sep 16 14:47:48.550223 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:47:48.550196    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf373164b_0b7d_44fe_9e16_ec02c70f59df.slice/crio-03ca34cbcf705541d309517297360eced3acedd40839b15acee44c3b55216234 WatchSource:0}: Error finding container 03ca34cbcf705541d309517297360eced3acedd40839b15acee44c3b55216234: Status 404 returned error can't find the container with id 03ca34cbcf705541d309517297360eced3acedd40839b15acee44c3b55216234
Sep 16 14:47:49.290874 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:49.290838    2587 generic.go:356] "Generic (PLEG): container finished" podID="f373164b-0b7d-44fe-9e16-ec02c70f59df" containerID="71fe776dfbc4f1e696834692c4a242b901ae8b89361ba1873c0bc0542ceea97d" exitCode=0
Sep 16 14:47:49.291242 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:49.290891    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a70ddb-j698z" event={"ID":"f373164b-0b7d-44fe-9e16-ec02c70f59df","Type":"ContainerDied","Data":"71fe776dfbc4f1e696834692c4a242b901ae8b89361ba1873c0bc0542ceea97d"}
Sep 16 14:47:49.291242 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:49.290935    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a70ddb-j698z" event={"ID":"f373164b-0b7d-44fe-9e16-ec02c70f59df","Type":"ContainerStarted","Data":"03ca34cbcf705541d309517297360eced3acedd40839b15acee44c3b55216234"}
Sep 16 14:47:50.425633 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:50.425608    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a70ddb-j698z"
Sep 16 14:47:50.522502 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:50.522466    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f373164b-0b7d-44fe-9e16-ec02c70f59df-maas-api-service-ca\") pod \"f373164b-0b7d-44fe-9e16-ec02c70f59df\" (UID: \"f373164b-0b7d-44fe-9e16-ec02c70f59df\") "
Sep 16 14:47:50.522798 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:50.522774    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f373164b-0b7d-44fe-9e16-ec02c70f59df-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f373164b-0b7d-44fe-9e16-ec02c70f59df" (UID: "f373164b-0b7d-44fe-9e16-ec02c70f59df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:47:50.624090 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:50.624022    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f373164b-0b7d-44fe-9e16-ec02c70f59df-maas-api-service-ca\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:47:51.302814 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:51.302785    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a70ddb-j698z"
Sep 16 14:47:51.302982 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:51.302779    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a70ddb-j698z" event={"ID":"f373164b-0b7d-44fe-9e16-ec02c70f59df","Type":"ContainerDied","Data":"03ca34cbcf705541d309517297360eced3acedd40839b15acee44c3b55216234"}
Sep 16 14:47:51.302982 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:47:51.302904    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03ca34cbcf705541d309517297360eced3acedd40839b15acee44c3b55216234"
Sep 16 14:48:03.576621 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:03.576582    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn"]
Sep 16 14:48:03.577083 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:03.576832    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn" podUID="b5b73e81-38bb-40e0-9c1b-6298fbc42720" containerName="payload-processing" containerID="cri-o://5926ecb724f8ac86bbe3000f17a3c21eb46ee099a81158d5e2f6d0579e0aeb21" gracePeriod=30
Sep 16 14:48:03.757840 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:03.757807    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a70ddb-j698z"]
Sep 16 14:48:03.762774 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:03.762746    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a70ddb-j698z"]
Sep 16 14:48:03.832662 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:03.832599    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn"
Sep 16 14:48:03.903261 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:03.903223    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f373164b-0b7d-44fe-9e16-ec02c70f59df" path="/var/lib/kubelet/pods/f373164b-0b7d-44fe-9e16-ec02c70f59df/volumes"
Sep 16 14:48:03.946453 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:03.946423    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b73e81-38bb-40e0-9c1b-6298fbc42720-plugins-config-volume\") pod \"b5b73e81-38bb-40e0-9c1b-6298fbc42720\" (UID: \"b5b73e81-38bb-40e0-9c1b-6298fbc42720\") "
Sep 16 14:48:03.946645 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:03.946564    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzgjs\" (UniqueName: \"kubernetes.io/projected/b5b73e81-38bb-40e0-9c1b-6298fbc42720-kube-api-access-hzgjs\") pod \"b5b73e81-38bb-40e0-9c1b-6298fbc42720\" (UID: \"b5b73e81-38bb-40e0-9c1b-6298fbc42720\") "
Sep 16 14:48:03.946828 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:03.946804    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b73e81-38bb-40e0-9c1b-6298fbc42720-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b5b73e81-38bb-40e0-9c1b-6298fbc42720" (UID: "b5b73e81-38bb-40e0-9c1b-6298fbc42720"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:48:03.946984 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:03.946965    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b73e81-38bb-40e0-9c1b-6298fbc42720-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:48:03.948900 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:03.948875    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b73e81-38bb-40e0-9c1b-6298fbc42720-kube-api-access-hzgjs" (OuterVolumeSpecName: "kube-api-access-hzgjs") pod "b5b73e81-38bb-40e0-9c1b-6298fbc42720" (UID: "b5b73e81-38bb-40e0-9c1b-6298fbc42720"). InnerVolumeSpecName "kube-api-access-hzgjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:48:04.048318 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:04.048285    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzgjs\" (UniqueName: \"kubernetes.io/projected/b5b73e81-38bb-40e0-9c1b-6298fbc42720-kube-api-access-hzgjs\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:48:04.370958 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:04.370917    2587 generic.go:356] "Generic (PLEG): container finished" podID="b5b73e81-38bb-40e0-9c1b-6298fbc42720" containerID="5926ecb724f8ac86bbe3000f17a3c21eb46ee099a81158d5e2f6d0579e0aeb21" exitCode=0
Sep 16 14:48:04.371123 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:04.370983    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn"
Sep 16 14:48:04.371123 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:04.370996    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn" event={"ID":"b5b73e81-38bb-40e0-9c1b-6298fbc42720","Type":"ContainerDied","Data":"5926ecb724f8ac86bbe3000f17a3c21eb46ee099a81158d5e2f6d0579e0aeb21"}
Sep 16 14:48:04.371123 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:04.371053    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn" event={"ID":"b5b73e81-38bb-40e0-9c1b-6298fbc42720","Type":"ContainerDied","Data":"e6dbe04821706d69d4f299d9edf7c7840fcc66d388a724d0598de64bb60ef522"}
Sep 16 14:48:04.371123 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:04.371075    2587 scope.go:117] "RemoveContainer" containerID="5926ecb724f8ac86bbe3000f17a3c21eb46ee099a81158d5e2f6d0579e0aeb21"
Sep 16 14:48:04.380127 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:04.380109    2587 scope.go:117] "RemoveContainer" containerID="5926ecb724f8ac86bbe3000f17a3c21eb46ee099a81158d5e2f6d0579e0aeb21"
Sep 16 14:48:04.380478 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:48:04.380450    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5926ecb724f8ac86bbe3000f17a3c21eb46ee099a81158d5e2f6d0579e0aeb21\": container with ID starting with 5926ecb724f8ac86bbe3000f17a3c21eb46ee099a81158d5e2f6d0579e0aeb21 not found: ID does not exist" containerID="5926ecb724f8ac86bbe3000f17a3c21eb46ee099a81158d5e2f6d0579e0aeb21"
Sep 16 14:48:04.380593 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:04.380485    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5926ecb724f8ac86bbe3000f17a3c21eb46ee099a81158d5e2f6d0579e0aeb21"} err="failed to get container status \"5926ecb724f8ac86bbe3000f17a3c21eb46ee099a81158d5e2f6d0579e0aeb21\": rpc error: code = NotFound desc = could not find container \"5926ecb724f8ac86bbe3000f17a3c21eb46ee099a81158d5e2f6d0579e0aeb21\": container with ID starting with 5926ecb724f8ac86bbe3000f17a3c21eb46ee099a81158d5e2f6d0579e0aeb21 not found: ID does not exist"
Sep 16 14:48:04.394236 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:04.394214    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn"]
Sep 16 14:48:04.396288 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:04.396264    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a70ddb-6744fdcdbf-5z5jn"]
Sep 16 14:48:05.902904 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:05.902864    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5b73e81-38bb-40e0-9c1b-6298fbc42720" path="/var/lib/kubelet/pods/b5b73e81-38bb-40e0-9c1b-6298fbc42720/volumes"
Sep 16 14:48:09.161756 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:09.161724    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"]
Sep 16 14:48:12.374782 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:12.374755    2587 scope.go:117] "RemoveContainer" containerID="175ba647863178fcbb0fe7e07930c8d3cbb848b1821df99f294c8cceb081bdf2"
Sep 16 14:48:12.384471 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:12.384449    2587 scope.go:117] "RemoveContainer" containerID="5341bdf9548bc057c2908e962c8ac1edcbbc4cf9bb1e81c0072c7bbb6d844427"
Sep 16 14:48:12.393041 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:12.393026    2587 scope.go:117] "RemoveContainer" containerID="3d596ea3f82a3619a445e873d55743bfa1c684a9003daad445333d282ca2ccb4"
Sep 16 14:48:24.181681 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.181621    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj" podUID="2e66be06-03c1-447b-ab9f-68c2f59868a2" containerName="main" containerID="cri-o://09d21e89304c549a9861f0b5933d21bc708744632ee4af8cd8b27d2827c6743f" gracePeriod=45
Sep 16 14:48:24.438005 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.437945    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"
Sep 16 14:48:24.481752 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.481658    2587 generic.go:356] "Generic (PLEG): container finished" podID="2e66be06-03c1-447b-ab9f-68c2f59868a2" containerID="09d21e89304c549a9861f0b5933d21bc708744632ee4af8cd8b27d2827c6743f" exitCode=0
Sep 16 14:48:24.481752 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.481691    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj" event={"ID":"2e66be06-03c1-447b-ab9f-68c2f59868a2","Type":"ContainerDied","Data":"09d21e89304c549a9861f0b5933d21bc708744632ee4af8cd8b27d2827c6743f"}
Sep 16 14:48:24.481752 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.481716    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"
Sep 16 14:48:24.481752 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.481742    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj" event={"ID":"2e66be06-03c1-447b-ab9f-68c2f59868a2","Type":"ContainerDied","Data":"9cc8ae55bb1181579b23cf8e218df1e6939da5f02b3197c66aa10306320a1de1"}
Sep 16 14:48:24.482114 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.481764    2587 scope.go:117] "RemoveContainer" containerID="09d21e89304c549a9861f0b5933d21bc708744632ee4af8cd8b27d2827c6743f"
Sep 16 14:48:24.490559 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.490540    2587 scope.go:117] "RemoveContainer" containerID="09d21e89304c549a9861f0b5933d21bc708744632ee4af8cd8b27d2827c6743f"
Sep 16 14:48:24.490845 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:48:24.490825    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09d21e89304c549a9861f0b5933d21bc708744632ee4af8cd8b27d2827c6743f\": container with ID starting with 09d21e89304c549a9861f0b5933d21bc708744632ee4af8cd8b27d2827c6743f not found: ID does not exist" containerID="09d21e89304c549a9861f0b5933d21bc708744632ee4af8cd8b27d2827c6743f"
Sep 16 14:48:24.490906 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.490854    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09d21e89304c549a9861f0b5933d21bc708744632ee4af8cd8b27d2827c6743f"} err="failed to get container status \"09d21e89304c549a9861f0b5933d21bc708744632ee4af8cd8b27d2827c6743f\": rpc error: code = NotFound desc = could not find container \"09d21e89304c549a9861f0b5933d21bc708744632ee4af8cd8b27d2827c6743f\": container with ID starting with 09d21e89304c549a9861f0b5933d21bc708744632ee4af8cd8b27d2827c6743f not found: ID does not exist"
Sep 16 14:48:24.532234 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.532210    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e66be06-03c1-447b-ab9f-68c2f59868a2-model-cache\") pod \"2e66be06-03c1-447b-ab9f-68c2f59868a2\" (UID: \"2e66be06-03c1-447b-ab9f-68c2f59868a2\") "
Sep 16 14:48:24.532380 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.532242    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e66be06-03c1-447b-ab9f-68c2f59868a2-dshm\") pod \"2e66be06-03c1-447b-ab9f-68c2f59868a2\" (UID: \"2e66be06-03c1-447b-ab9f-68c2f59868a2\") "
Sep 16 14:48:24.532380 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.532260    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e66be06-03c1-447b-ab9f-68c2f59868a2-tls-certs\") pod \"2e66be06-03c1-447b-ab9f-68c2f59868a2\" (UID: \"2e66be06-03c1-447b-ab9f-68c2f59868a2\") "
Sep 16 14:48:24.532380 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.532291    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e66be06-03c1-447b-ab9f-68c2f59868a2-home\") pod \"2e66be06-03c1-447b-ab9f-68c2f59868a2\" (UID: \"2e66be06-03c1-447b-ab9f-68c2f59868a2\") "
Sep 16 14:48:24.532380 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.532314    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e66be06-03c1-447b-ab9f-68c2f59868a2-tmp-dir\") pod \"2e66be06-03c1-447b-ab9f-68c2f59868a2\" (UID: \"2e66be06-03c1-447b-ab9f-68c2f59868a2\") "
Sep 16 14:48:24.532380 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.532367    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rwtt\" (UniqueName: \"kubernetes.io/projected/2e66be06-03c1-447b-ab9f-68c2f59868a2-kube-api-access-2rwtt\") pod \"2e66be06-03c1-447b-ab9f-68c2f59868a2\" (UID: \"2e66be06-03c1-447b-ab9f-68c2f59868a2\") "
Sep 16 14:48:24.532684 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.532477    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e66be06-03c1-447b-ab9f-68c2f59868a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e66be06-03c1-447b-ab9f-68c2f59868a2" (UID: "2e66be06-03c1-447b-ab9f-68c2f59868a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:48:24.532684 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.532581    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e66be06-03c1-447b-ab9f-68c2f59868a2-home" (OuterVolumeSpecName: "home") pod "2e66be06-03c1-447b-ab9f-68c2f59868a2" (UID: "2e66be06-03c1-447b-ab9f-68c2f59868a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:48:24.532684 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.532619    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e66be06-03c1-447b-ab9f-68c2f59868a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2e66be06-03c1-447b-ab9f-68c2f59868a2" (UID: "2e66be06-03c1-447b-ab9f-68c2f59868a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:48:24.532792 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.532775    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e66be06-03c1-447b-ab9f-68c2f59868a2-home\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:48:24.532827 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.532792    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e66be06-03c1-447b-ab9f-68c2f59868a2-tmp-dir\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:48:24.532827 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.532804    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e66be06-03c1-447b-ab9f-68c2f59868a2-model-cache\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:48:24.534751 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.534728    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e66be06-03c1-447b-ab9f-68c2f59868a2-kube-api-access-2rwtt" (OuterVolumeSpecName: "kube-api-access-2rwtt") pod "2e66be06-03c1-447b-ab9f-68c2f59868a2" (UID: "2e66be06-03c1-447b-ab9f-68c2f59868a2"). InnerVolumeSpecName "kube-api-access-2rwtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:48:24.534896 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.534879    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e66be06-03c1-447b-ab9f-68c2f59868a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e66be06-03c1-447b-ab9f-68c2f59868a2" (UID: "2e66be06-03c1-447b-ab9f-68c2f59868a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:48:24.534896 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.534880    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e66be06-03c1-447b-ab9f-68c2f59868a2-dshm" (OuterVolumeSpecName: "dshm") pod "2e66be06-03c1-447b-ab9f-68c2f59868a2" (UID: "2e66be06-03c1-447b-ab9f-68c2f59868a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:48:24.634177 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.634127    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e66be06-03c1-447b-ab9f-68c2f59868a2-dshm\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:48:24.634177 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.634173    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e66be06-03c1-447b-ab9f-68c2f59868a2-tls-certs\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:48:24.634177 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.634184    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rwtt\" (UniqueName: \"kubernetes.io/projected/2e66be06-03c1-447b-ab9f-68c2f59868a2-kube-api-access-2rwtt\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:48:24.803739 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.803707    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"]
Sep 16 14:48:24.807986 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:24.807961    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-a70ddb/ipp-route-a70ddb-kserve-5df89dc579-zw6kj"]
Sep 16 14:48:25.902496 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:25.902462    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e66be06-03c1-447b-ab9f-68c2f59868a2" path="/var/lib/kubelet/pods/2e66be06-03c1-447b-ab9f-68c2f59868a2/volumes"
Sep 16 14:48:52.667465 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:52.667369    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3eaa3c-826rg"]
Sep 16 14:48:52.667844 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:52.667827    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f373164b-0b7d-44fe-9e16-ec02c70f59df" containerName="revoke-keys"
Sep 16 14:48:52.667887 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:52.667847    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="f373164b-0b7d-44fe-9e16-ec02c70f59df" containerName="revoke-keys"
Sep 16 14:48:52.667887 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:52.667861    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e66be06-03c1-447b-ab9f-68c2f59868a2" containerName="main"
Sep 16 14:48:52.667887 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:52.667866    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e66be06-03c1-447b-ab9f-68c2f59868a2" containerName="main"
Sep 16 14:48:52.667989 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:52.667892    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5b73e81-38bb-40e0-9c1b-6298fbc42720" containerName="payload-processing"
Sep 16 14:48:52.667989 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:52.667899    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5b73e81-38bb-40e0-9c1b-6298fbc42720" containerName="payload-processing"
Sep 16 14:48:52.667989 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:52.667961    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5b73e81-38bb-40e0-9c1b-6298fbc42720" containerName="payload-processing"
Sep 16 14:48:52.667989 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:52.667970    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e66be06-03c1-447b-ab9f-68c2f59868a2" containerName="main"
Sep 16 14:48:52.667989 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:52.667976    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="f373164b-0b7d-44fe-9e16-ec02c70f59df" containerName="revoke-keys"
Sep 16 14:48:52.671062 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:52.671045    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3eaa3c-826rg"
Sep 16 14:48:52.674015 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:52.673993    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4jf\""
Sep 16 14:48:52.674124 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:52.673993    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:48:52.679886 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:52.679864    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3eaa3c-826rg"]
Sep 16 14:48:52.783068 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:52.783039    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/132d6485-d855-49d3-9a4f-ee06a7d59429-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3eaa3c-826rg\" (UID: \"132d6485-d855-49d3-9a4f-ee06a7d59429\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3eaa3c-826rg"
Sep 16 14:48:52.884340 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:52.884303    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/132d6485-d855-49d3-9a4f-ee06a7d59429-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3eaa3c-826rg\" (UID: \"132d6485-d855-49d3-9a4f-ee06a7d59429\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3eaa3c-826rg"
Sep 16 14:48:52.885009 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:52.884987    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/132d6485-d855-49d3-9a4f-ee06a7d59429-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3eaa3c-826rg\" (UID: \"132d6485-d855-49d3-9a4f-ee06a7d59429\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3eaa3c-826rg"
Sep 16 14:48:52.982162 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:52.982077    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3eaa3c-826rg"
Sep 16 14:48:53.256385 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:53.256335    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3eaa3c-826rg"]
Sep 16 14:48:53.258747 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:48:53.258721    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod132d6485_d855_49d3_9a4f_ee06a7d59429.slice/crio-7713e5d9094af1d0dbd87506ca49c5d4bee5608c2d820d07cd83285f4e27d117 WatchSource:0}: Error finding container 7713e5d9094af1d0dbd87506ca49c5d4bee5608c2d820d07cd83285f4e27d117: Status 404 returned error can't find the container with id 7713e5d9094af1d0dbd87506ca49c5d4bee5608c2d820d07cd83285f4e27d117
Sep 16 14:48:53.632878 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:53.632843    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3eaa3c-826rg" event={"ID":"132d6485-d855-49d3-9a4f-ee06a7d59429","Type":"ContainerStarted","Data":"7006c7967751f0d58741cfe428bdab3130cab28a892b3e55cb7702dbc4381cb9"}
Sep 16 14:48:53.633038 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:53.632897    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3eaa3c-826rg" event={"ID":"132d6485-d855-49d3-9a4f-ee06a7d59429","Type":"ContainerStarted","Data":"7713e5d9094af1d0dbd87506ca49c5d4bee5608c2d820d07cd83285f4e27d117"}
Sep 16 14:48:53.652561 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:53.652514    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3eaa3c-826rg" podStartSLOduration=1.652497952 podStartE2EDuration="1.652497952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:48:53.651486717 +0000 UTC m=+2082.255111750" watchObservedRunningTime="2026-09-16 14:48:53.652497952 +0000 UTC m=+2082.256122983"
Sep 16 14:48:54.639767 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:54.639732    2587 generic.go:356] "Generic (PLEG): container finished" podID="132d6485-d855-49d3-9a4f-ee06a7d59429" containerID="7006c7967751f0d58741cfe428bdab3130cab28a892b3e55cb7702dbc4381cb9" exitCode=0
Sep 16 14:48:54.640187 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:54.639820    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3eaa3c-826rg" event={"ID":"132d6485-d855-49d3-9a4f-ee06a7d59429","Type":"ContainerDied","Data":"7006c7967751f0d58741cfe428bdab3130cab28a892b3e55cb7702dbc4381cb9"}
Sep 16 14:48:55.780532 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:55.780511    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3eaa3c-826rg"
Sep 16 14:48:55.914014 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:55.913987    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/132d6485-d855-49d3-9a4f-ee06a7d59429-maas-api-service-ca\") pod \"132d6485-d855-49d3-9a4f-ee06a7d59429\" (UID: \"132d6485-d855-49d3-9a4f-ee06a7d59429\") "
Sep 16 14:48:55.914364 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:55.914337    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/132d6485-d855-49d3-9a4f-ee06a7d59429-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "132d6485-d855-49d3-9a4f-ee06a7d59429" (UID: "132d6485-d855-49d3-9a4f-ee06a7d59429"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:48:56.014915 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:56.014882    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/132d6485-d855-49d3-9a4f-ee06a7d59429-maas-api-service-ca\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:48:56.651608 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:56.651578    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3eaa3c-826rg"
Sep 16 14:48:56.651608 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:56.651582    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3eaa3c-826rg" event={"ID":"132d6485-d855-49d3-9a4f-ee06a7d59429","Type":"ContainerDied","Data":"7713e5d9094af1d0dbd87506ca49c5d4bee5608c2d820d07cd83285f4e27d117"}
Sep 16 14:48:56.651805 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:48:56.651631    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7713e5d9094af1d0dbd87506ca49c5d4bee5608c2d820d07cd83285f4e27d117"
Sep 16 14:49:03.233897 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.233857    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b"]
Sep 16 14:49:03.234891 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.234226    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b" podUID="eda20022-ec75-4082-b4df-c9f878c328f4" containerName="payload-processing" containerID="cri-o://c2312ee46693bd6d58ef65d2b3c01f1856f498a062978cdeedf7e2b00e20c42a" gracePeriod=30
Sep 16 14:49:03.236883 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.236858    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn"]
Sep 16 14:49:03.237083 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.237061    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn" podUID="2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5" containerName="payload-pre-processing" containerID="cri-o://e137ce917940e59ad3584b134a33715e4ac13a93bc63a843c88796991e07b247" gracePeriod=30
Sep 16 14:49:03.419523 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.419429    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3eaa3c-826rg"]
Sep 16 14:49:03.422537 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.422505    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3eaa3c-826rg"]
Sep 16 14:49:03.498504 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.498437    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b"
Sep 16 14:49:03.521364 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.521339    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn"
Sep 16 14:49:03.584933 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.584900    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pkll\" (UniqueName: \"kubernetes.io/projected/2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5-kube-api-access-9pkll\") pod \"2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5\" (UID: \"2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5\") "
Sep 16 14:49:03.584933 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.584956    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5-plugins-config-volume\") pod \"2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5\" (UID: \"2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5\") "
Sep 16 14:49:03.585188 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.585015    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4r6g8\" (UniqueName: \"kubernetes.io/projected/eda20022-ec75-4082-b4df-c9f878c328f4-kube-api-access-4r6g8\") pod \"eda20022-ec75-4082-b4df-c9f878c328f4\" (UID: \"eda20022-ec75-4082-b4df-c9f878c328f4\") "
Sep 16 14:49:03.585188 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.585055    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eda20022-ec75-4082-b4df-c9f878c328f4-plugins-config-volume\") pod \"eda20022-ec75-4082-b4df-c9f878c328f4\" (UID: \"eda20022-ec75-4082-b4df-c9f878c328f4\") "
Sep 16 14:49:03.585443 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.585378    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5" (UID: "2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:49:03.585686 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.585663    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eda20022-ec75-4082-b4df-c9f878c328f4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eda20022-ec75-4082-b4df-c9f878c328f4" (UID: "eda20022-ec75-4082-b4df-c9f878c328f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:49:03.587955 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.587928    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda20022-ec75-4082-b4df-c9f878c328f4-kube-api-access-4r6g8" (OuterVolumeSpecName: "kube-api-access-4r6g8") pod "eda20022-ec75-4082-b4df-c9f878c328f4" (UID: "eda20022-ec75-4082-b4df-c9f878c328f4"). InnerVolumeSpecName "kube-api-access-4r6g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:49:03.587955 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.587954    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5-kube-api-access-9pkll" (OuterVolumeSpecName: "kube-api-access-9pkll") pod "2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5" (UID: "2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5"). InnerVolumeSpecName "kube-api-access-9pkll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:49:03.685761 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.685733    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pkll\" (UniqueName: \"kubernetes.io/projected/2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5-kube-api-access-9pkll\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:49:03.685761 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.685757    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:49:03.685918 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.685768    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r6g8\" (UniqueName: \"kubernetes.io/projected/eda20022-ec75-4082-b4df-c9f878c328f4-kube-api-access-4r6g8\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:49:03.685918 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.685777    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eda20022-ec75-4082-b4df-c9f878c328f4-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:49:03.688826 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.688798    2587 generic.go:356] "Generic (PLEG): container finished" podID="eda20022-ec75-4082-b4df-c9f878c328f4" containerID="c2312ee46693bd6d58ef65d2b3c01f1856f498a062978cdeedf7e2b00e20c42a" exitCode=0
Sep 16 14:49:03.688954 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.688845    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b"
Sep 16 14:49:03.688954 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.688886    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b" event={"ID":"eda20022-ec75-4082-b4df-c9f878c328f4","Type":"ContainerDied","Data":"c2312ee46693bd6d58ef65d2b3c01f1856f498a062978cdeedf7e2b00e20c42a"}
Sep 16 14:49:03.688954 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.688941    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b" event={"ID":"eda20022-ec75-4082-b4df-c9f878c328f4","Type":"ContainerDied","Data":"92ea26b137581efd2ff8d0a508f8270503ed4d4c31eb246dca9c54f2ed27693f"}
Sep 16 14:49:03.689128 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.688969    2587 scope.go:117] "RemoveContainer" containerID="c2312ee46693bd6d58ef65d2b3c01f1856f498a062978cdeedf7e2b00e20c42a"
Sep 16 14:49:03.690281 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.690253    2587 generic.go:356] "Generic (PLEG): container finished" podID="2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5" containerID="e137ce917940e59ad3584b134a33715e4ac13a93bc63a843c88796991e07b247" exitCode=0
Sep 16 14:49:03.690424 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.690281    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn" event={"ID":"2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5","Type":"ContainerDied","Data":"e137ce917940e59ad3584b134a33715e4ac13a93bc63a843c88796991e07b247"}
Sep 16 14:49:03.690424 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.690308    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn"
Sep 16 14:49:03.690424 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.690333    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn" event={"ID":"2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5","Type":"ContainerDied","Data":"d4eed3ed7e3ad366817a66a802c6af0be82d5b9fb9b15f17cb9af93ef1e84107"}
Sep 16 14:49:03.701766 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.699947    2587 scope.go:117] "RemoveContainer" containerID="c2312ee46693bd6d58ef65d2b3c01f1856f498a062978cdeedf7e2b00e20c42a"
Sep 16 14:49:03.701979 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:49:03.701836    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2312ee46693bd6d58ef65d2b3c01f1856f498a062978cdeedf7e2b00e20c42a\": container with ID starting with c2312ee46693bd6d58ef65d2b3c01f1856f498a062978cdeedf7e2b00e20c42a not found: ID does not exist" containerID="c2312ee46693bd6d58ef65d2b3c01f1856f498a062978cdeedf7e2b00e20c42a"
Sep 16 14:49:03.701979 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.701868    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2312ee46693bd6d58ef65d2b3c01f1856f498a062978cdeedf7e2b00e20c42a"} err="failed to get container status \"c2312ee46693bd6d58ef65d2b3c01f1856f498a062978cdeedf7e2b00e20c42a\": rpc error: code = NotFound desc = could not find container \"c2312ee46693bd6d58ef65d2b3c01f1856f498a062978cdeedf7e2b00e20c42a\": container with ID starting with c2312ee46693bd6d58ef65d2b3c01f1856f498a062978cdeedf7e2b00e20c42a not found: ID does not exist"
Sep 16 14:49:03.701979 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.701892    2587 scope.go:117] "RemoveContainer" containerID="e137ce917940e59ad3584b134a33715e4ac13a93bc63a843c88796991e07b247"
Sep 16 14:49:03.709984 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.709970    2587 scope.go:117] "RemoveContainer" containerID="e137ce917940e59ad3584b134a33715e4ac13a93bc63a843c88796991e07b247"
Sep 16 14:49:03.710248 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:49:03.710230    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e137ce917940e59ad3584b134a33715e4ac13a93bc63a843c88796991e07b247\": container with ID starting with e137ce917940e59ad3584b134a33715e4ac13a93bc63a843c88796991e07b247 not found: ID does not exist" containerID="e137ce917940e59ad3584b134a33715e4ac13a93bc63a843c88796991e07b247"
Sep 16 14:49:03.710309 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.710251    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e137ce917940e59ad3584b134a33715e4ac13a93bc63a843c88796991e07b247"} err="failed to get container status \"e137ce917940e59ad3584b134a33715e4ac13a93bc63a843c88796991e07b247\": rpc error: code = NotFound desc = could not find container \"e137ce917940e59ad3584b134a33715e4ac13a93bc63a843c88796991e07b247\": container with ID starting with e137ce917940e59ad3584b134a33715e4ac13a93bc63a843c88796991e07b247 not found: ID does not exist"
Sep 16 14:49:03.716121 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.716100    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn"]
Sep 16 14:49:03.722972 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.722951    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3eaa3c-5b46784bcd-rhmjn"]
Sep 16 14:49:03.732506 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.732484    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b"]
Sep 16 14:49:03.735933 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.735906    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3eaa3c-5cf6485cf-fdw8b"]
Sep 16 14:49:03.902739 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.902708    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="132d6485-d855-49d3-9a4f-ee06a7d59429" path="/var/lib/kubelet/pods/132d6485-d855-49d3-9a4f-ee06a7d59429/volumes"
Sep 16 14:49:03.903040 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.903027    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5" path="/var/lib/kubelet/pods/2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5/volumes"
Sep 16 14:49:03.903348 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:03.903336    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eda20022-ec75-4082-b4df-c9f878c328f4" path="/var/lib/kubelet/pods/eda20022-ec75-4082-b4df-c9f878c328f4/volumes"
Sep 16 14:49:10.420087 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:10.420055    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f88e1-gc9mg"]
Sep 16 14:49:10.420571 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:10.420417    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5" containerName="payload-pre-processing"
Sep 16 14:49:10.420571 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:10.420428    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5" containerName="payload-pre-processing"
Sep 16 14:49:10.420571 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:10.420443    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="132d6485-d855-49d3-9a4f-ee06a7d59429" containerName="revoke-keys"
Sep 16 14:49:10.420571 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:10.420450    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="132d6485-d855-49d3-9a4f-ee06a7d59429" containerName="revoke-keys"
Sep 16 14:49:10.420571 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:10.420464    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eda20022-ec75-4082-b4df-c9f878c328f4" containerName="payload-processing"
Sep 16 14:49:10.420571 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:10.420470    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="eda20022-ec75-4082-b4df-c9f878c328f4" containerName="payload-processing"
Sep 16 14:49:10.420571 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:10.420524    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="eda20022-ec75-4082-b4df-c9f878c328f4" containerName="payload-processing"
Sep 16 14:49:10.420571 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:10.420533    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="132d6485-d855-49d3-9a4f-ee06a7d59429" containerName="revoke-keys"
Sep 16 14:49:10.420571 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:10.420539    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ee1bd44-79e7-41d6-ab3e-8b1f8edf02c5" containerName="payload-pre-processing"
Sep 16 14:49:10.424869 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:10.424847    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f88e1-gc9mg"
Sep 16 14:49:10.428337 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:10.428314    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4jf\""
Sep 16 14:49:10.428495 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:10.428335    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:49:10.431232 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:10.431198    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f88e1-gc9mg"]
Sep 16 14:49:10.540519 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:10.540487    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f3b1be3-2e40-480e-917f-be84ff47f3b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9f88e1-gc9mg\" (UID: \"5f3b1be3-2e40-480e-917f-be84ff47f3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f88e1-gc9mg"
Sep 16 14:49:10.641932 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:10.641898    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f3b1be3-2e40-480e-917f-be84ff47f3b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9f88e1-gc9mg\" (UID: \"5f3b1be3-2e40-480e-917f-be84ff47f3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f88e1-gc9mg"
Sep 16 14:49:10.642762 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:10.642740    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f3b1be3-2e40-480e-917f-be84ff47f3b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9f88e1-gc9mg\" (UID: \"5f3b1be3-2e40-480e-917f-be84ff47f3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f88e1-gc9mg"
Sep 16 14:49:10.738761 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:10.738681    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f88e1-gc9mg"
Sep 16 14:49:11.008448 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:11.008423    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f88e1-gc9mg"]
Sep 16 14:49:11.011571 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:49:11.011538    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f3b1be3_2e40_480e_917f_be84ff47f3b0.slice/crio-02a7de01cdd28db01556ca3767006840b878d705c7dbc0ad71e8c3eaf79cf5cb WatchSource:0}: Error finding container 02a7de01cdd28db01556ca3767006840b878d705c7dbc0ad71e8c3eaf79cf5cb: Status 404 returned error can't find the container with id 02a7de01cdd28db01556ca3767006840b878d705c7dbc0ad71e8c3eaf79cf5cb
Sep 16 14:49:11.736314 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:11.736284    2587 generic.go:356] "Generic (PLEG): container finished" podID="5f3b1be3-2e40-480e-917f-be84ff47f3b0" containerID="5a5c98f7efd6564ff38c6a2bfb7654853609a32aff56defbd3fc4b760a1bfe68" exitCode=0
Sep 16 14:49:11.736648 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:11.736345    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f88e1-gc9mg" event={"ID":"5f3b1be3-2e40-480e-917f-be84ff47f3b0","Type":"ContainerDied","Data":"5a5c98f7efd6564ff38c6a2bfb7654853609a32aff56defbd3fc4b760a1bfe68"}
Sep 16 14:49:11.736648 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:11.736430    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f88e1-gc9mg" event={"ID":"5f3b1be3-2e40-480e-917f-be84ff47f3b0","Type":"ContainerStarted","Data":"02a7de01cdd28db01556ca3767006840b878d705c7dbc0ad71e8c3eaf79cf5cb"}
Sep 16 14:49:12.424897 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:12.424863    2587 scope.go:117] "RemoveContainer" containerID="fc38a98d7f9e638a8a24c9a9c5c534a51f0aa698b72a98becfe33ceec1567c14"
Sep 16 14:49:12.885284 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:12.885249    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f88e1-gc9mg"
Sep 16 14:49:12.963663 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:12.963636    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f3b1be3-2e40-480e-917f-be84ff47f3b0-maas-api-service-ca\") pod \"5f3b1be3-2e40-480e-917f-be84ff47f3b0\" (UID: \"5f3b1be3-2e40-480e-917f-be84ff47f3b0\") "
Sep 16 14:49:12.963999 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:12.963978    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f3b1be3-2e40-480e-917f-be84ff47f3b0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5f3b1be3-2e40-480e-917f-be84ff47f3b0" (UID: "5f3b1be3-2e40-480e-917f-be84ff47f3b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:49:13.065417 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:13.065302    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f3b1be3-2e40-480e-917f-be84ff47f3b0-maas-api-service-ca\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:49:13.747215 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:13.747186    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f88e1-gc9mg"
Sep 16 14:49:13.747381 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:13.747222    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f88e1-gc9mg" event={"ID":"5f3b1be3-2e40-480e-917f-be84ff47f3b0","Type":"ContainerDied","Data":"02a7de01cdd28db01556ca3767006840b878d705c7dbc0ad71e8c3eaf79cf5cb"}
Sep 16 14:49:13.747381 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:13.747268    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02a7de01cdd28db01556ca3767006840b878d705c7dbc0ad71e8c3eaf79cf5cb"
Sep 16 14:49:20.889089 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:20.889057    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6"]
Sep 16 14:49:20.889514 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:20.889277    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6" podUID="6d0b1b65-75e8-4e27-8fdd-0061cf297144" containerName="payload-processing" containerID="cri-o://5a057ac2ba9b84c3fb55a84551aa6dbf445cb7c9a860b8616b7cfa5e9d612725" gracePeriod=30
Sep 16 14:49:20.894964 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:20.894937    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj"]
Sep 16 14:49:20.895209 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:20.895168    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj" podUID="51fc9064-7c2f-4b29-80a2-bd9747098af1" containerName="payload-pre-processing" containerID="cri-o://13d8de68ab1d044925ffc747d90f8a60c17d5ecc5778fb035b19ea6bc73dd508" gracePeriod=30
Sep 16 14:49:21.045663 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.045627    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f88e1-gc9mg"]
Sep 16 14:49:21.048490 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.048463    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f88e1-gc9mg"]
Sep 16 14:49:21.172446 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.172425    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6"
Sep 16 14:49:21.174502 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.174484    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj"
Sep 16 14:49:21.336574 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.336544    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5s2kr\" (UniqueName: \"kubernetes.io/projected/51fc9064-7c2f-4b29-80a2-bd9747098af1-kube-api-access-5s2kr\") pod \"51fc9064-7c2f-4b29-80a2-bd9747098af1\" (UID: \"51fc9064-7c2f-4b29-80a2-bd9747098af1\") "
Sep 16 14:49:21.336754 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.336589    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51fc9064-7c2f-4b29-80a2-bd9747098af1-plugins-config-volume\") pod \"51fc9064-7c2f-4b29-80a2-bd9747098af1\" (UID: \"51fc9064-7c2f-4b29-80a2-bd9747098af1\") "
Sep 16 14:49:21.336754 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.336610    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d0b1b65-75e8-4e27-8fdd-0061cf297144-plugins-config-volume\") pod \"6d0b1b65-75e8-4e27-8fdd-0061cf297144\" (UID: \"6d0b1b65-75e8-4e27-8fdd-0061cf297144\") "
Sep 16 14:49:21.336865 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.336754    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pl9vt\" (UniqueName: \"kubernetes.io/projected/6d0b1b65-75e8-4e27-8fdd-0061cf297144-kube-api-access-pl9vt\") pod \"6d0b1b65-75e8-4e27-8fdd-0061cf297144\" (UID: \"6d0b1b65-75e8-4e27-8fdd-0061cf297144\") "
Sep 16 14:49:21.336978 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.336950    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d0b1b65-75e8-4e27-8fdd-0061cf297144-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6d0b1b65-75e8-4e27-8fdd-0061cf297144" (UID: "6d0b1b65-75e8-4e27-8fdd-0061cf297144"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:49:21.337097 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.336986    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51fc9064-7c2f-4b29-80a2-bd9747098af1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "51fc9064-7c2f-4b29-80a2-bd9747098af1" (UID: "51fc9064-7c2f-4b29-80a2-bd9747098af1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:49:21.337184 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.337166    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51fc9064-7c2f-4b29-80a2-bd9747098af1-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:49:21.337223 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.337193    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d0b1b65-75e8-4e27-8fdd-0061cf297144-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:49:21.338899 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.338874    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d0b1b65-75e8-4e27-8fdd-0061cf297144-kube-api-access-pl9vt" (OuterVolumeSpecName: "kube-api-access-pl9vt") pod "6d0b1b65-75e8-4e27-8fdd-0061cf297144" (UID: "6d0b1b65-75e8-4e27-8fdd-0061cf297144"). InnerVolumeSpecName "kube-api-access-pl9vt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:49:21.338899 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.338887    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51fc9064-7c2f-4b29-80a2-bd9747098af1-kube-api-access-5s2kr" (OuterVolumeSpecName: "kube-api-access-5s2kr") pod "51fc9064-7c2f-4b29-80a2-bd9747098af1" (UID: "51fc9064-7c2f-4b29-80a2-bd9747098af1"). InnerVolumeSpecName "kube-api-access-5s2kr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:49:21.437938 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.437913    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl9vt\" (UniqueName: \"kubernetes.io/projected/6d0b1b65-75e8-4e27-8fdd-0061cf297144-kube-api-access-pl9vt\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:49:21.437938 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.437942    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s2kr\" (UniqueName: \"kubernetes.io/projected/51fc9064-7c2f-4b29-80a2-bd9747098af1-kube-api-access-5s2kr\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:49:21.787625 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.787548    2587 generic.go:356] "Generic (PLEG): container finished" podID="6d0b1b65-75e8-4e27-8fdd-0061cf297144" containerID="5a057ac2ba9b84c3fb55a84551aa6dbf445cb7c9a860b8616b7cfa5e9d612725" exitCode=0
Sep 16 14:49:21.787625 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.787604    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6"
Sep 16 14:49:21.787846 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.787623    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6" event={"ID":"6d0b1b65-75e8-4e27-8fdd-0061cf297144","Type":"ContainerDied","Data":"5a057ac2ba9b84c3fb55a84551aa6dbf445cb7c9a860b8616b7cfa5e9d612725"}
Sep 16 14:49:21.787846 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.787676    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6" event={"ID":"6d0b1b65-75e8-4e27-8fdd-0061cf297144","Type":"ContainerDied","Data":"19d03fd830cebb8a70c9235c03f632e93c37ec075b85d5b99e2fbeb5e7f3220a"}
Sep 16 14:49:21.787846 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.787701    2587 scope.go:117] "RemoveContainer" containerID="5a057ac2ba9b84c3fb55a84551aa6dbf445cb7c9a860b8616b7cfa5e9d612725"
Sep 16 14:49:21.789079 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.789057    2587 generic.go:356] "Generic (PLEG): container finished" podID="51fc9064-7c2f-4b29-80a2-bd9747098af1" containerID="13d8de68ab1d044925ffc747d90f8a60c17d5ecc5778fb035b19ea6bc73dd508" exitCode=0
Sep 16 14:49:21.789159 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.789097    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj"
Sep 16 14:49:21.789201 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.789174    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj" event={"ID":"51fc9064-7c2f-4b29-80a2-bd9747098af1","Type":"ContainerDied","Data":"13d8de68ab1d044925ffc747d90f8a60c17d5ecc5778fb035b19ea6bc73dd508"}
Sep 16 14:49:21.789237 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.789211    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj" event={"ID":"51fc9064-7c2f-4b29-80a2-bd9747098af1","Type":"ContainerDied","Data":"370b9a770b7aa775a7e11cca7ceaef2b9a2662e4b8df06252569adf03c0ad7bc"}
Sep 16 14:49:21.797682 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.797662    2587 scope.go:117] "RemoveContainer" containerID="5a057ac2ba9b84c3fb55a84551aa6dbf445cb7c9a860b8616b7cfa5e9d612725"
Sep 16 14:49:21.797996 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:49:21.797976    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a057ac2ba9b84c3fb55a84551aa6dbf445cb7c9a860b8616b7cfa5e9d612725\": container with ID starting with 5a057ac2ba9b84c3fb55a84551aa6dbf445cb7c9a860b8616b7cfa5e9d612725 not found: ID does not exist" containerID="5a057ac2ba9b84c3fb55a84551aa6dbf445cb7c9a860b8616b7cfa5e9d612725"
Sep 16 14:49:21.798072 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.798005    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a057ac2ba9b84c3fb55a84551aa6dbf445cb7c9a860b8616b7cfa5e9d612725"} err="failed to get container status \"5a057ac2ba9b84c3fb55a84551aa6dbf445cb7c9a860b8616b7cfa5e9d612725\": rpc error: code = NotFound desc = could not find container \"5a057ac2ba9b84c3fb55a84551aa6dbf445cb7c9a860b8616b7cfa5e9d612725\": container with ID starting with 5a057ac2ba9b84c3fb55a84551aa6dbf445cb7c9a860b8616b7cfa5e9d612725 not found: ID does not exist"
Sep 16 14:49:21.798072 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.798021    2587 scope.go:117] "RemoveContainer" containerID="13d8de68ab1d044925ffc747d90f8a60c17d5ecc5778fb035b19ea6bc73dd508"
Sep 16 14:49:21.806373 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.806358    2587 scope.go:117] "RemoveContainer" containerID="13d8de68ab1d044925ffc747d90f8a60c17d5ecc5778fb035b19ea6bc73dd508"
Sep 16 14:49:21.806663 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:49:21.806643    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13d8de68ab1d044925ffc747d90f8a60c17d5ecc5778fb035b19ea6bc73dd508\": container with ID starting with 13d8de68ab1d044925ffc747d90f8a60c17d5ecc5778fb035b19ea6bc73dd508 not found: ID does not exist" containerID="13d8de68ab1d044925ffc747d90f8a60c17d5ecc5778fb035b19ea6bc73dd508"
Sep 16 14:49:21.806716 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.806671    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13d8de68ab1d044925ffc747d90f8a60c17d5ecc5778fb035b19ea6bc73dd508"} err="failed to get container status \"13d8de68ab1d044925ffc747d90f8a60c17d5ecc5778fb035b19ea6bc73dd508\": rpc error: code = NotFound desc = could not find container \"13d8de68ab1d044925ffc747d90f8a60c17d5ecc5778fb035b19ea6bc73dd508\": container with ID starting with 13d8de68ab1d044925ffc747d90f8a60c17d5ecc5778fb035b19ea6bc73dd508 not found: ID does not exist"
Sep 16 14:49:21.812797 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.812774    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj"]
Sep 16 14:49:21.818828 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.818806    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9f88e1-ffb9c457d-826jj"]
Sep 16 14:49:21.828171 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.828149    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6"]
Sep 16 14:49:21.832728 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.832705    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9f88e1-6f9487cbfc-pfxs6"]
Sep 16 14:49:21.902884 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.902854    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51fc9064-7c2f-4b29-80a2-bd9747098af1" path="/var/lib/kubelet/pods/51fc9064-7c2f-4b29-80a2-bd9747098af1/volumes"
Sep 16 14:49:21.903320 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.903302    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f3b1be3-2e40-480e-917f-be84ff47f3b0" path="/var/lib/kubelet/pods/5f3b1be3-2e40-480e-917f-be84ff47f3b0/volumes"
Sep 16 14:49:21.903800 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:21.903781    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d0b1b65-75e8-4e27-8fdd-0061cf297144" path="/var/lib/kubelet/pods/6d0b1b65-75e8-4e27-8fdd-0061cf297144/volumes"
Sep 16 14:49:28.775267 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:28.775231    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-055ce3-q6bts"]
Sep 16 14:49:28.775765 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:28.775606    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3b1be3-2e40-480e-917f-be84ff47f3b0" containerName="revoke-keys"
Sep 16 14:49:28.775765 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:28.775621    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3b1be3-2e40-480e-917f-be84ff47f3b0" containerName="revoke-keys"
Sep 16 14:49:28.775765 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:28.775631    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d0b1b65-75e8-4e27-8fdd-0061cf297144" containerName="payload-processing"
Sep 16 14:49:28.775765 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:28.775637    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d0b1b65-75e8-4e27-8fdd-0061cf297144" containerName="payload-processing"
Sep 16 14:49:28.775765 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:28.775654    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51fc9064-7c2f-4b29-80a2-bd9747098af1" containerName="payload-pre-processing"
Sep 16 14:49:28.775765 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:28.775660    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="51fc9064-7c2f-4b29-80a2-bd9747098af1" containerName="payload-pre-processing"
Sep 16 14:49:28.775765 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:28.775712    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="51fc9064-7c2f-4b29-80a2-bd9747098af1" containerName="payload-pre-processing"
Sep 16 14:49:28.775765 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:28.775722    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d0b1b65-75e8-4e27-8fdd-0061cf297144" containerName="payload-processing"
Sep 16 14:49:28.775765 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:28.775729    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f3b1be3-2e40-480e-917f-be84ff47f3b0" containerName="revoke-keys"
Sep 16 14:49:28.780178 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:28.780156    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-055ce3-q6bts"
Sep 16 14:49:28.783056 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:28.783036    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:49:28.783163 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:28.783039    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4jf\""
Sep 16 14:49:28.790123 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:28.790097    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-055ce3-q6bts"]
Sep 16 14:49:28.795167 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:28.795141    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a388a02-882d-4a81-8579-8a74fa9ee3ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-055ce3-q6bts\" (UID: \"8a388a02-882d-4a81-8579-8a74fa9ee3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-055ce3-q6bts"
Sep 16 14:49:28.896339 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:28.896301    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a388a02-882d-4a81-8579-8a74fa9ee3ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-055ce3-q6bts\" (UID: \"8a388a02-882d-4a81-8579-8a74fa9ee3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-055ce3-q6bts"
Sep 16 14:49:28.896988 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:28.896969    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a388a02-882d-4a81-8579-8a74fa9ee3ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-055ce3-q6bts\" (UID: \"8a388a02-882d-4a81-8579-8a74fa9ee3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-055ce3-q6bts"
Sep 16 14:49:29.091566 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:29.091490    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-055ce3-q6bts"
Sep 16 14:49:29.367076 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:29.367047    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-055ce3-q6bts"]
Sep 16 14:49:29.368419 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:49:29.368373    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a388a02_882d_4a81_8579_8a74fa9ee3ac.slice/crio-a05887586ec00c9d3480fe53536270c52f201f49724e50b050402de3299113a3 WatchSource:0}: Error finding container a05887586ec00c9d3480fe53536270c52f201f49724e50b050402de3299113a3: Status 404 returned error can't find the container with id a05887586ec00c9d3480fe53536270c52f201f49724e50b050402de3299113a3
Sep 16 14:49:29.831958 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:29.831919    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-055ce3-q6bts" event={"ID":"8a388a02-882d-4a81-8579-8a74fa9ee3ac","Type":"ContainerStarted","Data":"e74620e59bc51931dc3297ca82c8f7714180a74d720477d7e8d3677625682ea9"}
Sep 16 14:49:29.832324 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:29.831984    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-055ce3-q6bts" event={"ID":"8a388a02-882d-4a81-8579-8a74fa9ee3ac","Type":"ContainerStarted","Data":"a05887586ec00c9d3480fe53536270c52f201f49724e50b050402de3299113a3"}
Sep 16 14:49:29.850179 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:29.850122    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-055ce3-q6bts" podStartSLOduration=1.850105865 podStartE2EDuration="1.850105865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:49:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:49:29.848843178 +0000 UTC m=+2118.452468209" watchObservedRunningTime="2026-09-16 14:49:29.850105865 +0000 UTC m=+2118.453730897"
Sep 16 14:49:30.839208 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:30.839168    2587 generic.go:356] "Generic (PLEG): container finished" podID="8a388a02-882d-4a81-8579-8a74fa9ee3ac" containerID="e74620e59bc51931dc3297ca82c8f7714180a74d720477d7e8d3677625682ea9" exitCode=0
Sep 16 14:49:30.839637 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:30.839252    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-055ce3-q6bts" event={"ID":"8a388a02-882d-4a81-8579-8a74fa9ee3ac","Type":"ContainerDied","Data":"e74620e59bc51931dc3297ca82c8f7714180a74d720477d7e8d3677625682ea9"}
Sep 16 14:49:31.976984 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:31.976960    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-055ce3-q6bts"
Sep 16 14:49:32.026561 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:32.026531    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a388a02-882d-4a81-8579-8a74fa9ee3ac-maas-api-service-ca\") pod \"8a388a02-882d-4a81-8579-8a74fa9ee3ac\" (UID: \"8a388a02-882d-4a81-8579-8a74fa9ee3ac\") "
Sep 16 14:49:32.026875 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:32.026852    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a388a02-882d-4a81-8579-8a74fa9ee3ac-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8a388a02-882d-4a81-8579-8a74fa9ee3ac" (UID: "8a388a02-882d-4a81-8579-8a74fa9ee3ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:49:32.127828 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:32.127746    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a388a02-882d-4a81-8579-8a74fa9ee3ac-maas-api-service-ca\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:49:32.850926 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:32.850891    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-055ce3-q6bts"
Sep 16 14:49:32.851114 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:32.850928    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-055ce3-q6bts" event={"ID":"8a388a02-882d-4a81-8579-8a74fa9ee3ac","Type":"ContainerDied","Data":"a05887586ec00c9d3480fe53536270c52f201f49724e50b050402de3299113a3"}
Sep 16 14:49:32.851114 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:32.850984    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a05887586ec00c9d3480fe53536270c52f201f49724e50b050402de3299113a3"
Sep 16 14:49:39.237497 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:39.237458    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-055ce3-29826165-6rc8k"]
Sep 16 14:49:39.242408 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:39.242368    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-055ce3-29826165-6rc8k"]
Sep 16 14:49:39.269087 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:39.269054    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl"]
Sep 16 14:49:39.269347 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:39.269319    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl" podUID="ff38b70d-db5f-4464-acf6-bbbaeb77f70c" containerName="payload-processing" containerID="cri-o://6958e2e31a73960eab29156c180e1f69afff15802e8fdf38a19d2c5df2047bf0" gracePeriod=30
Sep 16 14:49:39.482286 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:39.482249    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-055ce3-q6bts"]
Sep 16 14:49:39.487031 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:39.487000    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-055ce3-q6bts"]
Sep 16 14:49:39.528535 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:39.528512    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl"
Sep 16 14:49:39.596271 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:39.596236    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff38b70d-db5f-4464-acf6-bbbaeb77f70c-plugins-config-volume\") pod \"ff38b70d-db5f-4464-acf6-bbbaeb77f70c\" (UID: \"ff38b70d-db5f-4464-acf6-bbbaeb77f70c\") "
Sep 16 14:49:39.596478 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:39.596317    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnr8s\" (UniqueName: \"kubernetes.io/projected/ff38b70d-db5f-4464-acf6-bbbaeb77f70c-kube-api-access-rnr8s\") pod \"ff38b70d-db5f-4464-acf6-bbbaeb77f70c\" (UID: \"ff38b70d-db5f-4464-acf6-bbbaeb77f70c\") "
Sep 16 14:49:39.596665 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:39.596642    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff38b70d-db5f-4464-acf6-bbbaeb77f70c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ff38b70d-db5f-4464-acf6-bbbaeb77f70c" (UID: "ff38b70d-db5f-4464-acf6-bbbaeb77f70c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:49:39.598525 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:39.598503    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff38b70d-db5f-4464-acf6-bbbaeb77f70c-kube-api-access-rnr8s" (OuterVolumeSpecName: "kube-api-access-rnr8s") pod "ff38b70d-db5f-4464-acf6-bbbaeb77f70c" (UID: "ff38b70d-db5f-4464-acf6-bbbaeb77f70c"). InnerVolumeSpecName "kube-api-access-rnr8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:49:39.697308 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:39.697271    2587 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff38b70d-db5f-4464-acf6-bbbaeb77f70c-plugins-config-volume\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:49:39.697308 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:39.697300    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnr8s\" (UniqueName: \"kubernetes.io/projected/ff38b70d-db5f-4464-acf6-bbbaeb77f70c-kube-api-access-rnr8s\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:49:39.887357 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:39.887272    2587 generic.go:356] "Generic (PLEG): container finished" podID="ff38b70d-db5f-4464-acf6-bbbaeb77f70c" containerID="6958e2e31a73960eab29156c180e1f69afff15802e8fdf38a19d2c5df2047bf0" exitCode=0
Sep 16 14:49:39.887357 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:39.887308    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl"
Sep 16 14:49:39.887357 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:39.887343    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl" event={"ID":"ff38b70d-db5f-4464-acf6-bbbaeb77f70c","Type":"ContainerDied","Data":"6958e2e31a73960eab29156c180e1f69afff15802e8fdf38a19d2c5df2047bf0"}
Sep 16 14:49:39.887617 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:39.887409    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl" event={"ID":"ff38b70d-db5f-4464-acf6-bbbaeb77f70c","Type":"ContainerDied","Data":"c7bb8e2e205d8d5f3f6944caf7da403c5eb2073b51dd6d3fdc66ffe7693e5578"}
Sep 16 14:49:39.887617 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:39.887429    2587 scope.go:117] "RemoveContainer" containerID="6958e2e31a73960eab29156c180e1f69afff15802e8fdf38a19d2c5df2047bf0"
Sep 16 14:49:39.896408 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:39.896369    2587 scope.go:117] "RemoveContainer" containerID="6958e2e31a73960eab29156c180e1f69afff15802e8fdf38a19d2c5df2047bf0"
Sep 16 14:49:39.896771 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:49:39.896749    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6958e2e31a73960eab29156c180e1f69afff15802e8fdf38a19d2c5df2047bf0\": container with ID starting with 6958e2e31a73960eab29156c180e1f69afff15802e8fdf38a19d2c5df2047bf0 not found: ID does not exist" containerID="6958e2e31a73960eab29156c180e1f69afff15802e8fdf38a19d2c5df2047bf0"
Sep 16 14:49:39.896841 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:39.896782    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6958e2e31a73960eab29156c180e1f69afff15802e8fdf38a19d2c5df2047bf0"} err="failed to get container status \"6958e2e31a73960eab29156c180e1f69afff15802e8fdf38a19d2c5df2047bf0\": rpc error: code = NotFound desc = could not find container \"6958e2e31a73960eab29156c180e1f69afff15802e8fdf38a19d2c5df2047bf0\": container with ID starting with 6958e2e31a73960eab29156c180e1f69afff15802e8fdf38a19d2c5df2047bf0 not found: ID does not exist"
Sep 16 14:49:39.902739 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:39.902713    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a388a02-882d-4a81-8579-8a74fa9ee3ac" path="/var/lib/kubelet/pods/8a388a02-882d-4a81-8579-8a74fa9ee3ac/volumes"
Sep 16 14:49:39.902983 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:39.902970    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc8dea42-c4ae-487b-8889-6797bbf48675" path="/var/lib/kubelet/pods/bc8dea42-c4ae-487b-8889-6797bbf48675/volumes"
Sep 16 14:49:39.929551 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:39.929527    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl"]
Sep 16 14:49:39.939102 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:39.939068    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-055ce3-5bf4b97b95-qbrwl"]
Sep 16 14:49:41.904449 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:41.904410    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff38b70d-db5f-4464-acf6-bbbaeb77f70c" path="/var/lib/kubelet/pods/ff38b70d-db5f-4464-acf6-bbbaeb77f70c/volumes"
Sep 16 14:49:49.778692 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:49.778653    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d46d94b58-wxggj"]
Sep 16 14:49:49.779080 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:49.778906    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6d46d94b58-wxggj" podUID="da5ee3c7-9059-4506-9246-1d395a659bf2" containerName="manager" containerID="cri-o://1d87dc0114649c5cb3b601b5fa6cd6f5f0199ab13f20ec951408d881c63adcfc" gracePeriod=30
Sep 16 14:49:49.942340 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:49.942299    2587 generic.go:356] "Generic (PLEG): container finished" podID="da5ee3c7-9059-4506-9246-1d395a659bf2" containerID="1d87dc0114649c5cb3b601b5fa6cd6f5f0199ab13f20ec951408d881c63adcfc" exitCode=0
Sep 16 14:49:49.942517 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:49.942367    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d46d94b58-wxggj" event={"ID":"da5ee3c7-9059-4506-9246-1d395a659bf2","Type":"ContainerDied","Data":"1d87dc0114649c5cb3b601b5fa6cd6f5f0199ab13f20ec951408d881c63adcfc"}
Sep 16 14:49:50.027943 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:50.027916    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d46d94b58-wxggj"
Sep 16 14:49:50.090608 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:50.090523    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da5ee3c7-9059-4506-9246-1d395a659bf2-webhook-cert\") pod \"da5ee3c7-9059-4506-9246-1d395a659bf2\" (UID: \"da5ee3c7-9059-4506-9246-1d395a659bf2\") "
Sep 16 14:49:50.090608 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:50.090580    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfgsp\" (UniqueName: \"kubernetes.io/projected/da5ee3c7-9059-4506-9246-1d395a659bf2-kube-api-access-pfgsp\") pod \"da5ee3c7-9059-4506-9246-1d395a659bf2\" (UID: \"da5ee3c7-9059-4506-9246-1d395a659bf2\") "
Sep 16 14:49:50.090835 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:50.090608    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da5ee3c7-9059-4506-9246-1d395a659bf2-metrics-certs\") pod \"da5ee3c7-9059-4506-9246-1d395a659bf2\" (UID: \"da5ee3c7-9059-4506-9246-1d395a659bf2\") "
Sep 16 14:49:50.092790 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:50.092756    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da5ee3c7-9059-4506-9246-1d395a659bf2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "da5ee3c7-9059-4506-9246-1d395a659bf2" (UID: "da5ee3c7-9059-4506-9246-1d395a659bf2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:49:50.092790 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:50.092776    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da5ee3c7-9059-4506-9246-1d395a659bf2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "da5ee3c7-9059-4506-9246-1d395a659bf2" (UID: "da5ee3c7-9059-4506-9246-1d395a659bf2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:49:50.092962 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:50.092870    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da5ee3c7-9059-4506-9246-1d395a659bf2-kube-api-access-pfgsp" (OuterVolumeSpecName: "kube-api-access-pfgsp") pod "da5ee3c7-9059-4506-9246-1d395a659bf2" (UID: "da5ee3c7-9059-4506-9246-1d395a659bf2"). InnerVolumeSpecName "kube-api-access-pfgsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:49:50.191296 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:50.191268    2587 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da5ee3c7-9059-4506-9246-1d395a659bf2-webhook-cert\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:49:50.191296 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:50.191295    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfgsp\" (UniqueName: \"kubernetes.io/projected/da5ee3c7-9059-4506-9246-1d395a659bf2-kube-api-access-pfgsp\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:49:50.191296 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:50.191304    2587 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da5ee3c7-9059-4506-9246-1d395a659bf2-metrics-certs\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:49:50.949179 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:50.949141    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d46d94b58-wxggj"
Sep 16 14:49:50.949629 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:50.949168    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d46d94b58-wxggj" event={"ID":"da5ee3c7-9059-4506-9246-1d395a659bf2","Type":"ContainerDied","Data":"54fa3c135af56311026f887fe13062a188baba377e14b04cae5d0cb7c4e6e66e"}
Sep 16 14:49:50.949629 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:50.949238    2587 scope.go:117] "RemoveContainer" containerID="1d87dc0114649c5cb3b601b5fa6cd6f5f0199ab13f20ec951408d881c63adcfc"
Sep 16 14:49:50.975072 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:50.975041    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d46d94b58-wxggj"]
Sep 16 14:49:50.986032 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:50.986002    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6d46d94b58-wxggj"]
Sep 16 14:49:51.902760 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:49:51.902727    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da5ee3c7-9059-4506-9246-1d395a659bf2" path="/var/lib/kubelet/pods/da5ee3c7-9059-4506-9246-1d395a659bf2/volumes"
Sep 16 14:51:12.477370 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:51:12.477341    2587 scope.go:117] "RemoveContainer" containerID="db04ef57a2e383c4ff247719adfea8fdbdce4a660fd97f2c9674484afe9d75b6"
Sep 16 14:54:06.920521 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:06.920478    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"]
Sep 16 14:54:06.921115 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:06.920957    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a388a02-882d-4a81-8579-8a74fa9ee3ac" containerName="revoke-keys"
Sep 16 14:54:06.921115 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:06.920976    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a388a02-882d-4a81-8579-8a74fa9ee3ac" containerName="revoke-keys"
Sep 16 14:54:06.921115 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:06.920988    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff38b70d-db5f-4464-acf6-bbbaeb77f70c" containerName="payload-processing"
Sep 16 14:54:06.921115 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:06.920996    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff38b70d-db5f-4464-acf6-bbbaeb77f70c" containerName="payload-processing"
Sep 16 14:54:06.921115 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:06.921010    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da5ee3c7-9059-4506-9246-1d395a659bf2" containerName="manager"
Sep 16 14:54:06.921115 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:06.921019    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="da5ee3c7-9059-4506-9246-1d395a659bf2" containerName="manager"
Sep 16 14:54:06.921479 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:06.921147    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a388a02-882d-4a81-8579-8a74fa9ee3ac" containerName="revoke-keys"
Sep 16 14:54:06.921479 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:06.921161    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff38b70d-db5f-4464-acf6-bbbaeb77f70c" containerName="payload-processing"
Sep 16 14:54:06.921479 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:06.921175    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="da5ee3c7-9059-4506-9246-1d395a659bf2" containerName="manager"
Sep 16 14:54:06.924267 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:06.924247    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"
Sep 16 14:54:06.926884 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:06.926857    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-f0c0912d-kserve-self-signed-certs\""
Sep 16 14:54:06.936202 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:06.936176    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"]
Sep 16 14:54:06.989010 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:06.988976    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-tmp-dir\") pod \"e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k\" (UID: \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\") " pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"
Sep 16 14:54:06.989010 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:06.989013    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s9mt\" (UniqueName: \"kubernetes.io/projected/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-kube-api-access-8s9mt\") pod \"e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k\" (UID: \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\") " pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"
Sep 16 14:54:06.989275 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:06.989052    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-tls-certs\") pod \"e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k\" (UID: \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\") " pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"
Sep 16 14:54:06.989275 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:06.989136    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-dshm\") pod \"e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k\" (UID: \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\") " pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"
Sep 16 14:54:06.989275 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:06.989201    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-home\") pod \"e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k\" (UID: \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\") " pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"
Sep 16 14:54:06.989275 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:06.989253    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-model-cache\") pod \"e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k\" (UID: \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\") " pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"
Sep 16 14:54:07.089849 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:07.089800    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s9mt\" (UniqueName: \"kubernetes.io/projected/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-kube-api-access-8s9mt\") pod \"e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k\" (UID: \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\") " pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"
Sep 16 14:54:07.090001 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:07.089868    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-tls-certs\") pod \"e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k\" (UID: \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\") " pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"
Sep 16 14:54:07.090001 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:07.089902    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-dshm\") pod \"e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k\" (UID: \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\") " pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"
Sep 16 14:54:07.090001 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:07.089927    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-home\") pod \"e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k\" (UID: \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\") " pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"
Sep 16 14:54:07.090001 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:07.089947    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-model-cache\") pod \"e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k\" (UID: \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\") " pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"
Sep 16 14:54:07.090001 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:07.089966    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-tmp-dir\") pod \"e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k\" (UID: \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\") " pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"
Sep 16 14:54:07.090403 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:07.090371    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-home\") pod \"e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k\" (UID: \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\") " pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"
Sep 16 14:54:07.090520 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:07.090438    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-model-cache\") pod \"e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k\" (UID: \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\") " pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"
Sep 16 14:54:07.090520 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:07.090444    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-tmp-dir\") pod \"e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k\" (UID: \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\") " pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"
Sep 16 14:54:07.092332 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:07.092313    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-dshm\") pod \"e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k\" (UID: \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\") " pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"
Sep 16 14:54:07.092521 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:07.092504    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-tls-certs\") pod \"e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k\" (UID: \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\") " pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"
Sep 16 14:54:07.098122 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:07.098099    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s9mt\" (UniqueName: \"kubernetes.io/projected/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-kube-api-access-8s9mt\") pod \"e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k\" (UID: \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\") " pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"
Sep 16 14:54:07.233932 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:07.233846    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"
Sep 16 14:54:07.513629 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:07.513605    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"]
Sep 16 14:54:07.515751 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:54:07.515720    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9ce07f4_c7ff_4086_970d_4a5c2c37e063.slice/crio-7ddc71bb9c32c7c0a35f27e00e89d4bf16baf706f80fe94f9084b8c3adcb00cb WatchSource:0}: Error finding container 7ddc71bb9c32c7c0a35f27e00e89d4bf16baf706f80fe94f9084b8c3adcb00cb: Status 404 returned error can't find the container with id 7ddc71bb9c32c7c0a35f27e00e89d4bf16baf706f80fe94f9084b8c3adcb00cb
Sep 16 14:54:08.153982 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:08.153946    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k" event={"ID":"f9ce07f4-c7ff-4086-970d-4a5c2c37e063","Type":"ContainerStarted","Data":"3467d378c80f5802114fa756769aee339b045fbb3a98f392ffc5b1ffc635f0b8"}
Sep 16 14:54:08.154439 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:08.153993    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k" event={"ID":"f9ce07f4-c7ff-4086-970d-4a5c2c37e063","Type":"ContainerStarted","Data":"7ddc71bb9c32c7c0a35f27e00e89d4bf16baf706f80fe94f9084b8c3adcb00cb"}
Sep 16 14:54:08.173148 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:08.173099    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k" podStartSLOduration=2.173086119 podStartE2EDuration="2.173086119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:54:08.172029591 +0000 UTC m=+2396.775654622" watchObservedRunningTime="2026-09-16 14:54:08.173086119 +0000 UTC m=+2396.776711151"
Sep 16 14:54:12.507949 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:12.507916    2587 scope.go:117] "RemoveContainer" containerID="71fe776dfbc4f1e696834692c4a242b901ae8b89361ba1873c0bc0542ceea97d"
Sep 16 14:54:12.516268 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:12.516250    2587 scope.go:117] "RemoveContainer" containerID="82d4d82ebf966aff2ce8986de6d896d2668209ca2ea8c16eb94e0b62330e9996"
Sep 16 14:54:17.234651 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:17.234611    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"
Sep 16 14:54:17.235058 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:17.234667    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"
Sep 16 14:54:17.247179 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:17.247155    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"
Sep 16 14:54:18.215248 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:18.215217    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"
Sep 16 14:54:52.335913 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:54:52.335835    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"]
Sep 16 14:55:07.355285 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:07.355216    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k" podUID="f9ce07f4-c7ff-4086-970d-4a5c2c37e063" containerName="main" containerID="cri-o://3467d378c80f5802114fa756769aee339b045fbb3a98f392ffc5b1ffc635f0b8" gracePeriod=45
Sep 16 14:55:07.601649 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:07.601625    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"
Sep 16 14:55:07.730433 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:07.730380    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-home\") pod \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\" (UID: \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\") "
Sep 16 14:55:07.730615 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:07.730446    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-tmp-dir\") pod \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\" (UID: \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\") "
Sep 16 14:55:07.730615 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:07.730495    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8s9mt\" (UniqueName: \"kubernetes.io/projected/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-kube-api-access-8s9mt\") pod \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\" (UID: \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\") "
Sep 16 14:55:07.730615 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:07.730524    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-model-cache\") pod \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\" (UID: \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\") "
Sep 16 14:55:07.730756 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:07.730659    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-dshm\") pod \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\" (UID: \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\") "
Sep 16 14:55:07.730756 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:07.730706    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-tls-certs\") pod \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\" (UID: \"f9ce07f4-c7ff-4086-970d-4a5c2c37e063\") "
Sep 16 14:55:07.730756 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:07.730705    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-home" (OuterVolumeSpecName: "home") pod "f9ce07f4-c7ff-4086-970d-4a5c2c37e063" (UID: "f9ce07f4-c7ff-4086-970d-4a5c2c37e063"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:55:07.730756 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:07.730714    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f9ce07f4-c7ff-4086-970d-4a5c2c37e063" (UID: "f9ce07f4-c7ff-4086-970d-4a5c2c37e063"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:55:07.730962 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:07.730774    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-model-cache" (OuterVolumeSpecName: "model-cache") pod "f9ce07f4-c7ff-4086-970d-4a5c2c37e063" (UID: "f9ce07f4-c7ff-4086-970d-4a5c2c37e063"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:55:07.731042 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:07.731020    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-home\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:55:07.731102 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:07.731045    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-tmp-dir\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:55:07.731102 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:07.731056    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-model-cache\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:55:07.732915 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:07.732881    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-dshm" (OuterVolumeSpecName: "dshm") pod "f9ce07f4-c7ff-4086-970d-4a5c2c37e063" (UID: "f9ce07f4-c7ff-4086-970d-4a5c2c37e063"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:55:07.733031 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:07.732922    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f9ce07f4-c7ff-4086-970d-4a5c2c37e063" (UID: "f9ce07f4-c7ff-4086-970d-4a5c2c37e063"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:55:07.733031 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:07.732951    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-kube-api-access-8s9mt" (OuterVolumeSpecName: "kube-api-access-8s9mt") pod "f9ce07f4-c7ff-4086-970d-4a5c2c37e063" (UID: "f9ce07f4-c7ff-4086-970d-4a5c2c37e063"). InnerVolumeSpecName "kube-api-access-8s9mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:55:07.832477 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:07.832444    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s9mt\" (UniqueName: \"kubernetes.io/projected/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-kube-api-access-8s9mt\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:55:07.832477 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:07.832469    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-dshm\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:55:07.832477 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:07.832478    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9ce07f4-c7ff-4086-970d-4a5c2c37e063-tls-certs\") on node \"ip-10-0-132-217.ec2.internal\" DevicePath \"\""
Sep 16 14:55:08.439868 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:08.439834    2587 generic.go:356] "Generic (PLEG): container finished" podID="f9ce07f4-c7ff-4086-970d-4a5c2c37e063" containerID="3467d378c80f5802114fa756769aee339b045fbb3a98f392ffc5b1ffc635f0b8" exitCode=0
Sep 16 14:55:08.440277 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:08.439881    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"
Sep 16 14:55:08.440277 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:08.439923    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k" event={"ID":"f9ce07f4-c7ff-4086-970d-4a5c2c37e063","Type":"ContainerDied","Data":"3467d378c80f5802114fa756769aee339b045fbb3a98f392ffc5b1ffc635f0b8"}
Sep 16 14:55:08.440277 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:08.439968    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k" event={"ID":"f9ce07f4-c7ff-4086-970d-4a5c2c37e063","Type":"ContainerDied","Data":"7ddc71bb9c32c7c0a35f27e00e89d4bf16baf706f80fe94f9084b8c3adcb00cb"}
Sep 16 14:55:08.440277 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:08.439985    2587 scope.go:117] "RemoveContainer" containerID="3467d378c80f5802114fa756769aee339b045fbb3a98f392ffc5b1ffc635f0b8"
Sep 16 14:55:08.448450 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:08.448431    2587 scope.go:117] "RemoveContainer" containerID="3467d378c80f5802114fa756769aee339b045fbb3a98f392ffc5b1ffc635f0b8"
Sep 16 14:55:08.448708 ip-10-0-132-217 kubenswrapper[2587]: E0916 14:55:08.448688    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3467d378c80f5802114fa756769aee339b045fbb3a98f392ffc5b1ffc635f0b8\": container with ID starting with 3467d378c80f5802114fa756769aee339b045fbb3a98f392ffc5b1ffc635f0b8 not found: ID does not exist" containerID="3467d378c80f5802114fa756769aee339b045fbb3a98f392ffc5b1ffc635f0b8"
Sep 16 14:55:08.448755 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:08.448715    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3467d378c80f5802114fa756769aee339b045fbb3a98f392ffc5b1ffc635f0b8"} err="failed to get container status \"3467d378c80f5802114fa756769aee339b045fbb3a98f392ffc5b1ffc635f0b8\": rpc error: code = NotFound desc = could not find container \"3467d378c80f5802114fa756769aee339b045fbb3a98f392ffc5b1ffc635f0b8\": container with ID starting with 3467d378c80f5802114fa756769aee339b045fbb3a98f392ffc5b1ffc635f0b8 not found: ID does not exist"
Sep 16 14:55:08.471652 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:08.471628    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"]
Sep 16 14:55:08.482110 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:08.482088    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-f0c0912d-kserve-5bf6c5d575-bwc4k"]
Sep 16 14:55:09.901998 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:09.901967    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9ce07f4-c7ff-4086-970d-4a5c2c37e063" path="/var/lib/kubelet/pods/f9ce07f4-c7ff-4086-970d-4a5c2c37e063/volumes"
Sep 16 14:55:12.537807 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:12.537775    2587 scope.go:117] "RemoveContainer" containerID="7006c7967751f0d58741cfe428bdab3130cab28a892b3e55cb7702dbc4381cb9"
Sep 16 14:55:12.546024 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:12.546005    2587 scope.go:117] "RemoveContainer" containerID="5a5c98f7efd6564ff38c6a2bfb7654853609a32aff56defbd3fc4b760a1bfe68"
Sep 16 14:55:34.051339 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:34.051309    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7597cddb56-6jn88_1dbe5ecc-4d35-41ae-b95c-bd86da658146/authorino/0.log"
Sep 16 14:55:40.232625 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:40.232587    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b85944db-9bv9d_910e25fa-e7b6-4b60-a5f0-255d7fc3ca65/copy-manifests/0.log"
Sep 16 14:55:40.252916 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:40.252884    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b85944db-9bv9d_910e25fa-e7b6-4b60-a5f0-255d7fc3ca65/manager/0.log"
Sep 16 14:55:41.322731 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:41.322702    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29826165-nh6tj_f3e3982e-e6c8-4398-a095-45cb4be13377/cleanup/0.log"
Sep 16 14:55:42.242409 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:42.242350    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7597cddb56-6jn88_1dbe5ecc-4d35-41ae-b95c-bd86da658146/authorino/0.log"
Sep 16 14:55:42.592530 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:42.592451    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-m7tvq_40ef16cf-9f50-4e74-9e55-c8a6399d47fb/kuadrant-console-plugin/0.log"
Sep 16 14:55:43.582309 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:43.582270    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75f575964-h6spt_293cc966-ff29-490d-8886-0a734d91f817/kube-auth-proxy/0.log"
Sep 16 14:55:43.955921 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:43.955890    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-x4c4q_4a2432ed-9c4b-4fdf-8f68-bcff18302d6a/payload-pre-processing/0.log"
Sep 16 14:55:44.198632 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:44.198597    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f487956db-zccck_75e1ae3b-e19a-47f1-9395-c18e8e393062/router/0.log"
Sep 16 14:55:44.552077 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:44.552046    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-z7wt2_7c741af3-761f-4a25-9752-64324d62cb5f/main/0.log"
Sep 16 14:55:45.143671 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:45.143642    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-mw7qv_23a04036-268e-45c5-815a-12b48c81be4b/main/0.log"
Sep 16 14:55:45.266405 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:45.266372    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-jtzgh_8a32ca17-c160-4fa0-b9a2-67c13d1ffe00/main/0.log"
Sep 16 14:55:50.533037 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:50.532992    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vbsd8/must-gather-zlfst"]
Sep 16 14:55:50.534268 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:50.534238    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9ce07f4-c7ff-4086-970d-4a5c2c37e063" containerName="main"
Sep 16 14:55:50.534451 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:50.534438    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9ce07f4-c7ff-4086-970d-4a5c2c37e063" containerName="main"
Sep 16 14:55:50.534860 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:50.534838    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9ce07f4-c7ff-4086-970d-4a5c2c37e063" containerName="main"
Sep 16 14:55:50.539526 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:50.539504    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vbsd8/must-gather-zlfst"
Sep 16 14:55:50.542339 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:50.542315    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vbsd8\"/\"openshift-service-ca.crt\""
Sep 16 14:55:50.542474 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:50.542351    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vbsd8\"/\"default-dockercfg-gkpxd\""
Sep 16 14:55:50.543117 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:50.543078    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vbsd8\"/\"kube-root-ca.crt\""
Sep 16 14:55:50.543503 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:50.543483    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vbsd8/must-gather-zlfst"]
Sep 16 14:55:50.705408 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:50.705350    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6lxh\" (UniqueName: \"kubernetes.io/projected/2414765b-4603-4bb0-b21e-44fb2e7529a9-kube-api-access-r6lxh\") pod \"must-gather-zlfst\" (UID: \"2414765b-4603-4bb0-b21e-44fb2e7529a9\") " pod="openshift-must-gather-vbsd8/must-gather-zlfst"
Sep 16 14:55:50.705601 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:50.705422    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2414765b-4603-4bb0-b21e-44fb2e7529a9-must-gather-output\") pod \"must-gather-zlfst\" (UID: \"2414765b-4603-4bb0-b21e-44fb2e7529a9\") " pod="openshift-must-gather-vbsd8/must-gather-zlfst"
Sep 16 14:55:50.805959 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:50.805876    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6lxh\" (UniqueName: \"kubernetes.io/projected/2414765b-4603-4bb0-b21e-44fb2e7529a9-kube-api-access-r6lxh\") pod \"must-gather-zlfst\" (UID: \"2414765b-4603-4bb0-b21e-44fb2e7529a9\") " pod="openshift-must-gather-vbsd8/must-gather-zlfst"
Sep 16 14:55:50.805959 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:50.805916    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2414765b-4603-4bb0-b21e-44fb2e7529a9-must-gather-output\") pod \"must-gather-zlfst\" (UID: \"2414765b-4603-4bb0-b21e-44fb2e7529a9\") " pod="openshift-must-gather-vbsd8/must-gather-zlfst"
Sep 16 14:55:50.806280 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:50.806262    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2414765b-4603-4bb0-b21e-44fb2e7529a9-must-gather-output\") pod \"must-gather-zlfst\" (UID: \"2414765b-4603-4bb0-b21e-44fb2e7529a9\") " pod="openshift-must-gather-vbsd8/must-gather-zlfst"
Sep 16 14:55:50.814842 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:50.814816    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6lxh\" (UniqueName: \"kubernetes.io/projected/2414765b-4603-4bb0-b21e-44fb2e7529a9-kube-api-access-r6lxh\") pod \"must-gather-zlfst\" (UID: \"2414765b-4603-4bb0-b21e-44fb2e7529a9\") " pod="openshift-must-gather-vbsd8/must-gather-zlfst"
Sep 16 14:55:50.849017 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:50.848983    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vbsd8/must-gather-zlfst"
Sep 16 14:55:51.128658 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:51.128636    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vbsd8/must-gather-zlfst"]
Sep 16 14:55:51.130933 ip-10-0-132-217 kubenswrapper[2587]: W0916 14:55:51.130907    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2414765b_4603_4bb0_b21e_44fb2e7529a9.slice/crio-f6991ab0dedb0f6731c290c637404964072f40f3e2cd0d9c8f588fd93006d60a WatchSource:0}: Error finding container f6991ab0dedb0f6731c290c637404964072f40f3e2cd0d9c8f588fd93006d60a: Status 404 returned error can't find the container with id f6991ab0dedb0f6731c290c637404964072f40f3e2cd0d9c8f588fd93006d60a
Sep 16 14:55:51.133235 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:51.133218    2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 14:55:51.637287 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:51.637254    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vbsd8/must-gather-zlfst" event={"ID":"2414765b-4603-4bb0-b21e-44fb2e7529a9","Type":"ContainerStarted","Data":"f6991ab0dedb0f6731c290c637404964072f40f3e2cd0d9c8f588fd93006d60a"}
Sep 16 14:55:52.646308 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:52.646270    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vbsd8/must-gather-zlfst" event={"ID":"2414765b-4603-4bb0-b21e-44fb2e7529a9","Type":"ContainerStarted","Data":"ea60c60fae8de33fded5c55a2254fc7debec7a7aeb1adf60ed60686499002044"}
Sep 16 14:55:52.646743 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:52.646326    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vbsd8/must-gather-zlfst" event={"ID":"2414765b-4603-4bb0-b21e-44fb2e7529a9","Type":"ContainerStarted","Data":"a8a4a51262302469f59b7962d17b0e762c093f068c5e56fac91866458a311763"}
Sep 16 14:55:52.665858 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:52.665807    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vbsd8/must-gather-zlfst" podStartSLOduration=1.7827645250000002 podStartE2EDuration="2.665785431s" totalImagesPullingTime="883.020906ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:55:52.665351944 +0000 UTC m=+2501.268976978" watchObservedRunningTime="2026-09-16 14:55:52.665785431 +0000 UTC m=+2501.269410502"
Sep 16 14:55:53.773783 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:53.773750    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x59fz_6f12cf92-2a09-4fe6-a73e-5692728769b3/global-pull-secret-syncer/0.log"
Sep 16 14:55:53.918591 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:53.918564    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vk8ld_489a66c3-b01a-45d8-8747-4ee83b84958e/konnectivity-agent/0.log"
Sep 16 14:55:53.992226 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:53.992195    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-217.ec2.internal_e37d493554911b356e79c3add3a4b130/haproxy/0.log"
Sep 16 14:55:58.149064 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:58.149021    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7597cddb56-6jn88_1dbe5ecc-4d35-41ae-b95c-bd86da658146/authorino/0.log"
Sep 16 14:55:58.236892 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:55:58.236853    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-m7tvq_40ef16cf-9f50-4e74-9e55-c8a6399d47fb/kuadrant-console-plugin/0.log"
Sep 16 14:56:00.525260 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:00.525230    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-78569c95d-l6whj_380482d8-a477-4212-9214-ce63a801e2bc/metrics-server/0.log"
Sep 16 14:56:00.577828 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:00.577798    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-jn8t7_a9d47a49-cd8e-4a5f-88b4-a96ea09f9340/monitoring-plugin/0.log"
Sep 16 14:56:00.920599 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:00.920567    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zs8bz_7761e508-6372-4021-a780-1cdf293c9ea1/node-exporter/0.log"
Sep 16 14:56:00.959501 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:00.959467    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zs8bz_7761e508-6372-4021-a780-1cdf293c9ea1/kube-rbac-proxy/0.log"
Sep 16 14:56:01.009572 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:01.009538    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zs8bz_7761e508-6372-4021-a780-1cdf293c9ea1/init-textfile/0.log"
Sep 16 14:56:01.061488 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:01.061453    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-8w9hx_5257ed36-4242-4d92-96e6-9981d89366cc/kube-rbac-proxy-main/0.log"
Sep 16 14:56:01.114085 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:01.114024    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-8w9hx_5257ed36-4242-4d92-96e6-9981d89366cc/kube-rbac-proxy-self/0.log"
Sep 16 14:56:01.156256 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:01.156219    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-8w9hx_5257ed36-4242-4d92-96e6-9981d89366cc/openshift-state-metrics/0.log"
Sep 16 14:56:01.599560 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:01.599524    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-mp9z7_6a259dcd-592a-4579-9180-60f6dc5eac36/prometheus-operator/0.log"
Sep 16 14:56:01.670539 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:01.670508    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-mp9z7_6a259dcd-592a-4579-9180-60f6dc5eac36/kube-rbac-proxy/0.log"
Sep 16 14:56:04.843202 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:04.843096    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vbsd8/perf-node-gather-daemonset-r66vx"]
Sep 16 14:56:04.851381 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:04.851350    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-r66vx"
Sep 16 14:56:04.859737 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:04.859698    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vbsd8/perf-node-gather-daemonset-r66vx"]
Sep 16 14:56:04.949080 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:04.949044    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e67653c-cfd6-463f-b235-0fdcd958a267-lib-modules\") pod \"perf-node-gather-daemonset-r66vx\" (UID: \"2e67653c-cfd6-463f-b235-0fdcd958a267\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-r66vx"
Sep 16 14:56:04.949080 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:04.949087    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2e67653c-cfd6-463f-b235-0fdcd958a267-proc\") pod \"perf-node-gather-daemonset-r66vx\" (UID: \"2e67653c-cfd6-463f-b235-0fdcd958a267\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-r66vx"
Sep 16 14:56:04.949333 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:04.949137    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w59bx\" (UniqueName: \"kubernetes.io/projected/2e67653c-cfd6-463f-b235-0fdcd958a267-kube-api-access-w59bx\") pod \"perf-node-gather-daemonset-r66vx\" (UID: \"2e67653c-cfd6-463f-b235-0fdcd958a267\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-r66vx"
Sep 16 14:56:04.949333 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:04.949163    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e67653c-cfd6-463f-b235-0fdcd958a267-sys\") pod \"perf-node-gather-daemonset-r66vx\" (UID: \"2e67653c-cfd6-463f-b235-0fdcd958a267\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-r66vx"
Sep 16 14:56:04.949333 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:04.949187    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2e67653c-cfd6-463f-b235-0fdcd958a267-podres\") pod \"perf-node-gather-daemonset-r66vx\" (UID: \"2e67653c-cfd6-463f-b235-0fdcd958a267\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-r66vx"
Sep 16 14:56:05.050460 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:05.050421    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e67653c-cfd6-463f-b235-0fdcd958a267-lib-modules\") pod \"perf-node-gather-daemonset-r66vx\" (UID: \"2e67653c-cfd6-463f-b235-0fdcd958a267\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-r66vx"
Sep 16 14:56:05.050632 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:05.050479    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2e67653c-cfd6-463f-b235-0fdcd958a267-proc\") pod \"perf-node-gather-daemonset-r66vx\" (UID: \"2e67653c-cfd6-463f-b235-0fdcd958a267\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-r66vx"
Sep 16 14:56:05.050632 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:05.050516    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w59bx\" (UniqueName: \"kubernetes.io/projected/2e67653c-cfd6-463f-b235-0fdcd958a267-kube-api-access-w59bx\") pod \"perf-node-gather-daemonset-r66vx\" (UID: \"2e67653c-cfd6-463f-b235-0fdcd958a267\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-r66vx"
Sep 16 14:56:05.050632 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:05.050541    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e67653c-cfd6-463f-b235-0fdcd958a267-sys\") pod \"perf-node-gather-daemonset-r66vx\" (UID: \"2e67653c-cfd6-463f-b235-0fdcd958a267\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-r66vx"
Sep 16 14:56:05.050632 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:05.050606    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e67653c-cfd6-463f-b235-0fdcd958a267-lib-modules\") pod \"perf-node-gather-daemonset-r66vx\" (UID: \"2e67653c-cfd6-463f-b235-0fdcd958a267\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-r66vx"
Sep 16 14:56:05.050632 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:05.050606    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e67653c-cfd6-463f-b235-0fdcd958a267-sys\") pod \"perf-node-gather-daemonset-r66vx\" (UID: \"2e67653c-cfd6-463f-b235-0fdcd958a267\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-r66vx"
Sep 16 14:56:05.050898 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:05.050635    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2e67653c-cfd6-463f-b235-0fdcd958a267-proc\") pod \"perf-node-gather-daemonset-r66vx\" (UID: \"2e67653c-cfd6-463f-b235-0fdcd958a267\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-r66vx"
Sep 16 14:56:05.050898 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:05.050687    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2e67653c-cfd6-463f-b235-0fdcd958a267-podres\") pod \"perf-node-gather-daemonset-r66vx\" (UID: \"2e67653c-cfd6-463f-b235-0fdcd958a267\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-r66vx"
Sep 16 14:56:05.050898 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:05.050836    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2e67653c-cfd6-463f-b235-0fdcd958a267-podres\") pod \"perf-node-gather-daemonset-r66vx\" (UID: \"2e67653c-cfd6-463f-b235-0fdcd958a267\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-r66vx"
Sep 16 14:56:05.066090 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:05.065929    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w59bx\" (UniqueName: \"kubernetes.io/projected/2e67653c-cfd6-463f-b235-0fdcd958a267-kube-api-access-w59bx\") pod \"perf-node-gather-daemonset-r66vx\" (UID: \"2e67653c-cfd6-463f-b235-0fdcd958a267\") " pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-r66vx"
Sep 16 14:56:05.170570 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:05.170527    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-r66vx"
Sep 16 14:56:05.473614 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:05.473511    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vbsd8/perf-node-gather-daemonset-r66vx"]
Sep 16 14:56:05.744082 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:05.743983    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-r66vx" event={"ID":"2e67653c-cfd6-463f-b235-0fdcd958a267","Type":"ContainerStarted","Data":"919aa973a27a998695b3598db4175d02d2050b88b3768fc7e091612733fc3410"}
Sep 16 14:56:05.744082 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:05.744054    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-r66vx" event={"ID":"2e67653c-cfd6-463f-b235-0fdcd958a267","Type":"ContainerStarted","Data":"7b8ed252eb9087a86e7c98211576cdc4838693e98fc53ebbc7c5ddab3cf48a75"}
Sep 16 14:56:05.744274 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:05.744105    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-r66vx"
Sep 16 14:56:07.694115 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:07.694079    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4ntk2_38e73538-4f3c-4d56-bc47-e9bfb8525e0a/dns/0.log"
Sep 16 14:56:07.720500 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:07.720476    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4ntk2_38e73538-4f3c-4d56-bc47-e9bfb8525e0a/kube-rbac-proxy/0.log"
Sep 16 14:56:08.009054 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:08.008972    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-trl7p_4193914e-79e1-4c6a-ba3b-a4d9f5731637/dns-node-resolver/0.log"
Sep 16 14:56:08.705022 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:08.704987    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-f45d8ddc4-2zl52_39e08d27-3153-4a39-9ecd-7d162f1965aa/registry/0.log"
Sep 16 14:56:08.835297 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:08.835257    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-drp7m_d657d718-fd8c-4034-9cd8-232f90fc573e/node-ca/0.log"
Sep 16 14:56:11.079122 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:11.079088    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75f575964-h6spt_293cc966-ff29-490d-8886-0a734d91f817/kube-auth-proxy/0.log"
Sep 16 14:56:11.209119 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:11.209080    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-x4c4q_4a2432ed-9c4b-4fdf-8f68-bcff18302d6a/payload-pre-processing/0.log"
Sep 16 14:56:11.303870 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:11.303841    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f487956db-zccck_75e1ae3b-e19a-47f1-9395-c18e8e393062/router/0.log"
Sep 16 14:56:11.764561 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:11.764527    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-r66vx"
Sep 16 14:56:11.815296 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:11.815217    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vbsd8/perf-node-gather-daemonset-r66vx" podStartSLOduration=7.815193991 podStartE2EDuration="7.815193991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:56:05.777254525 +0000 UTC m=+2514.380879558" watchObservedRunningTime="2026-09-16 14:56:11.815193991 +0000 UTC m=+2520.418819024"
Sep 16 14:56:12.397633 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:12.397602    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v2zl4_bd778876-4a84-403c-b6a7-bfa346cfe0d5/serve-healthcheck-canary/0.log"
Sep 16 14:56:12.571276 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:12.571244    2587 scope.go:117] "RemoveContainer" containerID="e74620e59bc51931dc3297ca82c8f7714180a74d720477d7e8d3677625682ea9"
Sep 16 14:56:13.127312 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:13.127273    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-j674w_3e8e2284-b089-4525-80c5-93353f07252f/insights-operator/0.log"
Sep 16 14:56:13.128043 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:13.128016    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-j674w_3e8e2284-b089-4525-80c5-93353f07252f/insights-operator/1.log"
Sep 16 14:56:13.193650 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:13.193615    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lqtmz_1ae7e37a-60a5-4d09-8e0f-905cad155314/kube-rbac-proxy/0.log"
Sep 16 14:56:13.245817 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:13.245791    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lqtmz_1ae7e37a-60a5-4d09-8e0f-905cad155314/exporter/0.log"
Sep 16 14:56:13.287326 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:13.287290    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lqtmz_1ae7e37a-60a5-4d09-8e0f-905cad155314/extractor/0.log"
Sep 16 14:56:14.396322 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:14.396236    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75f575964-h6spt_293cc966-ff29-490d-8886-0a734d91f817/kube-auth-proxy/0.log"
Sep 16 14:56:14.553264 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:14.553235    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-x4c4q_4a2432ed-9c4b-4fdf-8f68-bcff18302d6a/payload-pre-processing/0.log"
Sep 16 14:56:14.698449 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:14.698297    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f487956db-zccck_75e1ae3b-e19a-47f1-9395-c18e8e393062/router/0.log"
Sep 16 14:56:17.363277 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:17.363246    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75f575964-h6spt_293cc966-ff29-490d-8886-0a734d91f817/kube-auth-proxy/0.log"
Sep 16 14:56:17.659167 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:17.659141    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-x4c4q_4a2432ed-9c4b-4fdf-8f68-bcff18302d6a/payload-pre-processing/0.log"
Sep 16 14:56:18.031978 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:18.031899    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f487956db-zccck_75e1ae3b-e19a-47f1-9395-c18e8e393062/router/0.log"
Sep 16 14:56:18.328565 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:18.328489    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b85944db-9bv9d_910e25fa-e7b6-4b60-a5f0-255d7fc3ca65/manager/0.log"
Sep 16 14:56:18.403235 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:18.403211    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b85944db-9bv9d_910e25fa-e7b6-4b60-a5f0-255d7fc3ca65/copy-manifests/0.log"
Sep 16 14:56:20.302744 ip-10-0-132-217 kubenswrapper[2587]: I0916 14:56:20.302717    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5cbd5cd6f8-bn5dg_7a322e55-06c1-44e7-993a-331af159e4e1/manager/0.log"