Sep 17 15:29:36.449184 ip-10-0-134-104 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 15:29:36.873491 ip-10-0-134-104 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:29:36.873491 ip-10-0-134-104 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 15:29:36.873491 ip-10-0-134-104 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:29:36.873491 ip-10-0-134-104 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 15:29:36.873491 ip-10-0-134-104 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:29:36.876415 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.876322    2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 15:29:36.879665 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879647    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:29:36.879665 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879664    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:29:36.879729 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879667    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:29:36.879729 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879671    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:29:36.879729 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879674    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:29:36.879729 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879677    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:29:36.879729 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879679    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:29:36.879729 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879684    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:29:36.879729 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879686    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:29:36.879729 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879689    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:29:36.879729 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879693    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:29:36.879729 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879698    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:29:36.879729 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879701    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:29:36.879729 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879704    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:29:36.879729 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879707    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:29:36.879729 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879710    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:29:36.879729 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879723    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:29:36.879729 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879726    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:29:36.879729 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879729    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:29:36.879729 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879732    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:29:36.879729 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879736    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:29:36.879729 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879739    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:29:36.880221 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879742    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:29:36.880221 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879745    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:29:36.880221 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879747    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:29:36.880221 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879750    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:29:36.880221 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879753    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:29:36.880221 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879755    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:29:36.880221 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879758    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:29:36.880221 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879760    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:29:36.880221 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879763    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:29:36.880221 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879765    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:29:36.880221 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879768    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:29:36.880221 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879770    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:29:36.880221 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879773    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:29:36.880221 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879775    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:29:36.880221 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879778    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:29:36.880221 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879780    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:29:36.880221 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879783    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:29:36.880221 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879785    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:29:36.880221 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879787    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:29:36.880221 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879790    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:29:36.880701 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879792    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:29:36.880701 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879795    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:29:36.880701 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879797    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:29:36.880701 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879800    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:29:36.880701 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879802    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:29:36.880701 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879805    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:29:36.880701 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879807    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:29:36.880701 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879815    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:29:36.880701 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879817    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:29:36.880701 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879820    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:29:36.880701 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879823    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:29:36.880701 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879827    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:29:36.880701 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879830    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:29:36.880701 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879833    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:29:36.880701 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879836    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:29:36.880701 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879838    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:29:36.880701 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879841    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:29:36.880701 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879844    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:29:36.880701 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879847    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:29:36.880701 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879850    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:29:36.881214 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879853    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:29:36.881214 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879855    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:29:36.881214 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879858    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:29:36.881214 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879872    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:29:36.881214 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879876    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:29:36.881214 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879880    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:29:36.881214 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879883    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:29:36.881214 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879887    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:29:36.881214 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879891    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:29:36.881214 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879894    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:29:36.881214 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879897    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:29:36.881214 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879900    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:29:36.881214 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879903    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:29:36.881214 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879905    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:29:36.881214 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879908    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:29:36.881214 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879911    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:29:36.881214 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879914    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:29:36.881214 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879916    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:29:36.881214 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879919    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:29:36.881673 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879922    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:29:36.881673 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879930    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:29:36.881673 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879933    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:29:36.881673 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879936    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:29:36.881673 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879938    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:29:36.881673 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879941    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:29:36.881673 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879945    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:29:36.881673 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879947    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:29:36.881673 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879950    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:29:36.881673 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879953    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:29:36.881673 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879956    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:29:36.881673 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879958    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:29:36.881673 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879961    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:29:36.881673 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879964    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:29:36.881673 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879966    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:29:36.881673 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879969    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:29:36.881673 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879972    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:29:36.881673 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.879974    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:29:36.882139 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882126    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:29:36.882139 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882138    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:29:36.882194 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882142    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:29:36.882194 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882146    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:29:36.882194 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882149    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:29:36.882194 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882151    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:29:36.882194 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882154    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:29:36.882194 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882157    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:29:36.882194 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882161    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:29:36.882194 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882165    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:29:36.882194 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882168    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:29:36.882194 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882171    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:29:36.882194 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882174    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:29:36.882194 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882177    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:29:36.882194 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882179    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:29:36.882194 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882182    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:29:36.882194 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882190    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:29:36.882194 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882193    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:29:36.882194 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882196    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:29:36.882194 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882198    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:29:36.882194 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882201    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:29:36.882661 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882204    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:29:36.882661 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882207    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:29:36.882661 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882210    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:29:36.882661 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882212    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:29:36.882661 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882215    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:29:36.882661 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882217    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:29:36.882661 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882220    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:29:36.882661 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882223    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:29:36.882661 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882225    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:29:36.882661 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882228    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:29:36.882661 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882230    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:29:36.882661 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882233    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:29:36.882661 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882235    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:29:36.882661 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882238    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:29:36.882661 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882241    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:29:36.882661 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882243    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:29:36.882661 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882246    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:29:36.882661 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882248    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:29:36.882661 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882251    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:29:36.882661 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882253    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:29:36.883179 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882255    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:29:36.883179 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882258    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:29:36.883179 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882260    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:29:36.883179 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882263    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:29:36.883179 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882265    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:29:36.883179 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882268    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:29:36.883179 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882270    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:29:36.883179 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882273    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:29:36.883179 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882276    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:29:36.883179 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882278    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:29:36.883179 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882281    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:29:36.883179 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882284    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:29:36.883179 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882286    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:29:36.883179 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882293    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:29:36.883179 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882296    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:29:36.883179 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882300    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:29:36.883179 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882303    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:29:36.883179 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882307    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:29:36.883179 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882310    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:29:36.883179 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882313    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:29:36.883686 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882315    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:29:36.883686 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882318    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:29:36.883686 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882320    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:29:36.883686 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882323    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:29:36.883686 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882326    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:29:36.883686 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882328    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:29:36.883686 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882331    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:29:36.883686 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882333    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:29:36.883686 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882336    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:29:36.883686 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882338    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:29:36.883686 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882341    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:29:36.883686 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882344    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:29:36.883686 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882346    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:29:36.883686 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882349    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:29:36.883686 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882351    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:29:36.883686 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882354    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:29:36.883686 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882356    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:29:36.883686 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882359    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:29:36.883686 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882361    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:29:36.883686 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882364    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:29:36.884185 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882366    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:29:36.884185 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882369    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:29:36.884185 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882372    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:29:36.884185 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882374    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:29:36.884185 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882377    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:29:36.884185 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882380    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:29:36.884185 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882383    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:29:36.884185 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882385    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:29:36.884185 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882388    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:29:36.884185 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882392    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:29:36.884185 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882394    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:29:36.884185 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882397    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:29:36.884185 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882401    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:29:36.884185 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882405    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:29:36.884185 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882408    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:29:36.884185 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882410    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:29:36.884185 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882413    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:29:36.884185 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882416    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:29:36.884185 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882822    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:29:36.884656 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882827    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:29:36.884656 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882830    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:29:36.884656 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882833    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:29:36.884656 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882836    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:29:36.884656 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882840    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:29:36.884656 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882842    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:29:36.884656 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882845    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:29:36.884656 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882848    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:29:36.884656 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882851    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:29:36.884656 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882853    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:29:36.884656 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882856    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:29:36.884656 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882874    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:29:36.884656 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882878    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:29:36.884656 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882881    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:29:36.884656 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882885    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:29:36.884656 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882888    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:29:36.884656 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882891    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:29:36.884656 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882894    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:29:36.884656 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882897    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:29:36.884656 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882900    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:29:36.885172 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882903    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:29:36.885172 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882906    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:29:36.885172 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882909    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:29:36.885172 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882913    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:29:36.885172 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882916    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:29:36.885172 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882919    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:29:36.885172 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882922    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:29:36.885172 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882925    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:29:36.885172 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882928    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:29:36.885172 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882931    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:29:36.885172 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882934    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:29:36.885172 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882936    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:29:36.885172 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882939    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:29:36.885172 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882942    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:29:36.885172 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882944    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:29:36.885172 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882947    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:29:36.885172 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882949    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:29:36.885172 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882952    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:29:36.885172 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882955    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:29:36.885641 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882957    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:29:36.885641 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882960    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:29:36.885641 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882962    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:29:36.885641 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882965    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:29:36.885641 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882968    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:29:36.885641 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882970    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:29:36.885641 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882973    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:29:36.885641 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882975    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:29:36.885641 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882978    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:29:36.885641 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882982    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:29:36.885641 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882984    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:29:36.885641 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882987    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:29:36.885641 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882990    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:29:36.885641 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882993    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:29:36.885641 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882996    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:29:36.885641 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.882999    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:29:36.885641 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883001    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:29:36.885641 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883004    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:29:36.885641 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883007    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:29:36.886132 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883009    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:29:36.886132 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883012    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:29:36.886132 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883014    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:29:36.886132 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883017    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:29:36.886132 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883020    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:29:36.886132 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883022    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:29:36.886132 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883025    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:29:36.886132 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883027    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:29:36.886132 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883029    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:29:36.886132 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883032    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:29:36.886132 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883035    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:29:36.886132 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883037    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:29:36.886132 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883041    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:29:36.886132 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883045    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:29:36.886132 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883048    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:29:36.886132 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883051    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:29:36.886132 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883054    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:29:36.886132 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883057    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:29:36.886132 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883061    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:29:36.886132 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883063    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:29:36.886657 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883066    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:29:36.886657 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883068    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:29:36.886657 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883072    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:29:36.886657 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883074    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:29:36.886657 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883080    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:29:36.886657 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883082    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:29:36.886657 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883085    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:29:36.886657 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883087    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:29:36.886657 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883090    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:29:36.886657 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883093    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:29:36.886657 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883095    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:29:36.886657 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883098    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:29:36.886657 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883101    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:29:36.886657 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883103    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:29:36.886657 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883106    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:29:36.886657 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883109    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:29:36.886657 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883111    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:29:36.886657 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883114    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:29:36.886657 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883116    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:29:36.886657 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883119    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:29:36.887161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883185    2556 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 15:29:36.887161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883221    2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 15:29:36.887161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883228    2556 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 15:29:36.887161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883233    2556 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 15:29:36.887161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883237    2556 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 15:29:36.887161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883240    2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 15:29:36.887161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883245    2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 15:29:36.887161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883249    2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 15:29:36.887161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883252    2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 15:29:36.887161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883255    2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 15:29:36.887161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883260    2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 15:29:36.887161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883263    2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 15:29:36.887161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883267    2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 15:29:36.887161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883270    2556 flags.go:64] FLAG: --cgroup-root=""
Sep 17 15:29:36.887161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883273    2556 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 15:29:36.887161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883277    2556 flags.go:64] FLAG: --client-ca-file=""
Sep 17 15:29:36.887161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883280    2556 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 15:29:36.887161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883283    2556 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 15:29:36.887161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883288    2556 flags.go:64] FLAG: --cluster-domain=""
Sep 17 15:29:36.887161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883291    2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 15:29:36.887161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883294    2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 15:29:36.887161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883297    2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 15:29:36.887161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883300    2556 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 15:29:36.887161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883303    2556 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 15:29:36.887777 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883306    2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 15:29:36.887777 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883309    2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 15:29:36.887777 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883313    2556 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 15:29:36.887777 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883316    2556 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 15:29:36.887777 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883319    2556 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 15:29:36.887777 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883322    2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 15:29:36.887777 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883325    2556 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 15:29:36.887777 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883328    2556 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 15:29:36.887777 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883332    2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 15:29:36.887777 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883335    2556 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 15:29:36.887777 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883338    2556 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 15:29:36.887777 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883341    2556 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 15:29:36.887777 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883344    2556 flags.go:64] FLAG: --enable-server="true"
Sep 17 15:29:36.887777 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883347    2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 15:29:36.887777 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883352    2556 flags.go:64] FLAG: --event-burst="100"
Sep 17 15:29:36.887777 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883356    2556 flags.go:64] FLAG: --event-qps="50"
Sep 17 15:29:36.887777 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883360    2556 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 15:29:36.887777 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883363    2556 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 15:29:36.887777 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883366    2556 flags.go:64] FLAG: --eviction-hard=""
Sep 17 15:29:36.887777 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883369    2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 15:29:36.887777 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883372    2556 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 15:29:36.887777 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883376    2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 15:29:36.887777 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883379    2556 flags.go:64] FLAG: --eviction-soft=""
Sep 17 15:29:36.887777 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883382    2556 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 15:29:36.887777 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883386    2556 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 15:29:36.888455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883389    2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 15:29:36.888455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883392    2556 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 15:29:36.888455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883395    2556 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 15:29:36.888455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883399    2556 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 15:29:36.888455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883402    2556 flags.go:64] FLAG: --feature-gates=""
Sep 17 15:29:36.888455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883406    2556 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 15:29:36.888455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883409    2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 15:29:36.888455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883412    2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 15:29:36.888455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883415    2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 15:29:36.888455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883427    2556 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 15:29:36.888455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883431    2556 flags.go:64] FLAG: --help="false"
Sep 17 15:29:36.888455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883434    2556 flags.go:64] FLAG: --hostname-override="ip-10-0-134-104.ec2.internal"
Sep 17 15:29:36.888455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883437    2556 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 15:29:36.888455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883440    2556 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 15:29:36.888455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883443    2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 15:29:36.888455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883447    2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 15:29:36.888455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883450    2556 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 15:29:36.888455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883454    2556 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 15:29:36.888455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883456    2556 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 15:29:36.888455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883459    2556 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 15:29:36.888455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883462    2556 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 15:29:36.888455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883465    2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 15:29:36.888455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883468    2556 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 15:29:36.888455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883471    2556 flags.go:64] FLAG: --kube-reserved=""
Sep 17 15:29:36.889085 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883474    2556 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 15:29:36.889085 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883477    2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 15:29:36.889085 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883480    2556 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 15:29:36.889085 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883483    2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 15:29:36.889085 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883486    2556 flags.go:64] FLAG: --lock-file=""
Sep 17 15:29:36.889085 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883491    2556 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 15:29:36.889085 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883494    2556 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 15:29:36.889085 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883497    2556 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 15:29:36.889085 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883503    2556 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 15:29:36.889085 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883507    2556 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 15:29:36.889085 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883510    2556 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 15:29:36.889085 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883513    2556 flags.go:64] FLAG: --logging-format="text"
Sep 17 15:29:36.889085 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883516    2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 15:29:36.889085 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883519    2556 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 15:29:36.889085 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883522    2556 flags.go:64] FLAG: --manifest-url=""
Sep 17 15:29:36.889085 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883524    2556 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 15:29:36.889085 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883529    2556 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 15:29:36.889085 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883532    2556 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 15:29:36.889085 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883536    2556 flags.go:64] FLAG: --max-pods="110"
Sep 17 15:29:36.889085 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883539    2556 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 15:29:36.889085 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883542    2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 15:29:36.889085 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883545    2556 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 15:29:36.889085 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883548    2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 15:29:36.889085 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883551    2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 15:29:36.889085 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883554    2556 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 15:29:36.889790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883558    2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 15:29:36.889790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883565    2556 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 15:29:36.889790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883568    2556 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 15:29:36.889790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883571    2556 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 15:29:36.889790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883574    2556 flags.go:64] FLAG: --pod-cidr=""
Sep 17 15:29:36.889790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883577    2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 17 15:29:36.889790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883583    2556 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 15:29:36.889790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883586    2556 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 15:29:36.889790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883589    2556 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 15:29:36.889790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883592    2556 flags.go:64] FLAG: --port="10250"
Sep 17 15:29:36.889790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883595    2556 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 15:29:36.889790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883598    2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0eb0ba30b1f12b220"
Sep 17 15:29:36.889790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883603    2556 flags.go:64] FLAG: --qos-reserved=""
Sep 17 15:29:36.889790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883606    2556 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 15:29:36.889790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883609    2556 flags.go:64] FLAG: --register-node="true"
Sep 17 15:29:36.889790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883612    2556 flags.go:64] FLAG: --register-with-taints=""
Sep 17 15:29:36.889790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883617    2556 flags.go:64] FLAG: --registry-burst="10"
Sep 17 15:29:36.889790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883620    2556 flags.go:64] FLAG: --registry-qps="5"
Sep 17 15:29:36.889790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883623    2556 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 15:29:36.889790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883625    2556 flags.go:64] FLAG: --reserved-memory=""
Sep 17 15:29:36.889790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883643    2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 15:29:36.889790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883647    2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 15:29:36.889790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883651    2556 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 15:29:36.889790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883654    2556 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 15:29:36.889790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883657    2556 flags.go:64] FLAG: --runonce="false"
Sep 17 15:29:36.890420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883660    2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 15:29:36.890420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883663    2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 15:29:36.890420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883666    2556 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 15:29:36.890420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883669    2556 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 15:29:36.890420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883672    2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 15:29:36.890420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883675    2556 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 15:29:36.890420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883678    2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 15:29:36.890420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883681    2556 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 15:29:36.890420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883684    2556 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 15:29:36.890420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883687    2556 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 15:29:36.890420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883692    2556 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 15:29:36.890420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883695    2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 15:29:36.890420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883699    2556 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 15:29:36.890420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883702    2556 flags.go:64] FLAG: --system-cgroups=""
Sep 17 15:29:36.890420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883705    2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 15:29:36.890420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883710    2556 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 15:29:36.890420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883713    2556 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 15:29:36.890420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883716    2556 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 15:29:36.890420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883720    2556 flags.go:64] FLAG: --tls-min-version=""
Sep 17 15:29:36.890420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883725    2556 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 15:29:36.890420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883728    2556 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 15:29:36.890420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883731    2556 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 15:29:36.890420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883734    2556 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 15:29:36.890420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883737    2556 flags.go:64] FLAG: --v="2"
Sep 17 15:29:36.890420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883742    2556 flags.go:64] FLAG: --version="false"
Sep 17 15:29:36.891339 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883758    2556 flags.go:64] FLAG: --vmodule=""
Sep 17 15:29:36.891339 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883762    2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 15:29:36.891339 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.883765    2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 15:29:36.891339 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883896    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:29:36.891339 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883901    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:29:36.891339 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883904    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:29:36.891339 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883908    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:29:36.891339 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883911    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:29:36.891339 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883914    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:29:36.891339 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883917    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:29:36.891339 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883920    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:29:36.891339 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883923    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:29:36.891339 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883925    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:29:36.891339 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883928    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:29:36.891339 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883931    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:29:36.891339 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883934    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:29:36.891339 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883936    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:29:36.891339 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883941    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:29:36.891339 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883944    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:29:36.891339 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883947    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:29:36.892142 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883950    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:29:36.892142 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883953    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:29:36.892142 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883956    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:29:36.892142 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883959    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:29:36.892142 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883961    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:29:36.892142 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883964    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:29:36.892142 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883968    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:29:36.892142 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883971    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:29:36.892142 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883973    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:29:36.892142 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883977    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:29:36.892142 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883981    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:29:36.892142 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883984    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:29:36.892142 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883988    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:29:36.892142 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883991    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:29:36.892142 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883994    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:29:36.892142 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.883998    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:29:36.892142 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884001    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:29:36.892142 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884004    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:29:36.892142 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884006    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:29:36.892683 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884009    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:29:36.892683 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884013    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:29:36.892683 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884017    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:29:36.892683 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884020    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:29:36.892683 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884022    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:29:36.892683 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884025    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:29:36.892683 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884028    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:29:36.892683 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884030    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:29:36.892683 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884033    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:29:36.892683 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884036    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:29:36.892683 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884040    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:29:36.892683 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884044    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:29:36.892683 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884048    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:29:36.892683 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884052    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:29:36.892683 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884054    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:29:36.892683 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884057    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:29:36.892683 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884060    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:29:36.892683 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884063    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:29:36.892683 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884066    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:29:36.893173 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884070    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:29:36.893173 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884073    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:29:36.893173 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884075    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:29:36.893173 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884078    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:29:36.893173 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884081    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:29:36.893173 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884084    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:29:36.893173 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884086    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:29:36.893173 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884089    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:29:36.893173 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884092    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:29:36.893173 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884094    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:29:36.893173 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884097    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:29:36.893173 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884099    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:29:36.893173 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884102    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:29:36.893173 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884105    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:29:36.893173 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884108    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:29:36.893173 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884110    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:29:36.893173 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884113    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:29:36.893173 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884116    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:29:36.893173 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884118    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:29:36.893173 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884121    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:29:36.893662 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884124    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:29:36.893662 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884127    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:29:36.893662 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884129    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:29:36.893662 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884134    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:29:36.893662 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884137    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:29:36.893662 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884140    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:29:36.893662 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884143    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:29:36.893662 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884146    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:29:36.893662 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884148    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:29:36.893662 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884151    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:29:36.893662 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884154    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:29:36.893662 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884156    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:29:36.893662 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884160    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:29:36.893662 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884163    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:29:36.893662 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884166    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:29:36.893662 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884169    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:29:36.893662 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884171    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:29:36.893662 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884174    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:29:36.893662 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884176    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:29:36.893662 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884179    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:29:36.894265 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884186    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:29:36.894265 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884189    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:29:36.894265 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884192    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:29:36.894265 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.884195    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:29:36.894265 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.884203    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:29:36.894265 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.893544    2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 15:29:36.894265 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.893563    2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 15:29:36.894455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.894404    2556 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 15:29:36.894455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.894412    2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 15:29:36.894508 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894474    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:29:36.894508 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894480    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:29:36.894508 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894484    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:29:36.894508 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894488    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:29:36.894508 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894492    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:29:36.894508 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894496    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:29:36.894508 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894500    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:29:36.894508 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894503    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:29:36.894508 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894506    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:29:36.894508 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894509    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:29:36.894508 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894512    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:29:36.894784 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894515    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:29:36.894784 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894518    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:29:36.894784 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894521    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:29:36.894784 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894524    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:29:36.894784 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894527    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:29:36.894784 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894529    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:29:36.894784 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894532    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:29:36.894784 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894535    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:29:36.894784 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894538    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:29:36.894784 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894541    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:29:36.894784 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894544    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:29:36.894784 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894547    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:29:36.894784 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894550    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:29:36.894784 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894553    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:29:36.894784 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894555    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:29:36.894784 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894559    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:29:36.894784 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894562    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:29:36.894784 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894565    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:29:36.894784 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894568    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:29:36.895266 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894571    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:29:36.895266 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894574    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:29:36.895266 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894577    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:29:36.895266 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894580    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:29:36.895266 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894583    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:29:36.895266 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894586    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:29:36.895266 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894588    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:29:36.895266 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894591    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:29:36.895266 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894594    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:29:36.895266 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894596    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:29:36.895266 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894599    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:29:36.895266 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894602    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:29:36.895266 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894604    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:29:36.895266 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894607    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:29:36.895266 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894612    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:29:36.895266 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894616    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:29:36.895266 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894619    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:29:36.895266 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894621    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:29:36.895266 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894624    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:29:36.895266 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894627    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:29:36.895761 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894630    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:29:36.895761 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894633    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:29:36.895761 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894635    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:29:36.895761 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894638    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:29:36.895761 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894641    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:29:36.895761 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894644    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:29:36.895761 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894646    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:29:36.895761 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894649    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:29:36.895761 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894652    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:29:36.895761 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894655    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:29:36.895761 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894657    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:29:36.895761 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894660    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:29:36.895761 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894663    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:29:36.895761 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894666    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:29:36.895761 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894668    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:29:36.895761 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894671    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:29:36.895761 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894674    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:29:36.895761 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894676    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:29:36.895761 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894679    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:29:36.896282 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894682    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:29:36.896282 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894684    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:29:36.896282 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894687    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:29:36.896282 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894690    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:29:36.896282 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894693    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:29:36.896282 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894695    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:29:36.896282 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894698    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:29:36.896282 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894702    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:29:36.896282 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894705    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:29:36.896282 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894708    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:29:36.896282 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894711    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:29:36.896282 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894714    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:29:36.896282 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894717    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:29:36.896282 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894720    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:29:36.896282 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894722    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:29:36.896282 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894725    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:29:36.896282 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894728    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:29:36.896282 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894731    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:29:36.896282 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894734    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:29:36.896282 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894736    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:29:36.898013 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894739    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:29:36.898013 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894742    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:29:36.898013 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894744    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:29:36.898013 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894747    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:29:36.898013 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894749    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:29:36.898013 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894752    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:29:36.898013 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894755    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:29:36.898013 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894758    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:29:36.898013 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894761    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:29:36.898013 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894763    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:29:36.898013 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.894768    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:29:36.898013 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894899    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:29:36.898013 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894905    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:29:36.898013 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894908    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:29:36.898013 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894913    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:29:36.898392 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894917    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:29:36.898392 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894920    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:29:36.898392 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894923    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:29:36.898392 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894926    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:29:36.898392 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894929    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:29:36.898392 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894933    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:29:36.898392 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894936    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:29:36.898392 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894939    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:29:36.898392 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894943    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:29:36.898392 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894946    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:29:36.898392 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894949    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:29:36.898392 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894951    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:29:36.898392 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894954    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:29:36.898392 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894957    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:29:36.898392 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894960    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:29:36.898392 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894962    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:29:36.898392 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894965    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:29:36.898392 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894968    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:29:36.898392 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894970    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:29:36.898392 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894973    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:29:36.898891 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894975    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:29:36.898891 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894978    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:29:36.898891 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894981    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:29:36.898891 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894984    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:29:36.898891 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894987    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:29:36.898891 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894989    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:29:36.898891 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894992    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:29:36.898891 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894994    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:29:36.898891 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.894997    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:29:36.898891 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895000    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:29:36.898891 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895003    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:29:36.898891 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895005    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:29:36.898891 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895008    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:29:36.898891 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895011    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:29:36.898891 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895013    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:29:36.898891 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895016    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:29:36.898891 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895020    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:29:36.898891 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895023    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:29:36.898891 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895027    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:29:36.898891 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895029    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:29:36.899377 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895032    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:29:36.899377 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895037    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:29:36.899377 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895040    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:29:36.899377 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895043    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:29:36.899377 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895046    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:29:36.899377 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895048    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:29:36.899377 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895051    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:29:36.899377 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895055    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:29:36.899377 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895058    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:29:36.899377 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895061    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:29:36.899377 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895063    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:29:36.899377 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895066    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:29:36.899377 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895069    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:29:36.899377 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895071    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:29:36.899377 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895074    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:29:36.899377 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895077    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:29:36.899377 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895079    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:29:36.899377 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895082    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:29:36.899377 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895085    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:29:36.899843 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895087    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:29:36.899843 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895090    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:29:36.899843 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895092    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:29:36.899843 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895095    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:29:36.899843 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895097    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:29:36.899843 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895101    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:29:36.899843 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895104    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:29:36.899843 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895108    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:29:36.899843 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895112    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:29:36.899843 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895114    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:29:36.899843 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895117    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:29:36.899843 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895120    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:29:36.899843 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895123    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:29:36.899843 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895126    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:29:36.899843 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895129    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:29:36.899843 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895133    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:29:36.899843 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895135    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:29:36.899843 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895138    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:29:36.899843 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895141    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:29:36.899843 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895143    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:29:36.900344 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895146    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:29:36.900344 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895148    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:29:36.900344 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895151    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:29:36.900344 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895154    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:29:36.900344 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895157    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:29:36.900344 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895159    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:29:36.900344 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895162    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:29:36.900344 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895164    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:29:36.900344 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895167    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:29:36.900344 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895170    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:29:36.900344 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895172    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:29:36.900344 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895175    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:29:36.900344 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895178    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:29:36.900344 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895180    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:29:36.900344 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895183    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:29:36.900344 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:36.895185    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:29:36.900729 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.895191    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:29:36.900729 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.895310    2556 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 15:29:36.900729 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.898255    2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 15:29:36.900729 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.899625    2556 server.go:1019] "Starting client certificate rotation"
Sep 17 15:29:36.900729 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.899723    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 15:29:36.900729 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.899765    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 15:29:36.902493 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.902478    2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 15:29:36.908049 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.908026    2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 15:29:36.916555 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.916534    2556 log.go:25] "Validated CRI v1 runtime API"
Sep 17 15:29:36.921266 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.921246    2556 log.go:25] "Validated CRI v1 image API"
Sep 17 15:29:36.922519 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.922413    2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 15:29:36.926511 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.926481    2556 fs.go:135] Filesystem UUIDs: map[13a11c96-b30f-439f-bf9d-2723323458d2:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a7746fa1-43b3-4ee0-92a9-683826b706fd:/dev/nvme0n1p3]
Sep 17 15:29:36.926511 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.926506    2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 15:29:36.932760 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.932624    2556 manager.go:216] Machine: {Timestamp:2026-09-17 15:29:36.930197768 +0000 UTC m=+0.366412674 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098386 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e3f1733db707fbb9de24bd8edd8c6 SystemUUID:ec2e3f17-33db-707f-bb9d-e24bd8edd8c6 BootID:5fced799-d9d2-4652-84a1-f538da38adeb Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fa:3a:b3:01 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fa:3a:b3:01 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:71:69:91:f2:85 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 15:29:36.932760 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.932752    2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 15:29:36.932906 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.932856    2556 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 15:29:36.933715 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.933697    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 15:29:36.936228 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.936198    2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 15:29:36.936380 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.936234    2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-104.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 15:29:36.937712 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.937700    2556 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 15:29:36.937755 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.937715    2556 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 15:29:36.937755 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.937724    2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 15:29:36.938426 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.938415    2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 15:29:36.939608 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.939598    2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 15:29:36.943091 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.943079    2556 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 15:29:36.943238 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.943228    2556 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 15:29:36.943280 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.943259    2556 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 15:29:36.943280 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.943270    2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 15:29:36.943349 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.943285    2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 15:29:36.943349 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.943296    2556 kubelet.go:398] "Adding apiserver pod source"
Sep 17 15:29:36.943349 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.943306    2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 15:29:36.943816 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.943800    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 15:29:36.943878 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.943822    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 15:29:36.946554 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.946536    2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 17 15:29:36.949798 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.949783    2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 15:29:36.949888 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.949813    2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 15:29:36.951610 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.951597    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 15:29:36.951682 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.951614    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 15:29:36.951682 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.951620    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 15:29:36.951682 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.951625    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 15:29:36.951682 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.951649    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 15:29:36.951682 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.951659    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 15:29:36.951682 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.951677    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 15:29:36.951682 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.951686    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 15:29:36.951997 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.951694    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 15:29:36.951997 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.951700    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 15:29:36.951997 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.951718    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 15:29:36.951997 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.951724    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 15:29:36.952593 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.952581    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 15:29:36.952593 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.952591    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 15:29:36.953601 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:36.953563    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-104.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 15:29:36.953994 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:36.953968    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 15:29:36.955893 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.955878    2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-104.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 15:29:36.956692 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.956680    2556 server.go:1268] "Started kubelet"
Sep 17 15:29:36.956782 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.956758    2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 15:29:36.956834 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.956813    2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 15:29:36.956893 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.956848    2556 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 15:29:36.958237 ip-10-0-134-104 systemd[1]: Started Kubernetes Kubelet.
Sep 17 15:29:36.958641 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.958621    2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 15:29:36.964296 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.964276    2556 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 15:29:36.965436 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:36.965389    2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 15:29:36.967280 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:36.966161    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-104.ec2.internal.18d6255b8697022c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-104.ec2.internal,UID:ip-10-0-134-104.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-104.ec2.internal,},FirstTimestamp:2026-09-17 15:29:36.956645932 +0000 UTC m=+0.392860838,LastTimestamp:2026-09-17 15:29:36.956645932 +0000 UTC m=+0.392860838,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-104.ec2.internal,}"
Sep 17 15:29:36.972571 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.972552    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 15:29:36.973361 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.973348    2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 15:29:36.973495 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:36.973473    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-104.ec2.internal\" not found"
Sep 17 15:29:36.974213 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.974193    2556 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 15:29:36.974213 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.974214    2556 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 15:29:36.974383 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.974193    2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 15:29:36.974383 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.974342    2556 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 15:29:36.974383 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.974348    2556 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 15:29:36.974969 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.974954    2556 factory.go:55] Registering systemd factory
Sep 17 15:29:36.975051 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.974973    2556 factory.go:223] Registration of the systemd container factory successfully
Sep 17 15:29:36.975209 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.975194    2556 factory.go:153] Registering CRI-O factory
Sep 17 15:29:36.975280 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.975212    2556 factory.go:223] Registration of the crio container factory successfully
Sep 17 15:29:36.975332 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.975289    2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 15:29:36.975332 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.975315    2556 factory.go:103] Registering Raw factory
Sep 17 15:29:36.975430 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:36.975323    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 15:29:36.975430 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.975341    2556 manager.go:1196] Started watching for new ooms in manager
Sep 17 15:29:36.975430 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:36.975351    2556 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-104.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 15:29:36.975663 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.975651    2556 manager.go:349] Starting recovery of all containers
Sep 17 15:29:36.986268 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.986091    2556 manager.go:354] Recovery completed
Sep 17 15:29:36.987139 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.987119    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w829k"
Sep 17 15:29:36.990978 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.990790    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:29:36.992717 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.992699    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-104.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:29:36.992766 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.992732    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-104.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:29:36.992766 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.992746    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-104.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:29:36.993397 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.993383    2556 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 15:29:36.993477 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.993404    2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 15:29:36.993477 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.993412    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:29:36.993477 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.993428    2556 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 15:29:36.994791 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.994774    2556 policy_none.go:49] "None policy: Start"
Sep 17 15:29:36.994883 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.994805    2556 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 15:29:36.994883 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.994819    2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 15:29:36.995038 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.995021    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-104.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:29:36.995097 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.995056    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-104.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:29:36.995097 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.995070    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-104.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:29:36.995715 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.995697    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w829k"
Sep 17 15:29:36.997016 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:36.997001    2556 policy_none.go:47] "Start"
Sep 17 15:29:37.026542 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.026524    2556 manager.go:337] "Starting Device Plugin manager"
Sep 17 15:29:37.050109 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:37.026569    2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 15:29:37.050109 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.026582    2556 server.go:87] "Starting device plugin registration server"
Sep 17 15:29:37.050109 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.026981    2556 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 15:29:37.050109 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.026993    2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 15:29:37.050109 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.027093    2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 15:29:37.050109 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.027174    2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 15:29:37.050109 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.027182    2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 15:29:37.050109 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:37.027960    2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 15:29:37.050109 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:37.028023    2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-104.ec2.internal\" not found"
Sep 17 15:29:37.071336 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.071301    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 15:29:37.072733 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.072716    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 15:29:37.072841 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.072791    2556 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 15:29:37.072841 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.072820    2556 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 15:29:37.072975 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:37.072945    2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 15:29:37.075337 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.075317    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:29:37.127643 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.127539    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:29:37.128638 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.128611    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-104.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:29:37.128761 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.128647    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-104.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:29:37.128761 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.128658    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-104.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:29:37.128761 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.128683    2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-104.ec2.internal"
Sep 17 15:29:37.136213 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.136197    2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-104.ec2.internal"
Sep 17 15:29:37.136286 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:37.136220    2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-104.ec2.internal\": node \"ip-10-0-134-104.ec2.internal\" not found"
Sep 17 15:29:37.166672 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:37.166644    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-104.ec2.internal\" not found"
Sep 17 15:29:37.173698 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.173672    2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-104.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-104.ec2.internal"]
Sep 17 15:29:37.173780 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.173769    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:29:37.176232 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.176215    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-104.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:29:37.176336 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.176249    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-104.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:29:37.176336 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.176263    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-104.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:29:37.178526 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.178511    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:29:37.178657 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.178631    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-104.ec2.internal"
Sep 17 15:29:37.178706 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.178663    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:29:37.179333 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.179317    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-104.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:29:37.179412 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.179349    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-104.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:29:37.179412 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.179363    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-104.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:29:37.179412 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.179320    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-104.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:29:37.179570 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.179427    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-104.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:29:37.179570 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.179438    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-104.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:29:37.181636 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.181621    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-104.ec2.internal"
Sep 17 15:29:37.181714 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.181652    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:29:37.182405 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.182391    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-104.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:29:37.182485 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.182420    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-104.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:29:37.182485 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.182433    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-104.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:29:37.204644 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:37.204619    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-104.ec2.internal\" not found" node="ip-10-0-134-104.ec2.internal"
Sep 17 15:29:37.209004 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:37.208984    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-104.ec2.internal\" not found" node="ip-10-0-134-104.ec2.internal"
Sep 17 15:29:37.267184 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:37.267159    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-104.ec2.internal\" not found"
Sep 17 15:29:37.275527 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.275500    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/623939e1a7296f8c7c9abdfb3dac6ecc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-104.ec2.internal\" (UID: \"623939e1a7296f8c7c9abdfb3dac6ecc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-104.ec2.internal"
Sep 17 15:29:37.275633 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.275535    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/623939e1a7296f8c7c9abdfb3dac6ecc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-104.ec2.internal\" (UID: \"623939e1a7296f8c7c9abdfb3dac6ecc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-104.ec2.internal"
Sep 17 15:29:37.275633 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.275562    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4965a8a85b93317838f07968164cf5cd-config\") pod \"kube-apiserver-proxy-ip-10-0-134-104.ec2.internal\" (UID: \"4965a8a85b93317838f07968164cf5cd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-104.ec2.internal"
Sep 17 15:29:37.368198 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:37.368150    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-104.ec2.internal\" not found"
Sep 17 15:29:37.376529 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.376508    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/623939e1a7296f8c7c9abdfb3dac6ecc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-104.ec2.internal\" (UID: \"623939e1a7296f8c7c9abdfb3dac6ecc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-104.ec2.internal"
Sep 17 15:29:37.376634 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.376541    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4965a8a85b93317838f07968164cf5cd-config\") pod \"kube-apiserver-proxy-ip-10-0-134-104.ec2.internal\" (UID: \"4965a8a85b93317838f07968164cf5cd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-104.ec2.internal"
Sep 17 15:29:37.376634 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.376562    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/623939e1a7296f8c7c9abdfb3dac6ecc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-104.ec2.internal\" (UID: \"623939e1a7296f8c7c9abdfb3dac6ecc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-104.ec2.internal"
Sep 17 15:29:37.376724 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.376633    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/623939e1a7296f8c7c9abdfb3dac6ecc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-104.ec2.internal\" (UID: \"623939e1a7296f8c7c9abdfb3dac6ecc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-104.ec2.internal"
Sep 17 15:29:37.376724 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.376651    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/623939e1a7296f8c7c9abdfb3dac6ecc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-104.ec2.internal\" (UID: \"623939e1a7296f8c7c9abdfb3dac6ecc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-104.ec2.internal"
Sep 17 15:29:37.376724 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.376633    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4965a8a85b93317838f07968164cf5cd-config\") pod \"kube-apiserver-proxy-ip-10-0-134-104.ec2.internal\" (UID: \"4965a8a85b93317838f07968164cf5cd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-104.ec2.internal"
Sep 17 15:29:37.469061 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:37.468975    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-104.ec2.internal\" not found"
Sep 17 15:29:37.507555 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.507526    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-104.ec2.internal"
Sep 17 15:29:37.511392 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:37.511370    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-104.ec2.internal"
Sep 17 15:29:37.570102 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:37.570047    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-104.ec2.internal\" not found"
Sep 17 15:29:38.726863 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:38.726816    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-104.ec2.internal\" not found"
Sep 17 15:29:38.827486 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:38.827409    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-104.ec2.internal\" not found"
Sep 17 15:29:38.879827 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:38.879796    2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:29:38.930133 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:38.930106    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-104.ec2.internal"
Sep 17 15:29:38.948712 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:38.948691    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 15:29:38.950950 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:38.950936    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-104.ec2.internal"
Sep 17 15:29:38.954752 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:38.954733    2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 15:29:38.954875 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:38.954857    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 15:29:38.954919 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:38.954890    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 15:29:38.954951 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:38.954930    2556 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://aa2a2cd1a3d4e43d0bde9fda70939a95-889843ab872114f7.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-machine-config-operator/pods\": read tcp 10.0.134.104:49090->54.226.235.184:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-104.ec2.internal"
Sep 17 15:29:38.999643 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:38.999611    2556 apiserver.go:52] "Watching apiserver"
Sep 17 15:29:39.009572 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.009547    2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:29:39.015941 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.015912    2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 15:29:39.017244 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.017222    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-wxm49","kube-system/global-pull-secret-syncer-r5jgb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r","openshift-image-registry/node-ca-s9svr","openshift-multus/multus-lkm6c","openshift-network-diagnostics/network-check-target-l2rxt","openshift-ovn-kubernetes/ovnkube-node-xjz9c","kube-system/konnectivity-agent-nh9lk","kube-system/kube-apiserver-proxy-ip-10-0-134-104.ec2.internal","openshift-cluster-node-tuning-operator/tuned-857mz","openshift-dns/node-resolver-xfzrp","openshift-multus/multus-additional-cni-plugins-cmrtd","openshift-multus/network-metrics-daemon-hn89v"]
Sep 17 15:29:39.019604 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.019587    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wxm49"
Sep 17 15:29:39.021574 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.021554    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:29:39.021669 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:39.021627    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5jgb" podUID="84728120-5cde-4b79-944d-7554db40cf41"
Sep 17 15:29:39.022331 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.022317    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:29:39.022384 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.022354    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 15:29:39.022578 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.022565    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sncnc\""
Sep 17 15:29:39.022657 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.022642    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 15:29:39.024400 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.024379    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r"
Sep 17 15:29:39.026152 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.026130    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s9svr"
Sep 17 15:29:39.027908 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.027892    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 15:29:39.028143 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.028128    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 15:29:39.028224 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.028208    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 15:29:39.028296 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.028282    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.029205 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.029187    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 15:29:39.030299 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.030282    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:29:39.030385 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:39.030346    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2rxt" podUID="352c01b0-7c02-4da9-a311-816815cd5e1e"
Sep 17 15:29:39.030559 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.030544    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q562k\""
Sep 17 15:29:39.030636 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.030613    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 15:29:39.030691 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.030665    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 15:29:39.031400 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.031374    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 15:29:39.031554 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.031523    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 15:29:39.031554 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.031533    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 15:29:39.031681 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.031631    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dclrc\""
Sep 17 15:29:39.031743 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.031700    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 15:29:39.031814 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.031799    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 15:29:39.031857 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.031834    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7fpxx\""
Sep 17 15:29:39.032614 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.032599    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.034705 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.034683    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nh9lk"
Sep 17 15:29:39.036401 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.036383    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-multus-cni-dir\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.036528 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.036406    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-host-var-lib-kubelet\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.036528 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.036422    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-host-cni-bin\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.036528 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.036436    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-host-cni-netd\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.036528 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.036453    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6dfl\" (UniqueName: \"kubernetes.io/projected/6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe-kube-api-access-f6dfl\") pod \"aws-ebs-csi-driver-node-ftp6r\" (UID: \"6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r"
Sep 17 15:29:39.036528 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.036493    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/872b5c05-9984-45aa-82ab-6ab103c385d9-multus-daemon-config\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.036528 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.036523    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plpr7\" (UniqueName: \"kubernetes.io/projected/352c01b0-7c02-4da9-a311-816815cd5e1e-kube-api-access-plpr7\") pod \"network-check-target-l2rxt\" (UID: \"352c01b0-7c02-4da9-a311-816815cd5e1e\") " pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:29:39.036925 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.036558    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-run-ovn\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.036925 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.036578    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-host-run-ovn-kubernetes\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.036925 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.036594    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-ovn-node-metrics-cert\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.036925 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.036611    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l68tb\" (UniqueName: \"kubernetes.io/projected/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-kube-api-access-l68tb\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.036925 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.036644    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/84728120-5cde-4b79-944d-7554db40cf41-dbus\") pod \"global-pull-secret-syncer-r5jgb\" (UID: \"84728120-5cde-4b79-944d-7554db40cf41\") " pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:29:39.036925 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.036666    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-multus-socket-dir-parent\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.036925 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.036682    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-host-var-lib-cni-multus\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.036925 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.036708    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-multus-conf-dir\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.036925 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.036756    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsptt\" (UniqueName: \"kubernetes.io/projected/872b5c05-9984-45aa-82ab-6ab103c385d9-kube-api-access-gsptt\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.036925 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.036784    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.036925 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.036826    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-ovnkube-config\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.036925 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.036855    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-ovnkube-script-lib\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.036925 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.036881    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/84728120-5cde-4b79-944d-7554db40cf41-kubelet-config\") pod \"global-pull-secret-syncer-r5jgb\" (UID: \"84728120-5cde-4b79-944d-7554db40cf41\") " pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:29:39.036925 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.036926    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-host-slash\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.037575 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.036964    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-etc-openvswitch\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.037575 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.037006    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-env-overrides\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.037575 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.037032    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe-socket-dir\") pod \"aws-ebs-csi-driver-node-ftp6r\" (UID: \"6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r"
Sep 17 15:29:39.037575 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.037072    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe-sys-fs\") pod \"aws-ebs-csi-driver-node-ftp6r\" (UID: \"6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r"
Sep 17 15:29:39.037575 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.037086    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.037575 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.037119    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b0cd1b81-7563-47df-bb6a-4b1f41073a2b-iptables-alerter-script\") pod \"iptables-alerter-wxm49\" (UID: \"b0cd1b81-7563-47df-bb6a-4b1f41073a2b\") " pod="openshift-network-operator/iptables-alerter-wxm49"
Sep 17 15:29:39.037575 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.037147    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-system-cni-dir\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.037575 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.037170    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-host-run-multus-certs\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.037575 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.037192    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-etc-kubernetes\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.037575 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.037213    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-systemd-units\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.037575 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.037234    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-host-run-netns\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.037575 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.037257    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84728120-5cde-4b79-944d-7554db40cf41-original-pull-secret\") pod \"global-pull-secret-syncer-r5jgb\" (UID: \"84728120-5cde-4b79-944d-7554db40cf41\") " pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:29:39.037575 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.037283    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-cnibin\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.037943 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.037888    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-host-kubelet\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.037943 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.037918    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-run-systemd\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.037943 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.037940    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-run-openvswitch\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.038035 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.037971    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe-device-dir\") pod \"aws-ebs-csi-driver-node-ftp6r\" (UID: \"6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r"
Sep 17 15:29:39.038035 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.038002    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe-etc-selinux\") pod \"aws-ebs-csi-driver-node-ftp6r\" (UID: \"6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r"
Sep 17 15:29:39.038035 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.038025    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0cd1b81-7563-47df-bb6a-4b1f41073a2b-host-slash\") pod \"iptables-alerter-wxm49\" (UID: \"b0cd1b81-7563-47df-bb6a-4b1f41073a2b\") " pod="openshift-network-operator/iptables-alerter-wxm49"
Sep 17 15:29:39.038207 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.038077    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkk7j\" (UniqueName: \"kubernetes.io/projected/b0cd1b81-7563-47df-bb6a-4b1f41073a2b-kube-api-access-bkk7j\") pod \"iptables-alerter-wxm49\" (UID: \"b0cd1b81-7563-47df-bb6a-4b1f41073a2b\") " pod="openshift-network-operator/iptables-alerter-wxm49"
Sep 17 15:29:39.038207 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.038096    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-host-run-netns\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.038207 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.038111    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-host-var-lib-cni-bin\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.038207 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.038123    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-hostroot\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.038207 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.038174    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-node-log\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.038207 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.038201    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ftp6r\" (UID: \"6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r"
Sep 17 15:29:39.038470 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.038235    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/872b5c05-9984-45aa-82ab-6ab103c385d9-cni-binary-copy\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.038470 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.038271    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-host-run-k8s-cni-cncf-io\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.038470 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.038290    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-var-lib-openvswitch\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.038470 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.038305    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-log-socket\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.038470 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.038320    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe-registration-dir\") pod \"aws-ebs-csi-driver-node-ftp6r\" (UID: \"6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r"
Sep 17 15:29:39.038470 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.038333    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fca9ba31-c254-4a6e-8607-fb6a4af8efc2-host\") pod \"node-ca-s9svr\" (UID: \"fca9ba31-c254-4a6e-8607-fb6a4af8efc2\") " pod="openshift-image-registry/node-ca-s9svr"
Sep 17 15:29:39.038470 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.038347    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fca9ba31-c254-4a6e-8607-fb6a4af8efc2-serviceca\") pod \"node-ca-s9svr\" (UID: \"fca9ba31-c254-4a6e-8607-fb6a4af8efc2\") " pod="openshift-image-registry/node-ca-s9svr"
Sep 17 15:29:39.038470 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.038355    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8dwgb\""
Sep 17 15:29:39.038470 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.038361    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f26zc\" (UniqueName: \"kubernetes.io/projected/fca9ba31-c254-4a6e-8607-fb6a4af8efc2-kube-api-access-f26zc\") pod \"node-ca-s9svr\" (UID: \"fca9ba31-c254-4a6e-8607-fb6a4af8efc2\") " pod="openshift-image-registry/node-ca-s9svr"
Sep 17 15:29:39.038832 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.038590    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 15:29:39.038832 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.038808    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 15:29:39.038926 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.038848    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 15:29:39.038926 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.038866    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 15:29:39.039021 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.038955    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 15:29:39.039179 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.039137    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 15:29:39.039179 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.039162    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gz2nh\""
Sep 17 15:29:39.039315 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.039179    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 15:29:39.039315 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.039167    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 15:29:39.039315 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.039240    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n48m4\""
Sep 17 15:29:39.039659 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.039644    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:29:39.039728 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.039699    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xfzrp"
Sep 17 15:29:39.040048 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.040030    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 15:29:39.042148 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.042133    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cmrtd"
Sep 17 15:29:39.044253 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.044235    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 15:29:39.044425 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.044358    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-j9xbm\""
Sep 17 15:29:39.044425 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.044377    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:29:39.044425 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.044397    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f75s5\""
Sep 17 15:29:39.044597 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:39.044434    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn89v" podUID="eae675d6-96cb-4843-9deb-31182313d8ac"
Sep 17 15:29:39.044703 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.044676    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 15:29:39.044748 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.044732    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 15:29:39.044790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.044776    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 15:29:39.048588 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.048566    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 15:29:39.056006 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.055959    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 15:24:38 +0000 UTC" deadline="2028-03-29 16:13:25.363137866 +0000 UTC"
Sep 17 15:29:39.056006 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.055999    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13416h43m46.307141449s"
Sep 17 15:29:39.078378 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.078317    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5rmrl"
Sep 17 15:29:39.089407 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.089332    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5rmrl"
Sep 17 15:29:39.131188 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.131168    2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 15:29:39.138626 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.138601    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-log-socket\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.138734 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.138645    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fca9ba31-c254-4a6e-8607-fb6a4af8efc2-host\") pod \"node-ca-s9svr\" (UID: \"fca9ba31-c254-4a6e-8607-fb6a4af8efc2\") " pod="openshift-image-registry/node-ca-s9svr"
Sep 17 15:29:39.138734 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.138674    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f26zc\" (UniqueName: \"kubernetes.io/projected/fca9ba31-c254-4a6e-8607-fb6a4af8efc2-kube-api-access-f26zc\") pod \"node-ca-s9svr\" (UID: \"fca9ba31-c254-4a6e-8607-fb6a4af8efc2\") " pod="openshift-image-registry/node-ca-s9svr"
Sep 17 15:29:39.138734 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.138698    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-log-socket\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.138734 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.138702    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-etc-kubernetes\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.138897 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.138756    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb7st\" (UniqueName: \"kubernetes.io/projected/5c65f398-6a2c-4838-8c61-bcabe5acfca7-kube-api-access-lb7st\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.138897 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.138776    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b086625-d2ab-4b2e-826b-8cf6a9534826-system-cni-dir\") pod \"multus-additional-cni-plugins-cmrtd\" (UID: \"2b086625-d2ab-4b2e-826b-8cf6a9534826\") " pod="openshift-multus/multus-additional-cni-plugins-cmrtd"
Sep 17 15:29:39.138897 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.138776    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fca9ba31-c254-4a6e-8607-fb6a4af8efc2-host\") pod \"node-ca-s9svr\" (UID: \"fca9ba31-c254-4a6e-8607-fb6a4af8efc2\") " pod="openshift-image-registry/node-ca-s9svr"
Sep 17 15:29:39.138897 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.138794    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-host-cni-netd\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.138897 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.138820    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b086625-d2ab-4b2e-826b-8cf6a9534826-cni-binary-copy\") pod \"multus-additional-cni-plugins-cmrtd\" (UID: \"2b086625-d2ab-4b2e-826b-8cf6a9534826\") " pod="openshift-multus/multus-additional-cni-plugins-cmrtd"
Sep 17 15:29:39.138897 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.138829    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-host-cni-netd\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.138897 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.138835    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/593418fc-096d-445c-9863-9ed9f57ee092-tmp-dir\") pod \"node-resolver-xfzrp\" (UID: \"593418fc-096d-445c-9863-9ed9f57ee092\") " pod="openshift-dns/node-resolver-xfzrp"
Sep 17 15:29:39.138897 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.138853    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eae675d6-96cb-4843-9deb-31182313d8ac-metrics-certs\") pod \"network-metrics-daemon-hn89v\" (UID: \"eae675d6-96cb-4843-9deb-31182313d8ac\") " pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:29:39.138897 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.138869    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/872b5c05-9984-45aa-82ab-6ab103c385d9-multus-daemon-config\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.138897 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.138891    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plpr7\" (UniqueName: \"kubernetes.io/projected/352c01b0-7c02-4da9-a311-816815cd5e1e-kube-api-access-plpr7\") pod \"network-check-target-l2rxt\" (UID: \"352c01b0-7c02-4da9-a311-816815cd5e1e\") " pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:29:39.139435 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.138906    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-run-ovn\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.139435 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.138922    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-host-run-ovn-kubernetes\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.139435 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.138966    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-host-run-ovn-kubernetes\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.139435 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.138984    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-run-ovn\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.139435 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139085    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-ovn-node-metrics-cert\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.139435 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139107    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/84728120-5cde-4b79-944d-7554db40cf41-dbus\") pod \"global-pull-secret-syncer-r5jgb\" (UID: \"84728120-5cde-4b79-944d-7554db40cf41\") " pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:29:39.139435 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139138    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b0cd1b81-7563-47df-bb6a-4b1f41073a2b-iptables-alerter-script\") pod \"iptables-alerter-wxm49\" (UID: \"b0cd1b81-7563-47df-bb6a-4b1f41073a2b\") " pod="openshift-network-operator/iptables-alerter-wxm49"
Sep 17 15:29:39.139435 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139156    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-etc-sysctl-d\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.139435 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139172    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsptt\" (UniqueName: \"kubernetes.io/projected/872b5c05-9984-45aa-82ab-6ab103c385d9-kube-api-access-gsptt\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.139435 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139190    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.139435 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139235    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-lib-modules\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.139435 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139256    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-host\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.139435 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139262    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/84728120-5cde-4b79-944d-7554db40cf41-dbus\") pod \"global-pull-secret-syncer-r5jgb\" (UID: \"84728120-5cde-4b79-944d-7554db40cf41\") " pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:29:39.139435 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139277    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c65f398-6a2c-4838-8c61-bcabe5acfca7-tmp\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.139435 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139320    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-host-slash\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.139435 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139347    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-etc-openvswitch\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.139435 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139352    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.140245 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139373    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe-socket-dir\") pod \"aws-ebs-csi-driver-node-ftp6r\" (UID: \"6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r"
Sep 17 15:29:39.140245 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139393    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-host-slash\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.140245 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139399    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-etc-kubernetes\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.140245 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139429    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-etc-openvswitch\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.140245 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139422    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-systemd-units\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.140245 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139465    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe-device-dir\") pod \"aws-ebs-csi-driver-node-ftp6r\" (UID: \"6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r"
Sep 17 15:29:39.140245 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139480    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/872b5c05-9984-45aa-82ab-6ab103c385d9-multus-daemon-config\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.140245 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139491    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-etc-kubernetes\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.140245 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139493    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/69c5bd0b-4c06-45dc-9b1b-a9af91086fcd-agent-certs\") pod \"konnectivity-agent-nh9lk\" (UID: \"69c5bd0b-4c06-45dc-9b1b-a9af91086fcd\") " pod="kube-system/konnectivity-agent-nh9lk"
Sep 17 15:29:39.140245 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139537    2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 15:29:39.140245 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139582    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b086625-d2ab-4b2e-826b-8cf6a9534826-cnibin\") pod \"multus-additional-cni-plugins-cmrtd\" (UID: \"2b086625-d2ab-4b2e-826b-8cf6a9534826\") " pod="openshift-multus/multus-additional-cni-plugins-cmrtd"
Sep 17 15:29:39.140245 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139605    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b086625-d2ab-4b2e-826b-8cf6a9534826-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cmrtd\" (UID: \"2b086625-d2ab-4b2e-826b-8cf6a9534826\") " pod="openshift-multus/multus-additional-cni-plugins-cmrtd"
Sep 17 15:29:39.140245 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139542    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe-socket-dir\") pod \"aws-ebs-csi-driver-node-ftp6r\" (UID: \"6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r"
Sep 17 15:29:39.140245 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139544    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-systemd-units\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.140245 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139653    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-run-openvswitch\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.140245 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139679    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe-etc-selinux\") pod \"aws-ebs-csi-driver-node-ftp6r\" (UID: \"6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r"
Sep 17 15:29:39.140245 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139680    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe-device-dir\") pod \"aws-ebs-csi-driver-node-ftp6r\" (UID: \"6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r"
Sep 17 15:29:39.141117 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139681    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b0cd1b81-7563-47df-bb6a-4b1f41073a2b-iptables-alerter-script\") pod \"iptables-alerter-wxm49\" (UID: \"b0cd1b81-7563-47df-bb6a-4b1f41073a2b\") " pod="openshift-network-operator/iptables-alerter-wxm49"
Sep 17 15:29:39.141117 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139712    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkk7j\" (UniqueName: \"kubernetes.io/projected/b0cd1b81-7563-47df-bb6a-4b1f41073a2b-kube-api-access-bkk7j\") pod \"iptables-alerter-wxm49\" (UID: \"b0cd1b81-7563-47df-bb6a-4b1f41073a2b\") " pod="openshift-network-operator/iptables-alerter-wxm49"
Sep 17 15:29:39.141117 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139721    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-run-openvswitch\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.141117 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139739    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-etc-modprobe-d\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.141117 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139762    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-run\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.141117 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139767    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe-etc-selinux\") pod \"aws-ebs-csi-driver-node-ftp6r\" (UID: \"6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r"
Sep 17 15:29:39.141117 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139788    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-host-var-lib-cni-bin\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.141117 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139813    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ftp6r\" (UID: \"6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r"
Sep 17 15:29:39.141117 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139841    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-var-lib-kubelet\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.141117 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139865    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b086625-d2ab-4b2e-826b-8cf6a9534826-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cmrtd\" (UID: \"2b086625-d2ab-4b2e-826b-8cf6a9534826\") " pod="openshift-multus/multus-additional-cni-plugins-cmrtd"
Sep 17 15:29:39.141117 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139870    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-host-var-lib-cni-bin\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.141117 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139892    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ftp6r\" (UID: \"6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r"
Sep 17 15:29:39.141117 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139891    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b086625-d2ab-4b2e-826b-8cf6a9534826-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cmrtd\" (UID: \"2b086625-d2ab-4b2e-826b-8cf6a9534826\") " pod="openshift-multus/multus-additional-cni-plugins-cmrtd"
Sep 17 15:29:39.141117 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139937    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-host-run-k8s-cni-cncf-io\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.141117 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139965    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe-registration-dir\") pod \"aws-ebs-csi-driver-node-ftp6r\" (UID: \"6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r"
Sep 17 15:29:39.141117 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.139988    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fca9ba31-c254-4a6e-8607-fb6a4af8efc2-serviceca\") pod \"node-ca-s9svr\" (UID: \"fca9ba31-c254-4a6e-8607-fb6a4af8efc2\") " pod="openshift-image-registry/node-ca-s9svr"
Sep 17 15:29:39.141117 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140007    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-host-run-k8s-cni-cncf-io\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.141990 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140012    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-multus-cni-dir\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.141990 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140037    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-host-var-lib-kubelet\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.141990 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140054    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-host-cni-bin\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.141990 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140074    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-multus-cni-dir\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.141990 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140040    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe-registration-dir\") pod \"aws-ebs-csi-driver-node-ftp6r\" (UID: \"6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r"
Sep 17 15:29:39.141990 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140110    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-host-cni-bin\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.141990 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140117    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-host-var-lib-kubelet\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.141990 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140134    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6dfl\" (UniqueName: \"kubernetes.io/projected/6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe-kube-api-access-f6dfl\") pod \"aws-ebs-csi-driver-node-ftp6r\" (UID: \"6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r"
Sep 17 15:29:39.141990 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140162    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l68tb\" (UniqueName: \"kubernetes.io/projected/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-kube-api-access-l68tb\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.141990 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140186    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-etc-sysconfig\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.141990 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140211    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5c65f398-6a2c-4838-8c61-bcabe5acfca7-etc-tuned\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.141990 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140236    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-multus-socket-dir-parent\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.141990 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140261    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-host-var-lib-cni-multus\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.141990 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140285    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-multus-conf-dir\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.141990 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140308    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-ovnkube-config\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.141990 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140334    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-ovnkube-script-lib\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.141990 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140361    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/84728120-5cde-4b79-944d-7554db40cf41-kubelet-config\") pod \"global-pull-secret-syncer-r5jgb\" (UID: \"84728120-5cde-4b79-944d-7554db40cf41\") " pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:29:39.142966 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140362    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-host-var-lib-cni-multus\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.142966 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140365    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-multus-conf-dir\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.142966 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140384    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe-sys-fs\") pod \"aws-ebs-csi-driver-node-ftp6r\" (UID: \"6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r"
Sep 17 15:29:39.142966 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140415    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-multus-socket-dir-parent\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.142966 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140455    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6fxh\" (UniqueName: \"kubernetes.io/projected/eae675d6-96cb-4843-9deb-31182313d8ac-kube-api-access-m6fxh\") pod \"network-metrics-daemon-hn89v\" (UID: \"eae675d6-96cb-4843-9deb-31182313d8ac\") " pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:29:39.142966 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140480    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-env-overrides\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.142966 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140503    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fca9ba31-c254-4a6e-8607-fb6a4af8efc2-serviceca\") pod \"node-ca-s9svr\" (UID: \"fca9ba31-c254-4a6e-8607-fb6a4af8efc2\") " pod="openshift-image-registry/node-ca-s9svr"
Sep 17 15:29:39.142966 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140512    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/84728120-5cde-4b79-944d-7554db40cf41-kubelet-config\") pod \"global-pull-secret-syncer-r5jgb\" (UID: \"84728120-5cde-4b79-944d-7554db40cf41\") " pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:29:39.142966 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140524    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe-sys-fs\") pod \"aws-ebs-csi-driver-node-ftp6r\" (UID: \"6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r"
Sep 17 15:29:39.142966 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140648    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-etc-systemd\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.142966 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140673    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/593418fc-096d-445c-9863-9ed9f57ee092-hosts-file\") pod \"node-resolver-xfzrp\" (UID: \"593418fc-096d-445c-9863-9ed9f57ee092\") " pod="openshift-dns/node-resolver-xfzrp"
Sep 17 15:29:39.142966 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140708    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2hvv\" (UniqueName: \"kubernetes.io/projected/593418fc-096d-445c-9863-9ed9f57ee092-kube-api-access-t2hvv\") pod \"node-resolver-xfzrp\" (UID: \"593418fc-096d-445c-9863-9ed9f57ee092\") " pod="openshift-dns/node-resolver-xfzrp"
Sep 17 15:29:39.142966 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140758    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-system-cni-dir\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.142966 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140804    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-system-cni-dir\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.142966 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140826    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-host-run-multus-certs\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.142966 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140848    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-host-run-netns\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.142966 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140883    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84728120-5cde-4b79-944d-7554db40cf41-original-pull-secret\") pod \"global-pull-secret-syncer-r5jgb\" (UID: \"84728120-5cde-4b79-944d-7554db40cf41\") " pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:29:39.143741 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140895    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-host-run-multus-certs\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.143741 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140909    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/69c5bd0b-4c06-45dc-9b1b-a9af91086fcd-konnectivity-ca\") pod \"konnectivity-agent-nh9lk\" (UID: \"69c5bd0b-4c06-45dc-9b1b-a9af91086fcd\") " pod="kube-system/konnectivity-agent-nh9lk"
Sep 17 15:29:39.143741 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140933    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-cnibin\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.143741 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140945    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-ovnkube-script-lib\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.143741 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140957    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-host-kubelet\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.143741 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140988    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-run-systemd\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.143741 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140994    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-host-kubelet\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.143741 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:39.141003    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:29:39.143741 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.141012    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0cd1b81-7563-47df-bb6a-4b1f41073a2b-host-slash\") pod \"iptables-alerter-wxm49\" (UID: \"b0cd1b81-7563-47df-bb6a-4b1f41073a2b\") " pod="openshift-network-operator/iptables-alerter-wxm49"
Sep 17 15:29:39.143741 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.141040    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-etc-sysctl-conf\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.143741 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.141047    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-env-overrides\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.143741 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.141096    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-ovnkube-config\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.143741 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.141052    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-cnibin\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.143741 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.141100    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-run-systemd\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.143741 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:39.141116    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84728120-5cde-4b79-944d-7554db40cf41-original-pull-secret podName:84728120-5cde-4b79-944d-7554db40cf41 nodeName:}" failed. No retries permitted until 2026-09-17 15:29:39.64105124 +0000 UTC m=+2.021068994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84728120-5cde-4b79-944d-7554db40cf41-original-pull-secret") pod "global-pull-secret-syncer-r5jgb" (UID: "84728120-5cde-4b79-944d-7554db40cf41") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:29:39.143741 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.140933    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-host-run-netns\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.143741 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.141124    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0cd1b81-7563-47df-bb6a-4b1f41073a2b-host-slash\") pod \"iptables-alerter-wxm49\" (UID: \"b0cd1b81-7563-47df-bb6a-4b1f41073a2b\") " pod="openshift-network-operator/iptables-alerter-wxm49"
Sep 17 15:29:39.144328 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.141134    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-sys\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.144328 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.141160    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-host-run-netns\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.144328 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.141183    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-hostroot\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.144328 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.141207    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-node-log\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.144328 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.141230    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-host-run-netns\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.144328 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.141232    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78tfq\" (UniqueName: \"kubernetes.io/projected/2b086625-d2ab-4b2e-826b-8cf6a9534826-kube-api-access-78tfq\") pod \"multus-additional-cni-plugins-cmrtd\" (UID: \"2b086625-d2ab-4b2e-826b-8cf6a9534826\") " pod="openshift-multus/multus-additional-cni-plugins-cmrtd"
Sep 17 15:29:39.144328 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.141257    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/872b5c05-9984-45aa-82ab-6ab103c385d9-cni-binary-copy\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.144328 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.141275    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/872b5c05-9984-45aa-82ab-6ab103c385d9-hostroot\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.144328 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.141301    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-var-lib-openvswitch\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.144328 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.141336    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-node-log\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.144328 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.141373    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-var-lib-openvswitch\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.144328 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.142083    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/872b5c05-9984-45aa-82ab-6ab103c385d9-cni-binary-copy\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.144328 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.143253    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-ovn-node-metrics-cert\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.144820 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:39.144802    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:29:39.144898 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:39.144829    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:29:39.144898 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:39.144842    2556 projected.go:196] Error preparing data for projected volume kube-api-access-plpr7 for pod openshift-network-diagnostics/network-check-target-l2rxt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:29:39.144996 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:39.144902    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/352c01b0-7c02-4da9-a311-816815cd5e1e-kube-api-access-plpr7 podName:352c01b0-7c02-4da9-a311-816815cd5e1e nodeName:}" failed. No retries permitted until 2026-09-17 15:29:39.644885313 +0000 UTC m=+2.024903069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-plpr7" (UniqueName: "kubernetes.io/projected/352c01b0-7c02-4da9-a311-816815cd5e1e-kube-api-access-plpr7") pod "network-check-target-l2rxt" (UID: "352c01b0-7c02-4da9-a311-816815cd5e1e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:29:39.146842 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.146824    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f26zc\" (UniqueName: \"kubernetes.io/projected/fca9ba31-c254-4a6e-8607-fb6a4af8efc2-kube-api-access-f26zc\") pod \"node-ca-s9svr\" (UID: \"fca9ba31-c254-4a6e-8607-fb6a4af8efc2\") " pod="openshift-image-registry/node-ca-s9svr"
Sep 17 15:29:39.152354 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.152327    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l68tb\" (UniqueName: \"kubernetes.io/projected/15b9edd1-debb-4dc7-8f59-71e2bdaa2d97-kube-api-access-l68tb\") pod \"ovnkube-node-xjz9c\" (UID: \"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97\") " pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.152572 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.152549    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsptt\" (UniqueName: \"kubernetes.io/projected/872b5c05-9984-45aa-82ab-6ab103c385d9-kube-api-access-gsptt\") pod \"multus-lkm6c\" (UID: \"872b5c05-9984-45aa-82ab-6ab103c385d9\") " pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.152646 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.152597    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkk7j\" (UniqueName: \"kubernetes.io/projected/b0cd1b81-7563-47df-bb6a-4b1f41073a2b-kube-api-access-bkk7j\") pod \"iptables-alerter-wxm49\" (UID: \"b0cd1b81-7563-47df-bb6a-4b1f41073a2b\") " pod="openshift-network-operator/iptables-alerter-wxm49"
Sep 17 15:29:39.154142 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.154123    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6dfl\" (UniqueName: \"kubernetes.io/projected/6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe-kube-api-access-f6dfl\") pod \"aws-ebs-csi-driver-node-ftp6r\" (UID: \"6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r"
Sep 17 15:29:39.241997 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.241966    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-etc-sysctl-d\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.241997 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242002    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-lib-modules\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.242351 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242022    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-host\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.242351 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242044    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c65f398-6a2c-4838-8c61-bcabe5acfca7-tmp\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.242351 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242087    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/69c5bd0b-4c06-45dc-9b1b-a9af91086fcd-agent-certs\") pod \"konnectivity-agent-nh9lk\" (UID: \"69c5bd0b-4c06-45dc-9b1b-a9af91086fcd\") " pod="kube-system/konnectivity-agent-nh9lk"
Sep 17 15:29:39.242351 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242119    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b086625-d2ab-4b2e-826b-8cf6a9534826-cnibin\") pod \"multus-additional-cni-plugins-cmrtd\" (UID: \"2b086625-d2ab-4b2e-826b-8cf6a9534826\") " pod="openshift-multus/multus-additional-cni-plugins-cmrtd"
Sep 17 15:29:39.242351 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242142    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b086625-d2ab-4b2e-826b-8cf6a9534826-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cmrtd\" (UID: \"2b086625-d2ab-4b2e-826b-8cf6a9534826\") " pod="openshift-multus/multus-additional-cni-plugins-cmrtd"
Sep 17 15:29:39.242351 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242167    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-etc-modprobe-d\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.242351 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242173    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-etc-sysctl-d\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.242351 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242190    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-run\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.242351 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242193    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-lib-modules\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.242351 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242202    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b086625-d2ab-4b2e-826b-8cf6a9534826-cnibin\") pod \"multus-additional-cni-plugins-cmrtd\" (UID: \"2b086625-d2ab-4b2e-826b-8cf6a9534826\") " pod="openshift-multus/multus-additional-cni-plugins-cmrtd"
Sep 17 15:29:39.242351 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242118    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-host\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.242351 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242237    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-var-lib-kubelet\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.242351 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242265    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b086625-d2ab-4b2e-826b-8cf6a9534826-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cmrtd\" (UID: \"2b086625-d2ab-4b2e-826b-8cf6a9534826\") " pod="openshift-multus/multus-additional-cni-plugins-cmrtd"
Sep 17 15:29:39.242351 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242275    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-etc-modprobe-d\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.242351 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242281    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-var-lib-kubelet\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.242351 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242247    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-run\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.242351 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242300    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b086625-d2ab-4b2e-826b-8cf6a9534826-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cmrtd\" (UID: \"2b086625-d2ab-4b2e-826b-8cf6a9534826\") " pod="openshift-multus/multus-additional-cni-plugins-cmrtd"
Sep 17 15:29:39.243238 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242319    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b086625-d2ab-4b2e-826b-8cf6a9534826-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cmrtd\" (UID: \"2b086625-d2ab-4b2e-826b-8cf6a9534826\") " pod="openshift-multus/multus-additional-cni-plugins-cmrtd"
Sep 17 15:29:39.243238 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242328    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-etc-sysconfig\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.243238 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242355    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-etc-sysconfig\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.243238 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242377    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5c65f398-6a2c-4838-8c61-bcabe5acfca7-etc-tuned\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.243238 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242410    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6fxh\" (UniqueName: \"kubernetes.io/projected/eae675d6-96cb-4843-9deb-31182313d8ac-kube-api-access-m6fxh\") pod \"network-metrics-daemon-hn89v\" (UID: \"eae675d6-96cb-4843-9deb-31182313d8ac\") " pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:29:39.243238 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242441    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-etc-systemd\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.243238 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242475    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/593418fc-096d-445c-9863-9ed9f57ee092-hosts-file\") pod \"node-resolver-xfzrp\" (UID: \"593418fc-096d-445c-9863-9ed9f57ee092\") " pod="openshift-dns/node-resolver-xfzrp"
Sep 17 15:29:39.243238 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242505    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2hvv\" (UniqueName: \"kubernetes.io/projected/593418fc-096d-445c-9863-9ed9f57ee092-kube-api-access-t2hvv\") pod \"node-resolver-xfzrp\" (UID: \"593418fc-096d-445c-9863-9ed9f57ee092\") " pod="openshift-dns/node-resolver-xfzrp"
Sep 17 15:29:39.243238 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242550    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/69c5bd0b-4c06-45dc-9b1b-a9af91086fcd-konnectivity-ca\") pod \"konnectivity-agent-nh9lk\" (UID: \"69c5bd0b-4c06-45dc-9b1b-a9af91086fcd\") " pod="kube-system/konnectivity-agent-nh9lk"
Sep 17 15:29:39.243238 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242579    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-etc-sysctl-conf\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.243238 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242603    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-sys\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.243238 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242650    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78tfq\" (UniqueName: \"kubernetes.io/projected/2b086625-d2ab-4b2e-826b-8cf6a9534826-kube-api-access-78tfq\") pod \"multus-additional-cni-plugins-cmrtd\" (UID: \"2b086625-d2ab-4b2e-826b-8cf6a9534826\") " pod="openshift-multus/multus-additional-cni-plugins-cmrtd"
Sep 17 15:29:39.243238 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242682    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-etc-kubernetes\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.243238 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242708    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb7st\" (UniqueName: \"kubernetes.io/projected/5c65f398-6a2c-4838-8c61-bcabe5acfca7-kube-api-access-lb7st\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.243238 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242732    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b086625-d2ab-4b2e-826b-8cf6a9534826-system-cni-dir\") pod \"multus-additional-cni-plugins-cmrtd\" (UID: \"2b086625-d2ab-4b2e-826b-8cf6a9534826\") " pod="openshift-multus/multus-additional-cni-plugins-cmrtd"
Sep 17 15:29:39.243238 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242757    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b086625-d2ab-4b2e-826b-8cf6a9534826-cni-binary-copy\") pod \"multus-additional-cni-plugins-cmrtd\" (UID: \"2b086625-d2ab-4b2e-826b-8cf6a9534826\") " pod="openshift-multus/multus-additional-cni-plugins-cmrtd"
Sep 17 15:29:39.243238 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242780    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/593418fc-096d-445c-9863-9ed9f57ee092-tmp-dir\") pod \"node-resolver-xfzrp\" (UID: \"593418fc-096d-445c-9863-9ed9f57ee092\") " pod="openshift-dns/node-resolver-xfzrp"
Sep 17 15:29:39.243878 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242821    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eae675d6-96cb-4843-9deb-31182313d8ac-metrics-certs\") pod \"network-metrics-daemon-hn89v\" (UID: \"eae675d6-96cb-4843-9deb-31182313d8ac\") " pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:29:39.243878 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242840    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b086625-d2ab-4b2e-826b-8cf6a9534826-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cmrtd\" (UID: \"2b086625-d2ab-4b2e-826b-8cf6a9534826\") " pod="openshift-multus/multus-additional-cni-plugins-cmrtd"
Sep 17 15:29:39.243878 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242912    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b086625-d2ab-4b2e-826b-8cf6a9534826-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cmrtd\" (UID: \"2b086625-d2ab-4b2e-826b-8cf6a9534826\") " pod="openshift-multus/multus-additional-cni-plugins-cmrtd"
Sep 17 15:29:39.243878 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242922    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-etc-kubernetes\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.243878 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:39.242956    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:29:39.243878 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.242981    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b086625-d2ab-4b2e-826b-8cf6a9534826-system-cni-dir\") pod \"multus-additional-cni-plugins-cmrtd\" (UID: \"2b086625-d2ab-4b2e-826b-8cf6a9534826\") " pod="openshift-multus/multus-additional-cni-plugins-cmrtd"
Sep 17 15:29:39.243878 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:39.243012    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eae675d6-96cb-4843-9deb-31182313d8ac-metrics-certs podName:eae675d6-96cb-4843-9deb-31182313d8ac nodeName:}" failed. No retries permitted until 2026-09-17 15:29:39.74299691 +0000 UTC m=+2.123014651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eae675d6-96cb-4843-9deb-31182313d8ac-metrics-certs") pod "network-metrics-daemon-hn89v" (UID: "eae675d6-96cb-4843-9deb-31182313d8ac") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:29:39.243878 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.243044    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-etc-sysctl-conf\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.243878 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.243168    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/69c5bd0b-4c06-45dc-9b1b-a9af91086fcd-konnectivity-ca\") pod \"konnectivity-agent-nh9lk\" (UID: \"69c5bd0b-4c06-45dc-9b1b-a9af91086fcd\") " pod="kube-system/konnectivity-agent-nh9lk"
Sep 17 15:29:39.243878 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.243175    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-sys\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.243878 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.243243    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/593418fc-096d-445c-9863-9ed9f57ee092-hosts-file\") pod \"node-resolver-xfzrp\" (UID: \"593418fc-096d-445c-9863-9ed9f57ee092\") " pod="openshift-dns/node-resolver-xfzrp"
Sep 17 15:29:39.243878 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.243313    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/593418fc-096d-445c-9863-9ed9f57ee092-tmp-dir\") pod \"node-resolver-xfzrp\" (UID: \"593418fc-096d-445c-9863-9ed9f57ee092\") " pod="openshift-dns/node-resolver-xfzrp"
Sep 17 15:29:39.243878 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.243358    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5c65f398-6a2c-4838-8c61-bcabe5acfca7-etc-systemd\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.243878 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.243494    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b086625-d2ab-4b2e-826b-8cf6a9534826-cni-binary-copy\") pod \"multus-additional-cni-plugins-cmrtd\" (UID: \"2b086625-d2ab-4b2e-826b-8cf6a9534826\") " pod="openshift-multus/multus-additional-cni-plugins-cmrtd"
Sep 17 15:29:39.245024 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.245005    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5c65f398-6a2c-4838-8c61-bcabe5acfca7-etc-tuned\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.245161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.245036    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c65f398-6a2c-4838-8c61-bcabe5acfca7-tmp\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.245161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.245120    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/69c5bd0b-4c06-45dc-9b1b-a9af91086fcd-agent-certs\") pod \"konnectivity-agent-nh9lk\" (UID: \"69c5bd0b-4c06-45dc-9b1b-a9af91086fcd\") " pod="kube-system/konnectivity-agent-nh9lk"
Sep 17 15:29:39.252945 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.252921    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6fxh\" (UniqueName: \"kubernetes.io/projected/eae675d6-96cb-4843-9deb-31182313d8ac-kube-api-access-m6fxh\") pod \"network-metrics-daemon-hn89v\" (UID: \"eae675d6-96cb-4843-9deb-31182313d8ac\") " pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:29:39.254463 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.254443    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb7st\" (UniqueName: \"kubernetes.io/projected/5c65f398-6a2c-4838-8c61-bcabe5acfca7-kube-api-access-lb7st\") pod \"tuned-857mz\" (UID: \"5c65f398-6a2c-4838-8c61-bcabe5acfca7\") " pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.254550 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.254508    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2hvv\" (UniqueName: \"kubernetes.io/projected/593418fc-096d-445c-9863-9ed9f57ee092-kube-api-access-t2hvv\") pod \"node-resolver-xfzrp\" (UID: \"593418fc-096d-445c-9863-9ed9f57ee092\") " pod="openshift-dns/node-resolver-xfzrp"
Sep 17 15:29:39.254781 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.254762    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78tfq\" (UniqueName: \"kubernetes.io/projected/2b086625-d2ab-4b2e-826b-8cf6a9534826-kube-api-access-78tfq\") pod \"multus-additional-cni-plugins-cmrtd\" (UID: \"2b086625-d2ab-4b2e-826b-8cf6a9534826\") " pod="openshift-multus/multus-additional-cni-plugins-cmrtd"
Sep 17 15:29:39.325178 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:39.325141    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623939e1a7296f8c7c9abdfb3dac6ecc.slice/crio-3eca865f036b87f99a349ae1e7b83f79babd26a57bd991949ba374b7f5dd6af2 WatchSource:0}: Error finding container 3eca865f036b87f99a349ae1e7b83f79babd26a57bd991949ba374b7f5dd6af2: Status 404 returned error can't find the container with id 3eca865f036b87f99a349ae1e7b83f79babd26a57bd991949ba374b7f5dd6af2
Sep 17 15:29:39.325471 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:39.325444    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4965a8a85b93317838f07968164cf5cd.slice/crio-80c967e59dfd0bb5b2175c9377117dba6bcd794cb7338932a35d2bb8242b149c WatchSource:0}: Error finding container 80c967e59dfd0bb5b2175c9377117dba6bcd794cb7338932a35d2bb8242b149c: Status 404 returned error can't find the container with id 80c967e59dfd0bb5b2175c9377117dba6bcd794cb7338932a35d2bb8242b149c
Sep 17 15:29:39.329888 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.329870    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:29:39.348390 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.348360    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wxm49"
Sep 17 15:29:39.356451 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.356428    2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:29:39.358129 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:39.358108    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0cd1b81_7563_47df_bb6a_4b1f41073a2b.slice/crio-87a1912c17d5ad1ec5f9ced74a42f5bc9220078a24bac431e68785afff06f5b1 WatchSource:0}: Error finding container 87a1912c17d5ad1ec5f9ced74a42f5bc9220078a24bac431e68785afff06f5b1: Status 404 returned error can't find the container with id 87a1912c17d5ad1ec5f9ced74a42f5bc9220078a24bac431e68785afff06f5b1
Sep 17 15:29:39.369614 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.369568    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r"
Sep 17 15:29:39.377525 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.377500    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s9svr"
Sep 17 15:29:39.377832 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:39.377814    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e5eacd8_cf9b_4ecb_9004_9fb59aefdffe.slice/crio-12f58d88597356228ce79be0a9ef678eae7ad3d2bcdef6368590cc30b6e60bb7 WatchSource:0}: Error finding container 12f58d88597356228ce79be0a9ef678eae7ad3d2bcdef6368590cc30b6e60bb7: Status 404 returned error can't find the container with id 12f58d88597356228ce79be0a9ef678eae7ad3d2bcdef6368590cc30b6e60bb7
Sep 17 15:29:39.385826 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:39.385803    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca9ba31_c254_4a6e_8607_fb6a4af8efc2.slice/crio-7c804fde3f6f03a8351fd8ba255ea6b8db3273980063d2642b7a883f060634c9 WatchSource:0}: Error finding container 7c804fde3f6f03a8351fd8ba255ea6b8db3273980063d2642b7a883f060634c9: Status 404 returned error can't find the container with id 7c804fde3f6f03a8351fd8ba255ea6b8db3273980063d2642b7a883f060634c9
Sep 17 15:29:39.386864 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.386842    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lkm6c"
Sep 17 15:29:39.395010 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:39.394988    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872b5c05_9984_45aa_82ab_6ab103c385d9.slice/crio-d448b3f0464ad7459e08e54dbe464ed763758d5a6fe9b071a99729252daafcc9 WatchSource:0}: Error finding container d448b3f0464ad7459e08e54dbe464ed763758d5a6fe9b071a99729252daafcc9: Status 404 returned error can't find the container with id d448b3f0464ad7459e08e54dbe464ed763758d5a6fe9b071a99729252daafcc9
Sep 17 15:29:39.402577 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.402551    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:29:39.409403 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.409385    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nh9lk"
Sep 17 15:29:39.411583 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:39.411556    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b9edd1_debb_4dc7_8f59_71e2bdaa2d97.slice/crio-e8ff7f0220822a6c0fb2d16bdc9f6fb665d2b9bd7e58a8f8ec260396b8d48468 WatchSource:0}: Error finding container e8ff7f0220822a6c0fb2d16bdc9f6fb665d2b9bd7e58a8f8ec260396b8d48468: Status 404 returned error can't find the container with id e8ff7f0220822a6c0fb2d16bdc9f6fb665d2b9bd7e58a8f8ec260396b8d48468
Sep 17 15:29:39.417128 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:39.417109    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c5bd0b_4c06_45dc_9b1b_a9af91086fcd.slice/crio-093385fd6c86ebf18075959763e1d2b7d75fc9cff18e94c7807ce9ed8a910e04 WatchSource:0}: Error finding container 093385fd6c86ebf18075959763e1d2b7d75fc9cff18e94c7807ce9ed8a910e04: Status 404 returned error can't find the container with id 093385fd6c86ebf18075959763e1d2b7d75fc9cff18e94c7807ce9ed8a910e04
Sep 17 15:29:39.425832 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.425812    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-857mz"
Sep 17 15:29:39.434538 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:39.434514    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c65f398_6a2c_4838_8c61_bcabe5acfca7.slice/crio-282b884bd5286f4dea435a5f12607015831dad1fc34da3cbcbb9029a695cf277 WatchSource:0}: Error finding container 282b884bd5286f4dea435a5f12607015831dad1fc34da3cbcbb9029a695cf277: Status 404 returned error can't find the container with id 282b884bd5286f4dea435a5f12607015831dad1fc34da3cbcbb9029a695cf277
Sep 17 15:29:39.440410 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.440385    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xfzrp"
Sep 17 15:29:39.447270 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.447252    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cmrtd"
Sep 17 15:29:39.448323 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:39.448299    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod593418fc_096d_445c_9863_9ed9f57ee092.slice/crio-24312b5420119fd5a89282bd46493fcb3dcaf4646202c4d902ebbc3be89c88ff WatchSource:0}: Error finding container 24312b5420119fd5a89282bd46493fcb3dcaf4646202c4d902ebbc3be89c88ff: Status 404 returned error can't find the container with id 24312b5420119fd5a89282bd46493fcb3dcaf4646202c4d902ebbc3be89c88ff
Sep 17 15:29:39.456160 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:29:39.456137    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b086625_d2ab_4b2e_826b_8cf6a9534826.slice/crio-b38c3c7a49252e05eeed682ed748c519fc185c940fc7ce1ee7775863b3fa9805 WatchSource:0}: Error finding container b38c3c7a49252e05eeed682ed748c519fc185c940fc7ce1ee7775863b3fa9805: Status 404 returned error can't find the container with id b38c3c7a49252e05eeed682ed748c519fc185c940fc7ce1ee7775863b3fa9805
Sep 17 15:29:39.458624 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:39.458603    2556 kuberuntime_manager.go:1459] "Unhandled Error" err="init container egress-router-binary-copy start failed in pod multus-additional-cni-plugins-cmrtd_openshift-multus(2b086625-d2ab-4b2e-826b-8cf6a9534826): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 17 15:29:39.458699 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:39.458642    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"egress-router-binary-copy\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-multus/multus-additional-cni-plugins-cmrtd" podUID="2b086625-d2ab-4b2e-826b-8cf6a9534826"
Sep 17 15:29:39.645092 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.644985    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84728120-5cde-4b79-944d-7554db40cf41-original-pull-secret\") pod \"global-pull-secret-syncer-r5jgb\" (UID: \"84728120-5cde-4b79-944d-7554db40cf41\") " pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:29:39.645092 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.645042    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plpr7\" (UniqueName: \"kubernetes.io/projected/352c01b0-7c02-4da9-a311-816815cd5e1e-kube-api-access-plpr7\") pod \"network-check-target-l2rxt\" (UID: \"352c01b0-7c02-4da9-a311-816815cd5e1e\") " pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:29:39.645286 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:39.645148    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:29:39.645286 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:39.645161    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:29:39.645286 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:39.645179    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:29:39.645286 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:39.645191    2556 projected.go:196] Error preparing data for projected volume kube-api-access-plpr7 for pod openshift-network-diagnostics/network-check-target-l2rxt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:29:39.645286 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:39.645206    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84728120-5cde-4b79-944d-7554db40cf41-original-pull-secret podName:84728120-5cde-4b79-944d-7554db40cf41 nodeName:}" failed. No retries permitted until 2026-09-17 15:29:40.645187503 +0000 UTC m=+3.025205250 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84728120-5cde-4b79-944d-7554db40cf41-original-pull-secret") pod "global-pull-secret-syncer-r5jgb" (UID: "84728120-5cde-4b79-944d-7554db40cf41") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:29:39.645286 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:39.645239    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/352c01b0-7c02-4da9-a311-816815cd5e1e-kube-api-access-plpr7 podName:352c01b0-7c02-4da9-a311-816815cd5e1e nodeName:}" failed. No retries permitted until 2026-09-17 15:29:40.645216076 +0000 UTC m=+3.025233811 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-plpr7" (UniqueName: "kubernetes.io/projected/352c01b0-7c02-4da9-a311-816815cd5e1e-kube-api-access-plpr7") pod "network-check-target-l2rxt" (UID: "352c01b0-7c02-4da9-a311-816815cd5e1e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:29:39.745866 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:39.745837    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eae675d6-96cb-4843-9deb-31182313d8ac-metrics-certs\") pod \"network-metrics-daemon-hn89v\" (UID: \"eae675d6-96cb-4843-9deb-31182313d8ac\") " pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:29:39.746032 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:39.745956    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:29:39.746032 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:39.746009    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eae675d6-96cb-4843-9deb-31182313d8ac-metrics-certs podName:eae675d6-96cb-4843-9deb-31182313d8ac nodeName:}" failed. No retries permitted until 2026-09-17 15:29:40.745993396 +0000 UTC m=+3.126011137 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eae675d6-96cb-4843-9deb-31182313d8ac-metrics-certs") pod "network-metrics-daemon-hn89v" (UID: "eae675d6-96cb-4843-9deb-31182313d8ac") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:29:40.092958 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:40.092572    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 15:24:39 +0000 UTC" deadline="2028-04-24 05:47:48.066939456 +0000 UTC"
Sep 17 15:29:40.092958 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:40.092828    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14030h18m7.974118508s"
Sep 17 15:29:40.147329 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:40.147151    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nh9lk" event={"ID":"69c5bd0b-4c06-45dc-9b1b-a9af91086fcd","Type":"ContainerStarted","Data":"093385fd6c86ebf18075959763e1d2b7d75fc9cff18e94c7807ce9ed8a910e04"}
Sep 17 15:29:40.152234 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:40.152048    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c" event={"ID":"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97","Type":"ContainerStarted","Data":"e8ff7f0220822a6c0fb2d16bdc9f6fb665d2b9bd7e58a8f8ec260396b8d48468"}
Sep 17 15:29:40.159130 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:40.159036    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lkm6c" event={"ID":"872b5c05-9984-45aa-82ab-6ab103c385d9","Type":"ContainerStarted","Data":"d448b3f0464ad7459e08e54dbe464ed763758d5a6fe9b071a99729252daafcc9"}
Sep 17 15:29:40.168903 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:40.168809    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s9svr" event={"ID":"fca9ba31-c254-4a6e-8607-fb6a4af8efc2","Type":"ContainerStarted","Data":"7c804fde3f6f03a8351fd8ba255ea6b8db3273980063d2642b7a883f060634c9"}
Sep 17 15:29:40.183966 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:40.183904    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-104.ec2.internal" event={"ID":"623939e1a7296f8c7c9abdfb3dac6ecc","Type":"ContainerStarted","Data":"3eca865f036b87f99a349ae1e7b83f79babd26a57bd991949ba374b7f5dd6af2"}
Sep 17 15:29:40.191871 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:40.191806    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-104.ec2.internal" event={"ID":"4965a8a85b93317838f07968164cf5cd","Type":"ContainerStarted","Data":"80c967e59dfd0bb5b2175c9377117dba6bcd794cb7338932a35d2bb8242b149c"}
Sep 17 15:29:40.210381 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:40.210340    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xfzrp" event={"ID":"593418fc-096d-445c-9863-9ed9f57ee092","Type":"ContainerStarted","Data":"24312b5420119fd5a89282bd46493fcb3dcaf4646202c4d902ebbc3be89c88ff"}
Sep 17 15:29:40.223822 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:40.223781    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r" event={"ID":"6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe","Type":"ContainerStarted","Data":"12f58d88597356228ce79be0a9ef678eae7ad3d2bcdef6368590cc30b6e60bb7"}
Sep 17 15:29:40.252338 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:40.252294    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wxm49" event={"ID":"b0cd1b81-7563-47df-bb6a-4b1f41073a2b","Type":"ContainerStarted","Data":"87a1912c17d5ad1ec5f9ced74a42f5bc9220078a24bac431e68785afff06f5b1"}
Sep 17 15:29:40.289995 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:40.289953    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmrtd" event={"ID":"2b086625-d2ab-4b2e-826b-8cf6a9534826","Type":"ContainerStarted","Data":"b38c3c7a49252e05eeed682ed748c519fc185c940fc7ce1ee7775863b3fa9805"}
Sep 17 15:29:40.292547 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:40.292514    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"egress-router-binary-copy\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:62c7cc09ff9f9684116e4c6b702d22ac67be29319eb961485e46f9441db87fb0\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-multus/multus-additional-cni-plugins-cmrtd" podUID="2b086625-d2ab-4b2e-826b-8cf6a9534826"
Sep 17 15:29:40.303737 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:40.303702    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-857mz" event={"ID":"5c65f398-6a2c-4838-8c61-bcabe5acfca7","Type":"ContainerStarted","Data":"282b884bd5286f4dea435a5f12607015831dad1fc34da3cbcbb9029a695cf277"}
Sep 17 15:29:40.467871 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:40.467594    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:29:40.655947 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:40.655905    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84728120-5cde-4b79-944d-7554db40cf41-original-pull-secret\") pod \"global-pull-secret-syncer-r5jgb\" (UID: \"84728120-5cde-4b79-944d-7554db40cf41\") " pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:29:40.656161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:40.655985    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plpr7\" (UniqueName: \"kubernetes.io/projected/352c01b0-7c02-4da9-a311-816815cd5e1e-kube-api-access-plpr7\") pod \"network-check-target-l2rxt\" (UID: \"352c01b0-7c02-4da9-a311-816815cd5e1e\") " pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:29:40.656226 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:40.656174    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:29:40.656275 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:40.656242    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84728120-5cde-4b79-944d-7554db40cf41-original-pull-secret podName:84728120-5cde-4b79-944d-7554db40cf41 nodeName:}" failed. No retries permitted until 2026-09-17 15:29:42.65622141 +0000 UTC m=+5.036239158 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84728120-5cde-4b79-944d-7554db40cf41-original-pull-secret") pod "global-pull-secret-syncer-r5jgb" (UID: "84728120-5cde-4b79-944d-7554db40cf41") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:29:40.656702 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:40.656680    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:29:40.656789 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:40.656707    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:29:40.656789 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:40.656720    2556 projected.go:196] Error preparing data for projected volume kube-api-access-plpr7 for pod openshift-network-diagnostics/network-check-target-l2rxt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:29:40.656789 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:40.656772    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/352c01b0-7c02-4da9-a311-816815cd5e1e-kube-api-access-plpr7 podName:352c01b0-7c02-4da9-a311-816815cd5e1e nodeName:}" failed. No retries permitted until 2026-09-17 15:29:42.656755319 +0000 UTC m=+5.036773066 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-plpr7" (UniqueName: "kubernetes.io/projected/352c01b0-7c02-4da9-a311-816815cd5e1e-kube-api-access-plpr7") pod "network-check-target-l2rxt" (UID: "352c01b0-7c02-4da9-a311-816815cd5e1e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:29:40.757753 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:40.757113    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eae675d6-96cb-4843-9deb-31182313d8ac-metrics-certs\") pod \"network-metrics-daemon-hn89v\" (UID: \"eae675d6-96cb-4843-9deb-31182313d8ac\") " pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:29:40.757753 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:40.757287    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:29:40.757753 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:40.757348    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eae675d6-96cb-4843-9deb-31182313d8ac-metrics-certs podName:eae675d6-96cb-4843-9deb-31182313d8ac nodeName:}" failed. No retries permitted until 2026-09-17 15:29:42.75733156 +0000 UTC m=+5.137349312 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eae675d6-96cb-4843-9deb-31182313d8ac-metrics-certs") pod "network-metrics-daemon-hn89v" (UID: "eae675d6-96cb-4843-9deb-31182313d8ac") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:29:41.094041 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:41.093889    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 15:24:39 +0000 UTC" deadline="2028-03-02 18:29:34.033585792 +0000 UTC"
Sep 17 15:29:41.094041 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:41.093930    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12770h59m52.939660444s"
Sep 17 15:29:41.130311 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:41.129474    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:29:41.130311 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:41.129612    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5jgb" podUID="84728120-5cde-4b79-944d-7554db40cf41"
Sep 17 15:29:41.130311 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:41.130010    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:29:41.130311 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:41.130118    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2rxt" podUID="352c01b0-7c02-4da9-a311-816815cd5e1e"
Sep 17 15:29:41.130311 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:41.130185    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:29:41.130311 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:41.130258    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn89v" podUID="eae675d6-96cb-4843-9deb-31182313d8ac"
Sep 17 15:29:41.472542 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:41.472450    2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:29:42.678079 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:42.678024    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plpr7\" (UniqueName: \"kubernetes.io/projected/352c01b0-7c02-4da9-a311-816815cd5e1e-kube-api-access-plpr7\") pod \"network-check-target-l2rxt\" (UID: \"352c01b0-7c02-4da9-a311-816815cd5e1e\") " pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:29:42.678544 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:42.678126    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84728120-5cde-4b79-944d-7554db40cf41-original-pull-secret\") pod \"global-pull-secret-syncer-r5jgb\" (UID: \"84728120-5cde-4b79-944d-7554db40cf41\") " pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:29:42.678544 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:42.678217    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:29:42.678544 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:42.678236    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:29:42.678544 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:42.678242    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:29:42.678544 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:42.678260    2556 projected.go:196] Error preparing data for projected volume kube-api-access-plpr7 for pod openshift-network-diagnostics/network-check-target-l2rxt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:29:42.678544 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:42.678302    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84728120-5cde-4b79-944d-7554db40cf41-original-pull-secret podName:84728120-5cde-4b79-944d-7554db40cf41 nodeName:}" failed. No retries permitted until 2026-09-17 15:29:46.678283014 +0000 UTC m=+9.058300751 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84728120-5cde-4b79-944d-7554db40cf41-original-pull-secret") pod "global-pull-secret-syncer-r5jgb" (UID: "84728120-5cde-4b79-944d-7554db40cf41") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:29:42.678544 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:42.678322    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/352c01b0-7c02-4da9-a311-816815cd5e1e-kube-api-access-plpr7 podName:352c01b0-7c02-4da9-a311-816815cd5e1e nodeName:}" failed. No retries permitted until 2026-09-17 15:29:46.678312603 +0000 UTC m=+9.058330343 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-plpr7" (UniqueName: "kubernetes.io/projected/352c01b0-7c02-4da9-a311-816815cd5e1e-kube-api-access-plpr7") pod "network-check-target-l2rxt" (UID: "352c01b0-7c02-4da9-a311-816815cd5e1e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:29:42.779122 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:42.779083    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eae675d6-96cb-4843-9deb-31182313d8ac-metrics-certs\") pod \"network-metrics-daemon-hn89v\" (UID: \"eae675d6-96cb-4843-9deb-31182313d8ac\") " pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:29:42.779349 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:42.779323    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:29:42.779428 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:42.779410    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eae675d6-96cb-4843-9deb-31182313d8ac-metrics-certs podName:eae675d6-96cb-4843-9deb-31182313d8ac nodeName:}" failed. No retries permitted until 2026-09-17 15:29:46.779388919 +0000 UTC m=+9.159406658 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eae675d6-96cb-4843-9deb-31182313d8ac-metrics-certs") pod "network-metrics-daemon-hn89v" (UID: "eae675d6-96cb-4843-9deb-31182313d8ac") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:29:43.130523 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:43.129606    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:29:43.130523 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:43.129762    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn89v" podUID="eae675d6-96cb-4843-9deb-31182313d8ac"
Sep 17 15:29:43.130523 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:43.130195    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:29:43.130523 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:43.130334    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5jgb" podUID="84728120-5cde-4b79-944d-7554db40cf41"
Sep 17 15:29:43.130523 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:43.130388    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:29:43.130523 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:43.130454    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2rxt" podUID="352c01b0-7c02-4da9-a311-816815cd5e1e"
Sep 17 15:29:45.129981 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:45.129938    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:29:45.130492 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:45.129950    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:29:45.130492 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:45.130105    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn89v" podUID="eae675d6-96cb-4843-9deb-31182313d8ac"
Sep 17 15:29:45.130492 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:45.130117    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:29:45.130492 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:45.130223    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5jgb" podUID="84728120-5cde-4b79-944d-7554db40cf41"
Sep 17 15:29:45.130492 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:45.130301    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2rxt" podUID="352c01b0-7c02-4da9-a311-816815cd5e1e"
Sep 17 15:29:46.715523 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:46.715470    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84728120-5cde-4b79-944d-7554db40cf41-original-pull-secret\") pod \"global-pull-secret-syncer-r5jgb\" (UID: \"84728120-5cde-4b79-944d-7554db40cf41\") " pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:29:46.715981 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:46.715553    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plpr7\" (UniqueName: \"kubernetes.io/projected/352c01b0-7c02-4da9-a311-816815cd5e1e-kube-api-access-plpr7\") pod \"network-check-target-l2rxt\" (UID: \"352c01b0-7c02-4da9-a311-816815cd5e1e\") " pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:29:46.715981 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:46.715693    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:29:46.715981 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:46.715723    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:29:46.715981 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:46.715748    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:29:46.715981 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:46.715762    2556 projected.go:196] Error preparing data for projected volume kube-api-access-plpr7 for pod openshift-network-diagnostics/network-check-target-l2rxt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:29:46.715981 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:46.715767    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84728120-5cde-4b79-944d-7554db40cf41-original-pull-secret podName:84728120-5cde-4b79-944d-7554db40cf41 nodeName:}" failed. No retries permitted until 2026-09-17 15:29:54.71575246 +0000 UTC m=+17.095770198 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84728120-5cde-4b79-944d-7554db40cf41-original-pull-secret") pod "global-pull-secret-syncer-r5jgb" (UID: "84728120-5cde-4b79-944d-7554db40cf41") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:29:46.715981 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:46.715813    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/352c01b0-7c02-4da9-a311-816815cd5e1e-kube-api-access-plpr7 podName:352c01b0-7c02-4da9-a311-816815cd5e1e nodeName:}" failed. No retries permitted until 2026-09-17 15:29:54.715796749 +0000 UTC m=+17.095814487 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-plpr7" (UniqueName: "kubernetes.io/projected/352c01b0-7c02-4da9-a311-816815cd5e1e-kube-api-access-plpr7") pod "network-check-target-l2rxt" (UID: "352c01b0-7c02-4da9-a311-816815cd5e1e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:29:46.816765 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:46.816727    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eae675d6-96cb-4843-9deb-31182313d8ac-metrics-certs\") pod \"network-metrics-daemon-hn89v\" (UID: \"eae675d6-96cb-4843-9deb-31182313d8ac\") " pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:29:46.816960 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:46.816879    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:29:46.816960 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:46.816952    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eae675d6-96cb-4843-9deb-31182313d8ac-metrics-certs podName:eae675d6-96cb-4843-9deb-31182313d8ac nodeName:}" failed. No retries permitted until 2026-09-17 15:29:54.816931963 +0000 UTC m=+17.196949705 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eae675d6-96cb-4843-9deb-31182313d8ac-metrics-certs") pod "network-metrics-daemon-hn89v" (UID: "eae675d6-96cb-4843-9deb-31182313d8ac") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:29:47.130322 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:47.129891    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:29:47.130322 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:47.130033    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5jgb" podUID="84728120-5cde-4b79-944d-7554db40cf41"
Sep 17 15:29:47.130322 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:47.130084    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:29:47.130322 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:47.130108    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:29:47.130322 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:47.130178    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2rxt" podUID="352c01b0-7c02-4da9-a311-816815cd5e1e"
Sep 17 15:29:47.130322 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:47.130283    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn89v" podUID="eae675d6-96cb-4843-9deb-31182313d8ac"
Sep 17 15:29:49.130266 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:49.130224    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:29:49.130720 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:49.130224    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:29:49.130720 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:49.130343    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2rxt" podUID="352c01b0-7c02-4da9-a311-816815cd5e1e"
Sep 17 15:29:49.130720 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:49.130232    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:29:49.130720 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:49.130427    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5jgb" podUID="84728120-5cde-4b79-944d-7554db40cf41"
Sep 17 15:29:49.130720 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:49.130526    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn89v" podUID="eae675d6-96cb-4843-9deb-31182313d8ac"
Sep 17 15:29:51.129381 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:51.129340    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:29:51.129970 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:51.129409    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:29:51.129970 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:51.129356    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:29:51.129970 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:51.129492    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5jgb" podUID="84728120-5cde-4b79-944d-7554db40cf41"
Sep 17 15:29:51.129970 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:51.129606    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2rxt" podUID="352c01b0-7c02-4da9-a311-816815cd5e1e"
Sep 17 15:29:51.129970 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:51.129694    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn89v" podUID="eae675d6-96cb-4843-9deb-31182313d8ac"
Sep 17 15:29:53.129351 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:53.129308    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:29:53.129871 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:53.129320    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:29:53.129871 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:53.129572    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2rxt" podUID="352c01b0-7c02-4da9-a311-816815cd5e1e"
Sep 17 15:29:53.129871 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:53.129441    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5jgb" podUID="84728120-5cde-4b79-944d-7554db40cf41"
Sep 17 15:29:53.129871 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:53.129328    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:29:53.129871 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:53.129687    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn89v" podUID="eae675d6-96cb-4843-9deb-31182313d8ac"
Sep 17 15:29:54.779510 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:54.779471    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84728120-5cde-4b79-944d-7554db40cf41-original-pull-secret\") pod \"global-pull-secret-syncer-r5jgb\" (UID: \"84728120-5cde-4b79-944d-7554db40cf41\") " pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:29:54.780010 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:54.779532    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plpr7\" (UniqueName: \"kubernetes.io/projected/352c01b0-7c02-4da9-a311-816815cd5e1e-kube-api-access-plpr7\") pod \"network-check-target-l2rxt\" (UID: \"352c01b0-7c02-4da9-a311-816815cd5e1e\") " pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:29:54.780010 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:54.779635    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:29:54.780010 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:54.779646    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:29:54.780010 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:54.779663    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:29:54.780010 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:54.779675    2556 projected.go:196] Error preparing data for projected volume kube-api-access-plpr7 for pod openshift-network-diagnostics/network-check-target-l2rxt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:29:54.780010 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:54.779706    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84728120-5cde-4b79-944d-7554db40cf41-original-pull-secret podName:84728120-5cde-4b79-944d-7554db40cf41 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:10.779687972 +0000 UTC m=+33.159705719 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84728120-5cde-4b79-944d-7554db40cf41-original-pull-secret") pod "global-pull-secret-syncer-r5jgb" (UID: "84728120-5cde-4b79-944d-7554db40cf41") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:29:54.780010 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:54.779726    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/352c01b0-7c02-4da9-a311-816815cd5e1e-kube-api-access-plpr7 podName:352c01b0-7c02-4da9-a311-816815cd5e1e nodeName:}" failed. No retries permitted until 2026-09-17 15:30:10.77971739 +0000 UTC m=+33.159735125 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-plpr7" (UniqueName: "kubernetes.io/projected/352c01b0-7c02-4da9-a311-816815cd5e1e-kube-api-access-plpr7") pod "network-check-target-l2rxt" (UID: "352c01b0-7c02-4da9-a311-816815cd5e1e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:29:54.879962 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:54.879920    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eae675d6-96cb-4843-9deb-31182313d8ac-metrics-certs\") pod \"network-metrics-daemon-hn89v\" (UID: \"eae675d6-96cb-4843-9deb-31182313d8ac\") " pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:29:54.880167 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:54.880088    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:29:54.880167 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:54.880151    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eae675d6-96cb-4843-9deb-31182313d8ac-metrics-certs podName:eae675d6-96cb-4843-9deb-31182313d8ac nodeName:}" failed. No retries permitted until 2026-09-17 15:30:10.880137783 +0000 UTC m=+33.260155517 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eae675d6-96cb-4843-9deb-31182313d8ac-metrics-certs") pod "network-metrics-daemon-hn89v" (UID: "eae675d6-96cb-4843-9deb-31182313d8ac") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:29:55.129847 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:55.129740    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:29:55.129847 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:55.129740    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:29:55.130107 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:55.129894    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn89v" podUID="eae675d6-96cb-4843-9deb-31182313d8ac"
Sep 17 15:29:55.130107 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:55.129740    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:29:55.130107 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:55.129950    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2rxt" podUID="352c01b0-7c02-4da9-a311-816815cd5e1e"
Sep 17 15:29:55.130107 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:55.130048    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5jgb" podUID="84728120-5cde-4b79-944d-7554db40cf41"
Sep 17 15:29:57.129497 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:57.129463    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:29:57.129862 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:57.129583    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:29:57.129862 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:57.129697    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:29:57.129862 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:57.129724    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5jgb" podUID="84728120-5cde-4b79-944d-7554db40cf41"
Sep 17 15:29:57.129862 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:57.129782    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2rxt" podUID="352c01b0-7c02-4da9-a311-816815cd5e1e"
Sep 17 15:29:57.129862 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:57.129825    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn89v" podUID="eae675d6-96cb-4843-9deb-31182313d8ac"
Sep 17 15:29:57.368441 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:57.368233    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-104.ec2.internal" event={"ID":"4965a8a85b93317838f07968164cf5cd","Type":"ContainerStarted","Data":"b7cf34738223885acfee8ec47eb2ca440a9f26c1026ab0a80b611d92ce1ac500"}
Sep 17 15:29:57.375549 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:57.374655    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-857mz" event={"ID":"5c65f398-6a2c-4838-8c61-bcabe5acfca7","Type":"ContainerStarted","Data":"64f63906843dd6d29c788c6ee7c238c8a40a04a11ba64235f30c5d98a7305d19"}
Sep 17 15:29:57.383366 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:57.383333    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c" event={"ID":"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97","Type":"ContainerStarted","Data":"b57ac950f69fb51108003c5cfc243b572af6bf999b17e1d047221eb024cbaf4c"}
Sep 17 15:29:57.387735 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:57.387672    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-104.ec2.internal" podStartSLOduration=19.387652962 podStartE2EDuration="19.387652962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:29:57.387335769 +0000 UTC m=+19.767353526" watchObservedRunningTime="2026-09-17 15:29:57.387652962 +0000 UTC m=+19.767670720"
Sep 17 15:29:58.387725 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:58.387391    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nh9lk" event={"ID":"69c5bd0b-4c06-45dc-9b1b-a9af91086fcd","Type":"ContainerStarted","Data":"8c2ee4f14e245909d1f484ebb09b2468ed0900c9205679ff9f3df0e247716663"}
Sep 17 15:29:58.390938 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:58.390902    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c" event={"ID":"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97","Type":"ContainerStarted","Data":"cd9d5c619a9e813bbcca5b8c249cfc01bb0f7eb64292f6dc85d1d125e352956d"}
Sep 17 15:29:58.391153 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:58.390991    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c" event={"ID":"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97","Type":"ContainerStarted","Data":"25afa5f6539ecd4f90a74fdd239bcf92b9d89a28e85c53454905cb74681d3f98"}
Sep 17 15:29:58.391153 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:58.391040    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c" event={"ID":"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97","Type":"ContainerStarted","Data":"d64fe7c41ed323106f7de9bedccaf52ae49251874eb803576db15c03b26cd427"}
Sep 17 15:29:58.391153 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:58.391114    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c" event={"ID":"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97","Type":"ContainerStarted","Data":"5f154ed5e64a3cb720451310997b4e5b51b26696db2db82db184da1efe66366e"}
Sep 17 15:29:58.391153 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:58.391153    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c" event={"ID":"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97","Type":"ContainerStarted","Data":"1e169467d20687cb9e194067d0537f490ac100a629c8a588d150ca79584e579b"}
Sep 17 15:29:58.392563 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:58.392510    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lkm6c" event={"ID":"872b5c05-9984-45aa-82ab-6ab103c385d9","Type":"ContainerStarted","Data":"ebd148283d85a8cce8541808ae9fc448ffe8e3d18fb2a863c6b924a9614fbd37"}
Sep 17 15:29:58.394125 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:58.394097    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s9svr" event={"ID":"fca9ba31-c254-4a6e-8607-fb6a4af8efc2","Type":"ContainerStarted","Data":"bd8300bf1a724c88454de8b34758d8c4fbb7d336e91be647e551189652ea8ec3"}
Sep 17 15:29:58.395763 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:58.395740    2556 generic.go:356] "Generic (PLEG): container finished" podID="623939e1a7296f8c7c9abdfb3dac6ecc" containerID="3724f87a753d886670d65c94851ca438dade752464202daa5fd2f5cba153ceac" exitCode=0
Sep 17 15:29:58.395868 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:58.395832    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-104.ec2.internal" event={"ID":"623939e1a7296f8c7c9abdfb3dac6ecc","Type":"ContainerDied","Data":"3724f87a753d886670d65c94851ca438dade752464202daa5fd2f5cba153ceac"}
Sep 17 15:29:58.395972 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:58.395959    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-104.ec2.internal"
Sep 17 15:29:58.397420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:58.397395    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xfzrp" event={"ID":"593418fc-096d-445c-9863-9ed9f57ee092","Type":"ContainerStarted","Data":"a3c57c24559564a6dcbf0679d8cff50f33072543e726489875f26e9add2bf200"}
Sep 17 15:29:58.400005 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:58.399979    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r" event={"ID":"6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe","Type":"ContainerStarted","Data":"2a837ad2cb3a728a42a510ddfc4cb68d54411c0f88ba1daedb140c986685fee1"}
Sep 17 15:29:58.407407 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:58.407322    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nh9lk" podStartSLOduration=2.64769132 podStartE2EDuration="20.404455351s" totalImagesPullingTime="17.756764031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:29:58.403665587 +0000 UTC m=+20.783683344" watchObservedRunningTime="2026-09-17 15:29:58.404455351 +0000 UTC m=+20.784473110"
Sep 17 15:29:58.407888 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:58.407855    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-857mz" podStartSLOduration=2.644602755 podStartE2EDuration="20.40784258s" totalImagesPullingTime="17.763239825s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:29:57.410863055 +0000 UTC m=+19.790880812" watchObservedRunningTime="2026-09-17 15:29:58.40784258 +0000 UTC m=+20.787860335"
Sep 17 15:29:58.409036 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:58.409018    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-104.ec2.internal"]
Sep 17 15:29:58.409710 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:58.409690    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 15:29:58.440137 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:58.440043    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xfzrp" podStartSLOduration=2.749364666 podStartE2EDuration="20.440026563s" totalImagesPullingTime="17.690661897s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:29:58.423617388 +0000 UTC m=+20.803635143" watchObservedRunningTime="2026-09-17 15:29:58.440026563 +0000 UTC m=+20.820044319"
Sep 17 15:29:58.440511 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:58.440477    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-s9svr" podStartSLOduration=6.171584033 podStartE2EDuration="20.44046573s" totalImagesPullingTime="14.268881697s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:29:58.439727912 +0000 UTC m=+20.819745668" watchObservedRunningTime="2026-09-17 15:29:58.44046573 +0000 UTC m=+20.820483487"
Sep 17 15:29:58.476931 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:58.476816    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lkm6c" podStartSLOduration=2.541391915 podStartE2EDuration="20.476791857s" totalImagesPullingTime="17.935399942s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:29:58.476075457 +0000 UTC m=+20.856093207" watchObservedRunningTime="2026-09-17 15:29:58.476791857 +0000 UTC m=+20.856809613"
Sep 17 15:29:59.129436 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:59.129407    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:29:59.129559 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:59.129504    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2rxt" podUID="352c01b0-7c02-4da9-a311-816815cd5e1e"
Sep 17 15:29:59.129559 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:59.129425    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:29:59.129559 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:59.129418    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:29:59.129723 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:59.129585    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn89v" podUID="eae675d6-96cb-4843-9deb-31182313d8ac"
Sep 17 15:29:59.129723 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:29:59.129683    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5jgb" podUID="84728120-5cde-4b79-944d-7554db40cf41"
Sep 17 15:29:59.144383 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:59.144359    2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 15:29:59.406006 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:59.405905    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-104.ec2.internal" event={"ID":"623939e1a7296f8c7c9abdfb3dac6ecc","Type":"ContainerStarted","Data":"51463cd12b539269a618f69aa7b7dfcb1f55e1f628bc1a3d74097e3c3fec161d"}
Sep 17 15:29:59.407838 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:59.407815    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r" event={"ID":"6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe","Type":"ContainerStarted","Data":"aea1466ed5975a6dfc0e49d8fdef0fc557e78f92391b644dfe1f790fe4aa1b24"}
Sep 17 15:29:59.409350 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:59.409310    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wxm49" event={"ID":"b0cd1b81-7563-47df-bb6a-4b1f41073a2b","Type":"ContainerStarted","Data":"4a2ca10870ee2fc82665257a2c5b3fc826d48f43e66b9dc6a89639b71f135ed0"}
Sep 17 15:29:59.411033 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:59.411011    2556 generic.go:356] "Generic (PLEG): container finished" podID="2b086625-d2ab-4b2e-826b-8cf6a9534826" containerID="6415859c8e8351b79773cdb59094a6fedf69a6f82a5e27c0c7a44f3c33f85922" exitCode=0
Sep 17 15:29:59.411142 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:59.411121    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmrtd" event={"ID":"2b086625-d2ab-4b2e-826b-8cf6a9534826","Type":"ContainerDied","Data":"6415859c8e8351b79773cdb59094a6fedf69a6f82a5e27c0c7a44f3c33f85922"}
Sep 17 15:29:59.423737 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:59.423697    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-104.ec2.internal" podStartSLOduration=1.423679632 podStartE2EDuration="1.423679632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:29:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:29:59.423474924 +0000 UTC m=+21.803492690" watchObservedRunningTime="2026-09-17 15:29:59.423679632 +0000 UTC m=+21.803697387"
Sep 17 15:29:59.495554 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:29:59.495500    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wxm49" podStartSLOduration=3.65660016 podStartE2EDuration="21.495477574s" totalImagesPullingTime="17.838877414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:29:59.457468041 +0000 UTC m=+21.837485796" watchObservedRunningTime="2026-09-17 15:29:59.495477574 +0000 UTC m=+21.875495333"
Sep 17 15:30:00.096603 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:00.096198    2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T15:29:59.144378978Z","UUID":"0a3edc6e-4c7d-4529-8c36-d0d86abb4cbd","Handler":null,"Name":"","Endpoint":""}
Sep 17 15:30:00.097600 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:00.097575    2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 15:30:00.097600 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:00.097607    2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 15:30:00.416538 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:00.416447    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r" event={"ID":"6e5eacd8-cf9b-4ecb-9004-9fb59aefdffe","Type":"ContainerStarted","Data":"65e4f6d6b1616ca0fb3baf5b942feb6533175d0447dccb5946c4bdeba6b7c2f2"}
Sep 17 15:30:00.420883 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:00.420854    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c" event={"ID":"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97","Type":"ContainerStarted","Data":"0557e7390a5c11729a8520c8399bb3058850306e7a1c639242200e5512b87217"}
Sep 17 15:30:01.129876 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:01.129839    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:30:01.130104 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:01.129880    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:30:01.130104 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:01.129880    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:30:01.130104 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:01.129979    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5jgb" podUID="84728120-5cde-4b79-944d-7554db40cf41"
Sep 17 15:30:01.130104 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:01.130055    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2rxt" podUID="352c01b0-7c02-4da9-a311-816815cd5e1e"
Sep 17 15:30:01.130325 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:01.130149    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn89v" podUID="eae675d6-96cb-4843-9deb-31182313d8ac"
Sep 17 15:30:01.624928 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:01.624837    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nh9lk"
Sep 17 15:30:01.625481 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:01.625460    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nh9lk"
Sep 17 15:30:01.645281 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:01.645227    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ftp6r" podStartSLOduration=3.816782444 podStartE2EDuration="23.645205523s" totalImagesPullingTime="19.828423079s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:29:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:00.446012305 +0000 UTC m=+22.826030064" watchObservedRunningTime="2026-09-17 15:30:01.645205523 +0000 UTC m=+24.025223281"
Sep 17 15:30:02.425919 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:02.425892    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nh9lk"
Sep 17 15:30:02.426467 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:02.426444    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nh9lk"
Sep 17 15:30:03.129488 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:03.129434    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:30:03.129488 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:03.129454    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:30:03.130034 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:03.129548    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:30:03.130034 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:03.129556    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2rxt" podUID="352c01b0-7c02-4da9-a311-816815cd5e1e"
Sep 17 15:30:03.130034 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:03.129641    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5jgb" podUID="84728120-5cde-4b79-944d-7554db40cf41"
Sep 17 15:30:03.130034 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:03.129740    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn89v" podUID="eae675d6-96cb-4843-9deb-31182313d8ac"
Sep 17 15:30:04.432371 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:04.432180    2556 generic.go:356] "Generic (PLEG): container finished" podID="2b086625-d2ab-4b2e-826b-8cf6a9534826" containerID="2395e3c40c970ea0f3ab9ccd103e996619d4f4c6b2f4be835eba57a208ca92e9" exitCode=0
Sep 17 15:30:04.432962 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:04.432253    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmrtd" event={"ID":"2b086625-d2ab-4b2e-826b-8cf6a9534826","Type":"ContainerDied","Data":"2395e3c40c970ea0f3ab9ccd103e996619d4f4c6b2f4be835eba57a208ca92e9"}
Sep 17 15:30:04.436542 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:04.436514    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c" event={"ID":"15b9edd1-debb-4dc7-8f59-71e2bdaa2d97","Type":"ContainerStarted","Data":"9e2811e476e48a02c302d7c8ac382639734309fee81e47ce460f2cba1de1cfad"}
Sep 17 15:30:04.484468 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:04.484418    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c" podStartSLOduration=8.647214626 podStartE2EDuration="26.484398689s" totalImagesPullingTime="17.837184063s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:04.483118377 +0000 UTC m=+26.863136132" watchObservedRunningTime="2026-09-17 15:30:04.484398689 +0000 UTC m=+26.864416474"
Sep 17 15:30:05.130017 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:05.129981    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:30:05.130185 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:05.130137    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:30:05.130185 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:05.130162    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5jgb" podUID="84728120-5cde-4b79-944d-7554db40cf41"
Sep 17 15:30:05.130274 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:05.130225    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2rxt" podUID="352c01b0-7c02-4da9-a311-816815cd5e1e"
Sep 17 15:30:05.130274 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:05.130243    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:30:05.130347 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:05.130330    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn89v" podUID="eae675d6-96cb-4843-9deb-31182313d8ac"
Sep 17 15:30:05.336362 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:05.336330    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r5jgb"]
Sep 17 15:30:05.339348 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:05.339319    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hn89v"]
Sep 17 15:30:05.341929 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:05.341901    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l2rxt"]
Sep 17 15:30:05.438587 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:05.438501    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:30:05.439039 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:05.438502    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:30:05.439039 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:05.438600    2556 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 15:30:05.439039 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:05.438630    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5jgb" podUID="84728120-5cde-4b79-944d-7554db40cf41"
Sep 17 15:30:05.439039 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:05.438502    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:30:05.439039 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:05.438680    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2rxt" podUID="352c01b0-7c02-4da9-a311-816815cd5e1e"
Sep 17 15:30:05.439339 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:05.439172    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn89v" podUID="eae675d6-96cb-4843-9deb-31182313d8ac"
Sep 17 15:30:05.439339 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:05.439212    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:30:05.439339 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:05.439289    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:30:05.454635 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:05.454603    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:30:05.454781 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:05.454731    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:30:06.443419 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:06.443381    2556 generic.go:356] "Generic (PLEG): container finished" podID="2b086625-d2ab-4b2e-826b-8cf6a9534826" containerID="3355ded44f645bdbc484ef733cab9d1e4eede194ce5651d880a1c689cd20d2e1" exitCode=0
Sep 17 15:30:06.444131 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:06.443452    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmrtd" event={"ID":"2b086625-d2ab-4b2e-826b-8cf6a9534826","Type":"ContainerDied","Data":"3355ded44f645bdbc484ef733cab9d1e4eede194ce5651d880a1c689cd20d2e1"}
Sep 17 15:30:06.444131 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:06.443539    2556 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 15:30:07.129857 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:07.129660    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:30:07.130047 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:07.129702    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:30:07.130047 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:07.129932    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5jgb" podUID="84728120-5cde-4b79-944d-7554db40cf41"
Sep 17 15:30:07.130047 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:07.129706    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:30:07.130047 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:07.130023    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2rxt" podUID="352c01b0-7c02-4da9-a311-816815cd5e1e"
Sep 17 15:30:07.130230 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:07.130114    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn89v" podUID="eae675d6-96cb-4843-9deb-31182313d8ac"
Sep 17 15:30:07.448369 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:07.448281    2556 generic.go:356] "Generic (PLEG): container finished" podID="2b086625-d2ab-4b2e-826b-8cf6a9534826" containerID="b3ee7f8b1ccd9fd68fbdd05d05921bc593b18e1753fb47b4c7aeba89508fbf86" exitCode=0
Sep 17 15:30:07.448369 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:07.448347    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmrtd" event={"ID":"2b086625-d2ab-4b2e-826b-8cf6a9534826","Type":"ContainerDied","Data":"b3ee7f8b1ccd9fd68fbdd05d05921bc593b18e1753fb47b4c7aeba89508fbf86"}
Sep 17 15:30:07.448777 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:07.448429    2556 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 15:30:08.932770 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:08.932671    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:30:08.933329 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:08.932931    2556 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 15:30:08.944853 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:08.944830    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xjz9c"
Sep 17 15:30:09.129856 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:09.129819    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:30:09.129856 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:09.129841    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:30:09.130113 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:09.129843    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:30:09.130113 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:09.129959    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2rxt" podUID="352c01b0-7c02-4da9-a311-816815cd5e1e"
Sep 17 15:30:09.130218 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:09.130100    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5jgb" podUID="84728120-5cde-4b79-944d-7554db40cf41"
Sep 17 15:30:09.130270 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:09.130242    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hn89v" podUID="eae675d6-96cb-4843-9deb-31182313d8ac"
Sep 17 15:30:09.934680 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:09.934648    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-104.ec2.internal" event="NodeReady"
Sep 17 15:30:09.935098 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:09.934804    2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 15:30:09.977655 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:09.977616    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69d6bcf58c-bq4rc"]
Sep 17 15:30:09.985586 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:09.985559    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:09.989255 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:09.989231    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lzxqs\""
Sep 17 15:30:09.990732 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:09.990708    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 15:30:09.991014 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:09.990996    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 15:30:09.991130 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:09.991088    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 15:30:09.994406 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:09.994338    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-k7kbh"]
Sep 17 15:30:10.001795 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.001769    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k7kbh"
Sep 17 15:30:10.003485 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.003466    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69d6bcf58c-bq4rc"]
Sep 17 15:30:10.006035 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.005943    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 15:30:10.006927 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.006734    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l7hr4\""
Sep 17 15:30:10.006927 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.006805    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 15:30:10.007727 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.007667    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-r975h"]
Sep 17 15:30:10.007917 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.007896    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 15:30:10.022128 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.022094    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k7kbh"]
Sep 17 15:30:10.022256 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.022224    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r975h"
Sep 17 15:30:10.030144 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.029890    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 15:30:10.030144 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.029934    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r975h"]
Sep 17 15:30:10.030144 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.029937    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 15:30:10.030144 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.030015    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bcwx7\""
Sep 17 15:30:10.030144 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.030029    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 15:30:10.091167 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.091089    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2581da35-8109-4df3-a553-c8992a31b8d1-registry-certificates\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:10.091167 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.091137    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2581da35-8109-4df3-a553-c8992a31b8d1-ca-trust-extracted\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:10.091411 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.091181    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de6d8a3f-074e-4f18-9650-0ca1cb32100f-metrics-tls\") pod \"dns-default-k7kbh\" (UID: \"de6d8a3f-074e-4f18-9650-0ca1cb32100f\") " pod="openshift-dns/dns-default-k7kbh"
Sep 17 15:30:10.091411 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.091202    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz9sr\" (UniqueName: \"kubernetes.io/projected/ac3c1f9c-dbfd-4097-b8c5-50d3fa2962c5-kube-api-access-hz9sr\") pod \"ingress-canary-r975h\" (UID: \"ac3c1f9c-dbfd-4097-b8c5-50d3fa2962c5\") " pod="openshift-ingress-canary/ingress-canary-r975h"
Sep 17 15:30:10.091411 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.091236    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjzgp\" (UniqueName: \"kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-kube-api-access-wjzgp\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:10.091411 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.091260    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2581da35-8109-4df3-a553-c8992a31b8d1-trusted-ca\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:10.091411 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.091297    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac3c1f9c-dbfd-4097-b8c5-50d3fa2962c5-cert\") pod \"ingress-canary-r975h\" (UID: \"ac3c1f9c-dbfd-4097-b8c5-50d3fa2962c5\") " pod="openshift-ingress-canary/ingress-canary-r975h"
Sep 17 15:30:10.091411 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.091338    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2581da35-8109-4df3-a553-c8992a31b8d1-installation-pull-secrets\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:10.091411 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.091353    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de6d8a3f-074e-4f18-9650-0ca1cb32100f-tmp-dir\") pod \"dns-default-k7kbh\" (UID: \"de6d8a3f-074e-4f18-9650-0ca1cb32100f\") " pod="openshift-dns/dns-default-k7kbh"
Sep 17 15:30:10.091411 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.091374    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2581da35-8109-4df3-a553-c8992a31b8d1-image-registry-private-configuration\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:10.091411 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.091397    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-registry-tls\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:10.091783 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.091418    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/de6d8a3f-074e-4f18-9650-0ca1cb32100f-config-volume\") pod \"dns-default-k7kbh\" (UID: \"de6d8a3f-074e-4f18-9650-0ca1cb32100f\") " pod="openshift-dns/dns-default-k7kbh"
Sep 17 15:30:10.091783 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.091480    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdwc5\" (UniqueName: \"kubernetes.io/projected/de6d8a3f-074e-4f18-9650-0ca1cb32100f-kube-api-access-tdwc5\") pod \"dns-default-k7kbh\" (UID: \"de6d8a3f-074e-4f18-9650-0ca1cb32100f\") " pod="openshift-dns/dns-default-k7kbh"
Sep 17 15:30:10.091783 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.091518    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-bound-sa-token\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:10.116554 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.116522    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:10.192296 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.192259    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac3c1f9c-dbfd-4097-b8c5-50d3fa2962c5-cert\") pod \"ingress-canary-r975h\" (UID: \"ac3c1f9c-dbfd-4097-b8c5-50d3fa2962c5\") " pod="openshift-ingress-canary/ingress-canary-r975h"
Sep 17 15:30:10.192484 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.192312    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2581da35-8109-4df3-a553-c8992a31b8d1-installation-pull-secrets\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:10.192484 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.192338    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de6d8a3f-074e-4f18-9650-0ca1cb32100f-tmp-dir\") pod \"dns-default-k7kbh\" (UID: \"de6d8a3f-074e-4f18-9650-0ca1cb32100f\") " pod="openshift-dns/dns-default-k7kbh"
Sep 17 15:30:10.192484 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.192369    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2581da35-8109-4df3-a553-c8992a31b8d1-image-registry-private-configuration\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:10.192653 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.192490    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-registry-tls\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:10.192653 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.192535    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/de6d8a3f-074e-4f18-9650-0ca1cb32100f-config-volume\") pod \"dns-default-k7kbh\" (UID: \"de6d8a3f-074e-4f18-9650-0ca1cb32100f\") " pod="openshift-dns/dns-default-k7kbh"
Sep 17 15:30:10.192653 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.192589    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdwc5\" (UniqueName: \"kubernetes.io/projected/de6d8a3f-074e-4f18-9650-0ca1cb32100f-kube-api-access-tdwc5\") pod \"dns-default-k7kbh\" (UID: \"de6d8a3f-074e-4f18-9650-0ca1cb32100f\") " pod="openshift-dns/dns-default-k7kbh"
Sep 17 15:30:10.192653 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.192621    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-bound-sa-token\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:10.192845 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.192653    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2581da35-8109-4df3-a553-c8992a31b8d1-registry-certificates\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:10.192845 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.192667    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de6d8a3f-074e-4f18-9650-0ca1cb32100f-tmp-dir\") pod \"dns-default-k7kbh\" (UID: \"de6d8a3f-074e-4f18-9650-0ca1cb32100f\") " pod="openshift-dns/dns-default-k7kbh"
Sep 17 15:30:10.192845 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.192720    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2581da35-8109-4df3-a553-c8992a31b8d1-ca-trust-extracted\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:10.192845 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.192774    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de6d8a3f-074e-4f18-9650-0ca1cb32100f-metrics-tls\") pod \"dns-default-k7kbh\" (UID: \"de6d8a3f-074e-4f18-9650-0ca1cb32100f\") " pod="openshift-dns/dns-default-k7kbh"
Sep 17 15:30:10.192845 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.192800    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz9sr\" (UniqueName: \"kubernetes.io/projected/ac3c1f9c-dbfd-4097-b8c5-50d3fa2962c5-kube-api-access-hz9sr\") pod \"ingress-canary-r975h\" (UID: \"ac3c1f9c-dbfd-4097-b8c5-50d3fa2962c5\") " pod="openshift-ingress-canary/ingress-canary-r975h"
Sep 17 15:30:10.192845 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.192837    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjzgp\" (UniqueName: \"kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-kube-api-access-wjzgp\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:10.193164 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.192868    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2581da35-8109-4df3-a553-c8992a31b8d1-trusted-ca\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:10.193164 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:10.193126    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:30:10.193164 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:10.193142    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69d6bcf58c-bq4rc: secret "image-registry-tls" not found
Sep 17 15:30:10.193318 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:10.193194    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-registry-tls podName:2581da35-8109-4df3-a553-c8992a31b8d1 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:10.693176472 +0000 UTC m=+33.073194208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-registry-tls") pod "image-registry-69d6bcf58c-bq4rc" (UID: "2581da35-8109-4df3-a553-c8992a31b8d1") : secret "image-registry-tls" not found
Sep 17 15:30:10.193318 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.193200    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/de6d8a3f-074e-4f18-9650-0ca1cb32100f-config-volume\") pod \"dns-default-k7kbh\" (UID: \"de6d8a3f-074e-4f18-9650-0ca1cb32100f\") " pod="openshift-dns/dns-default-k7kbh"
Sep 17 15:30:10.193427 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:10.193419    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:30:10.193479 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:10.193456    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de6d8a3f-074e-4f18-9650-0ca1cb32100f-metrics-tls podName:de6d8a3f-074e-4f18-9650-0ca1cb32100f nodeName:}" failed. No retries permitted until 2026-09-17 15:30:10.693444362 +0000 UTC m=+33.073462099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de6d8a3f-074e-4f18-9650-0ca1cb32100f-metrics-tls") pod "dns-default-k7kbh" (UID: "de6d8a3f-074e-4f18-9650-0ca1cb32100f") : secret "dns-default-metrics-tls" not found
Sep 17 15:30:10.193851 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.193777    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2581da35-8109-4df3-a553-c8992a31b8d1-ca-trust-extracted\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:10.193969 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.193887    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2581da35-8109-4df3-a553-c8992a31b8d1-registry-certificates\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:10.194029 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.193979    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2581da35-8109-4df3-a553-c8992a31b8d1-trusted-ca\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:10.196496 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.196470    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2581da35-8109-4df3-a553-c8992a31b8d1-image-registry-private-configuration\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:10.196573 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.196499    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2581da35-8109-4df3-a553-c8992a31b8d1-installation-pull-secrets\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:10.196573 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.196548    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac3c1f9c-dbfd-4097-b8c5-50d3fa2962c5-cert\") pod \"ingress-canary-r975h\" (UID: \"ac3c1f9c-dbfd-4097-b8c5-50d3fa2962c5\") " pod="openshift-ingress-canary/ingress-canary-r975h"
Sep 17 15:30:10.218720 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.218686    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz9sr\" (UniqueName: \"kubernetes.io/projected/ac3c1f9c-dbfd-4097-b8c5-50d3fa2962c5-kube-api-access-hz9sr\") pod \"ingress-canary-r975h\" (UID: \"ac3c1f9c-dbfd-4097-b8c5-50d3fa2962c5\") " pod="openshift-ingress-canary/ingress-canary-r975h"
Sep 17 15:30:10.219920 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.219891    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjzgp\" (UniqueName: \"kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-kube-api-access-wjzgp\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:10.220193 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.220175    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-bound-sa-token\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:10.223789 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.223762    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdwc5\" (UniqueName: \"kubernetes.io/projected/de6d8a3f-074e-4f18-9650-0ca1cb32100f-kube-api-access-tdwc5\") pod \"dns-default-k7kbh\" (UID: \"de6d8a3f-074e-4f18-9650-0ca1cb32100f\") " pod="openshift-dns/dns-default-k7kbh"
Sep 17 15:30:10.293581 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.293538    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:10.332729 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.332683    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r975h"
Sep 17 15:30:10.398669 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.398628    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:10.695814 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.695732    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-registry-tls\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:10.695970 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.695849    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de6d8a3f-074e-4f18-9650-0ca1cb32100f-metrics-tls\") pod \"dns-default-k7kbh\" (UID: \"de6d8a3f-074e-4f18-9650-0ca1cb32100f\") " pod="openshift-dns/dns-default-k7kbh"
Sep 17 15:30:10.695970 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:10.695873    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:30:10.695970 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:10.695894    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69d6bcf58c-bq4rc: secret "image-registry-tls" not found
Sep 17 15:30:10.695970 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:10.695957    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-registry-tls podName:2581da35-8109-4df3-a553-c8992a31b8d1 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:11.695937198 +0000 UTC m=+34.075954952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-registry-tls") pod "image-registry-69d6bcf58c-bq4rc" (UID: "2581da35-8109-4df3-a553-c8992a31b8d1") : secret "image-registry-tls" not found
Sep 17 15:30:10.695970 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:10.695960    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:30:10.696190 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:10.696008    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de6d8a3f-074e-4f18-9650-0ca1cb32100f-metrics-tls podName:de6d8a3f-074e-4f18-9650-0ca1cb32100f nodeName:}" failed. No retries permitted until 2026-09-17 15:30:11.695992839 +0000 UTC m=+34.076010575 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de6d8a3f-074e-4f18-9650-0ca1cb32100f-metrics-tls") pod "dns-default-k7kbh" (UID: "de6d8a3f-074e-4f18-9650-0ca1cb32100f") : secret "dns-default-metrics-tls" not found
Sep 17 15:30:10.797166 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.797110    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plpr7\" (UniqueName: \"kubernetes.io/projected/352c01b0-7c02-4da9-a311-816815cd5e1e-kube-api-access-plpr7\") pod \"network-check-target-l2rxt\" (UID: \"352c01b0-7c02-4da9-a311-816815cd5e1e\") " pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:30:10.797361 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.797238    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84728120-5cde-4b79-944d-7554db40cf41-original-pull-secret\") pod \"global-pull-secret-syncer-r5jgb\" (UID: \"84728120-5cde-4b79-944d-7554db40cf41\") " pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:30:10.797428 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:10.797385    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:30:10.797482 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:10.797450    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84728120-5cde-4b79-944d-7554db40cf41-original-pull-secret podName:84728120-5cde-4b79-944d-7554db40cf41 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:42.797433042 +0000 UTC m=+65.177450777 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/84728120-5cde-4b79-944d-7554db40cf41-original-pull-secret") pod "global-pull-secret-syncer-r5jgb" (UID: "84728120-5cde-4b79-944d-7554db40cf41") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:30:10.797728 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:10.797700    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:30:10.797728 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:10.797734    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:30:10.797728 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:10.797749    2556 projected.go:196] Error preparing data for projected volume kube-api-access-plpr7 for pod openshift-network-diagnostics/network-check-target-l2rxt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:30:10.797944 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:10.797808    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/352c01b0-7c02-4da9-a311-816815cd5e1e-kube-api-access-plpr7 podName:352c01b0-7c02-4da9-a311-816815cd5e1e nodeName:}" failed. No retries permitted until 2026-09-17 15:30:42.797791059 +0000 UTC m=+65.177808798 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-plpr7" (UniqueName: "kubernetes.io/projected/352c01b0-7c02-4da9-a311-816815cd5e1e-kube-api-access-plpr7") pod "network-check-target-l2rxt" (UID: "352c01b0-7c02-4da9-a311-816815cd5e1e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:30:10.898032 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:10.897986    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eae675d6-96cb-4843-9deb-31182313d8ac-metrics-certs\") pod \"network-metrics-daemon-hn89v\" (UID: \"eae675d6-96cb-4843-9deb-31182313d8ac\") " pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:30:10.898236 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:10.898192    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:30:10.898310 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:10.898295    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eae675d6-96cb-4843-9deb-31182313d8ac-metrics-certs podName:eae675d6-96cb-4843-9deb-31182313d8ac nodeName:}" failed. No retries permitted until 2026-09-17 15:30:42.898273661 +0000 UTC m=+65.278291418 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eae675d6-96cb-4843-9deb-31182313d8ac-metrics-certs") pod "network-metrics-daemon-hn89v" (UID: "eae675d6-96cb-4843-9deb-31182313d8ac") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:30:11.096134 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:11.096043    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:11.129638 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:11.129602    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:30:11.129819 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:11.129708    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:30:11.129819 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:11.129748    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:30:11.132340 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:11.132258    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 15:30:11.132340 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:11.132281    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zrbtw\""
Sep 17 15:30:11.133016 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:11.132887    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 15:30:11.133016 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:11.132906    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 15:30:11.133016 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:11.132945    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 15:30:11.133016 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:11.132964    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pmg9m\""
Sep 17 15:30:11.197125 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:11.197092    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:11.297980 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:11.297933    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:11.362048 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:11.361964    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:11.703985 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:11.703959    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-registry-tls\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:11.704120 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:11.704049    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de6d8a3f-074e-4f18-9650-0ca1cb32100f-metrics-tls\") pod \"dns-default-k7kbh\" (UID: \"de6d8a3f-074e-4f18-9650-0ca1cb32100f\") " pod="openshift-dns/dns-default-k7kbh"
Sep 17 15:30:11.704178 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:11.704126    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:30:11.704178 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:11.704140    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:30:11.704178 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:11.704144    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69d6bcf58c-bq4rc: secret "image-registry-tls" not found
Sep 17 15:30:11.704298 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:11.704213    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de6d8a3f-074e-4f18-9650-0ca1cb32100f-metrics-tls podName:de6d8a3f-074e-4f18-9650-0ca1cb32100f nodeName:}" failed. No retries permitted until 2026-09-17 15:30:13.704197292 +0000 UTC m=+36.084215026 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de6d8a3f-074e-4f18-9650-0ca1cb32100f-metrics-tls") pod "dns-default-k7kbh" (UID: "de6d8a3f-074e-4f18-9650-0ca1cb32100f") : secret "dns-default-metrics-tls" not found
Sep 17 15:30:11.704298 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:11.704227    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-registry-tls podName:2581da35-8109-4df3-a553-c8992a31b8d1 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:13.704221737 +0000 UTC m=+36.084239471 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-registry-tls") pod "image-registry-69d6bcf58c-bq4rc" (UID: "2581da35-8109-4df3-a553-c8992a31b8d1") : secret "image-registry-tls" not found
Sep 17 15:30:11.940105 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:11.939875    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r975h"]
Sep 17 15:30:11.943144 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:30:11.943102    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac3c1f9c_dbfd_4097_b8c5_50d3fa2962c5.slice/crio-8603ab5a5daaa0e8283f41d3528697d09b6efda5cd98c859424658e6a79e305a WatchSource:0}: Error finding container 8603ab5a5daaa0e8283f41d3528697d09b6efda5cd98c859424658e6a79e305a: Status 404 returned error can't find the container with id 8603ab5a5daaa0e8283f41d3528697d09b6efda5cd98c859424658e6a79e305a
Sep 17 15:30:12.463572 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:12.463539    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r975h" event={"ID":"ac3c1f9c-dbfd-4097-b8c5-50d3fa2962c5","Type":"ContainerStarted","Data":"8603ab5a5daaa0e8283f41d3528697d09b6efda5cd98c859424658e6a79e305a"}
Sep 17 15:30:12.466921 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:12.466891    2556 generic.go:356] "Generic (PLEG): container finished" podID="2b086625-d2ab-4b2e-826b-8cf6a9534826" containerID="5345513f97a35446b6338a7d448cf988c483dec43ab0e4b3697101d41713928f" exitCode=0
Sep 17 15:30:12.467046 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:12.466958    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmrtd" event={"ID":"2b086625-d2ab-4b2e-826b-8cf6a9534826","Type":"ContainerDied","Data":"5345513f97a35446b6338a7d448cf988c483dec43ab0e4b3697101d41713928f"}
Sep 17 15:30:13.474001 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:13.473969    2556 generic.go:356] "Generic (PLEG): container finished" podID="2b086625-d2ab-4b2e-826b-8cf6a9534826" containerID="d6e8f24c309147701dfe994ca1aedf1a00d1b30be1fc217b15d3e4d13b11e865" exitCode=0
Sep 17 15:30:13.474734 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:13.474013    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmrtd" event={"ID":"2b086625-d2ab-4b2e-826b-8cf6a9534826","Type":"ContainerDied","Data":"d6e8f24c309147701dfe994ca1aedf1a00d1b30be1fc217b15d3e4d13b11e865"}
Sep 17 15:30:13.722228 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:13.722194    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-registry-tls\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:13.722413 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:13.722297    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de6d8a3f-074e-4f18-9650-0ca1cb32100f-metrics-tls\") pod \"dns-default-k7kbh\" (UID: \"de6d8a3f-074e-4f18-9650-0ca1cb32100f\") " pod="openshift-dns/dns-default-k7kbh"
Sep 17 15:30:13.722413 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:13.722379    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:30:13.722413 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:13.722393    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:30:13.722413 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:13.722400    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69d6bcf58c-bq4rc: secret "image-registry-tls" not found
Sep 17 15:30:13.722598 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:13.722454    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de6d8a3f-074e-4f18-9650-0ca1cb32100f-metrics-tls podName:de6d8a3f-074e-4f18-9650-0ca1cb32100f nodeName:}" failed. No retries permitted until 2026-09-17 15:30:17.722438035 +0000 UTC m=+40.102455768 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de6d8a3f-074e-4f18-9650-0ca1cb32100f-metrics-tls") pod "dns-default-k7kbh" (UID: "de6d8a3f-074e-4f18-9650-0ca1cb32100f") : secret "dns-default-metrics-tls" not found
Sep 17 15:30:13.722598 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:13.722476    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-registry-tls podName:2581da35-8109-4df3-a553-c8992a31b8d1 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:17.722465215 +0000 UTC m=+40.102482956 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-registry-tls") pod "image-registry-69d6bcf58c-bq4rc" (UID: "2581da35-8109-4df3-a553-c8992a31b8d1") : secret "image-registry-tls" not found
Sep 17 15:30:14.134006 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:14.133962    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:14.216449 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:14.216410    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:14.318617 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:14.318582    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:14.417707 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:14.417676    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:14.480316 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:14.480282    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmrtd" event={"ID":"2b086625-d2ab-4b2e-826b-8cf6a9534826","Type":"ContainerStarted","Data":"4b4dfff492c31c6c6fa15012304c1a604dde369d21a88bc604ee946544392867"}
Sep 17 15:30:14.504142 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:14.504091    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cmrtd" podStartSLOduration=4.26172713 podStartE2EDuration="36.504055093s" totalImagesPullingTime="32.242327963s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:29:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-17 15:30:14.502956904 +0000 UTC m=+36.882974661" watchObservedRunningTime="2026-09-17 15:30:14.504055093 +0000 UTC m=+36.884072874"
Sep 17 15:30:14.507964 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:14.507942    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:14.580177 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:14.580130    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:15.170309 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:15.170078    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:15.270875 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:15.270831    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:15.484648 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:15.484567    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r975h" event={"ID":"ac3c1f9c-dbfd-4097-b8c5-50d3fa2962c5","Type":"ContainerStarted","Data":"691fc1f0fce111c756915f6947ad6a4156e593453666272239384603f636ff78"}
Sep 17 15:30:15.550883 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:15.550849    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:15.612643 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:15.612612    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:17.753947 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:17.753900    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-registry-tls\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:17.754414 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:17.754088    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:30:17.754414 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:17.754115    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69d6bcf58c-bq4rc: secret "image-registry-tls" not found
Sep 17 15:30:17.754414 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:17.754128    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de6d8a3f-074e-4f18-9650-0ca1cb32100f-metrics-tls\") pod \"dns-default-k7kbh\" (UID: \"de6d8a3f-074e-4f18-9650-0ca1cb32100f\") " pod="openshift-dns/dns-default-k7kbh"
Sep 17 15:30:17.754414 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:17.754177    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-registry-tls podName:2581da35-8109-4df3-a553-c8992a31b8d1 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:25.754157478 +0000 UTC m=+48.134175214 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-registry-tls") pod "image-registry-69d6bcf58c-bq4rc" (UID: "2581da35-8109-4df3-a553-c8992a31b8d1") : secret "image-registry-tls" not found
Sep 17 15:30:17.754414 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:17.754228    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:30:17.754414 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:17.754264    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de6d8a3f-074e-4f18-9650-0ca1cb32100f-metrics-tls podName:de6d8a3f-074e-4f18-9650-0ca1cb32100f nodeName:}" failed. No retries permitted until 2026-09-17 15:30:25.754255548 +0000 UTC m=+48.134273286 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de6d8a3f-074e-4f18-9650-0ca1cb32100f-metrics-tls") pod "dns-default-k7kbh" (UID: "de6d8a3f-074e-4f18-9650-0ca1cb32100f") : secret "dns-default-metrics-tls" not found
Sep 17 15:30:17.887082 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:17.887032    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:17.943656 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:17.943626    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:19.954839 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:19.954777    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-r975h" podStartSLOduration=8.334858364 podStartE2EDuration="10.954758524s" totalImagesPullingTime="2.61990016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:15.506170748 +0000 UTC m=+37.886188503" watchObservedRunningTime="2026-09-17 15:30:19.954758524 +0000 UTC m=+42.334776329"
Sep 17 15:30:19.955276 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:19.954950    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-r8mp6"]
Sep 17 15:30:19.981445 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:19.981413    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-cnf4j"]
Sep 17 15:30:19.981616 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:19.981541    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-r8mp6"
Sep 17 15:30:19.983805 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:19.983759    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 15:30:19.983805 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:19.983764    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:30:19.984076 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:19.983882    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 15:30:19.984076 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:19.983883    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-hc7h8\""
Sep 17 15:30:19.984640 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:19.984626    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 15:30:19.998389 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:19.998367    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-r8mp6"]
Sep 17 15:30:19.998501 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:19.998403    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-cnf4j"]
Sep 17 15:30:19.998501 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:19.998418    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-cnf4j"
Sep 17 15:30:20.000825 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.000798    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 15:30:20.000825 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.000801    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-t2f54\""
Sep 17 15:30:20.000983 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.000801    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 15:30:20.000983 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.000804    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:30:20.057100 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.057052    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8pwxw"]
Sep 17 15:30:20.069952 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.069910    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cdc3f7b4-7834-4853-87dc-cd68e3cdad3a-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-r8mp6\" (UID: \"cdc3f7b4-7834-4853-87dc-cd68e3cdad3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-r8mp6"
Sep 17 15:30:20.070130 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.069973    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cdc3f7b4-7834-4853-87dc-cd68e3cdad3a-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-r8mp6\" (UID: \"cdc3f7b4-7834-4853-87dc-cd68e3cdad3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-r8mp6"
Sep 17 15:30:20.070130 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.070054    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q6km\" (UniqueName: \"kubernetes.io/projected/cdc3f7b4-7834-4853-87dc-cd68e3cdad3a-kube-api-access-5q6km\") pod \"kube-storage-version-migrator-operator-6c8f84f897-r8mp6\" (UID: \"cdc3f7b4-7834-4853-87dc-cd68e3cdad3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-r8mp6"
Sep 17 15:30:20.088789 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.088761    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-t58k8"]
Sep 17 15:30:20.088958 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.088907    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8pwxw"
Sep 17 15:30:20.091047 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.091025    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:30:20.091164 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.091119    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-c2q2p\""
Sep 17 15:30:20.097159 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.097141    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 15:30:20.103530 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.103500    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-nb2mr"]
Sep 17 15:30:20.103530 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.103524    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-t58k8"
Sep 17 15:30:20.106084 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.106041    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 15:30:20.106502 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.106479    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-nvj49\""
Sep 17 15:30:20.106861 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.106838    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 15:30:20.107027 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.106955    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 15:30:20.107412 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.107392    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 15:30:20.121764 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.121739    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-45zt7"]
Sep 17 15:30:20.121869 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.121846    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nb2mr"
Sep 17 15:30:20.123866 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.123847    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:30:20.124070 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.124046    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-clccn\""
Sep 17 15:30:20.124128 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.124112    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 15:30:20.124178 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.124119    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 15:30:20.124220 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.124191    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 15:30:20.137873 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.137844    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-45zt7"
Sep 17 15:30:20.140381 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.140360    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 15:30:20.140381 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.140360    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 15:30:20.140555 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.140366    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-b4kll\""
Sep 17 15:30:20.140679 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.140662    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 15:30:20.140717 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.140684    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:30:20.141900 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.141879    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-85cf78879b-z2qnj"]
Sep 17 15:30:20.145651 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.145630    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 15:30:20.155853 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.155835    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8pwxw"]
Sep 17 15:30:20.155957 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.155863    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-t58k8"]
Sep 17 15:30:20.155957 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.155878    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-45zt7"]
Sep 17 15:30:20.155957 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.155893    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-85cf78879b-z2qnj"]
Sep 17 15:30:20.155957 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.155907    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-nb2mr"]
Sep 17 15:30:20.155957 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.155953    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:20.158258 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.158238    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 15:30:20.158415 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.158392    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 15:30:20.158621 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.158604    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 15:30:20.158850 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.158836    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 15:30:20.158933 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.158869    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 15:30:20.158933 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.158892    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 15:30:20.159161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.159147    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wz88k\""
Sep 17 15:30:20.159433 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.159415    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-xqb4f"]
Sep 17 15:30:20.171055 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.171032    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cdc3f7b4-7834-4853-87dc-cd68e3cdad3a-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-r8mp6\" (UID: \"cdc3f7b4-7834-4853-87dc-cd68e3cdad3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-r8mp6"
Sep 17 15:30:20.171180 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.171085    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64zmg\" (UniqueName: \"kubernetes.io/projected/7fb3a087-736b-47bd-99ee-2acd40aa017e-kube-api-access-64zmg\") pod \"cluster-samples-operator-86499c89c4-cnf4j\" (UID: \"7fb3a087-736b-47bd-99ee-2acd40aa017e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-cnf4j"
Sep 17 15:30:20.171180 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.171108    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghzmt\" (UniqueName: \"kubernetes.io/projected/c9d7e7c9-d70d-4fd3-86df-ada691b1d830-kube-api-access-ghzmt\") pod \"volume-data-source-validator-86695b7cc7-8pwxw\" (UID: \"c9d7e7c9-d70d-4fd3-86df-ada691b1d830\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8pwxw"
Sep 17 15:30:20.171180 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.171126    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cdc3f7b4-7834-4853-87dc-cd68e3cdad3a-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-r8mp6\" (UID: \"cdc3f7b4-7834-4853-87dc-cd68e3cdad3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-r8mp6"
Sep 17 15:30:20.171180 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.171167    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q6km\" (UniqueName: \"kubernetes.io/projected/cdc3f7b4-7834-4853-87dc-cd68e3cdad3a-kube-api-access-5q6km\") pod \"kube-storage-version-migrator-operator-6c8f84f897-r8mp6\" (UID: \"cdc3f7b4-7834-4853-87dc-cd68e3cdad3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-r8mp6"
Sep 17 15:30:20.171334 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.171186    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fb3a087-736b-47bd-99ee-2acd40aa017e-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-cnf4j\" (UID: \"7fb3a087-736b-47bd-99ee-2acd40aa017e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-cnf4j"
Sep 17 15:30:20.171557 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.171541    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cdc3f7b4-7834-4853-87dc-cd68e3cdad3a-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-r8mp6\" (UID: \"cdc3f7b4-7834-4853-87dc-cd68e3cdad3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-r8mp6"
Sep 17 15:30:20.172052 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.172028    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-xqb4f"
Sep 17 15:30:20.174336 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.174316    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 15:30:20.174434 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.174410    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 15:30:20.174506 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.174455    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cdc3f7b4-7834-4853-87dc-cd68e3cdad3a-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-r8mp6\" (UID: \"cdc3f7b4-7834-4853-87dc-cd68e3cdad3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-r8mp6"
Sep 17 15:30:20.174708 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.174691    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 15:30:20.174780 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.174723    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-tv8qk\""
Sep 17 15:30:20.175390 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.175373    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 15:30:20.177161 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.177142    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-xqb4f"]
Sep 17 15:30:20.181227 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.181208    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 15:30:20.191939 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.191918    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q6km\" (UniqueName: \"kubernetes.io/projected/cdc3f7b4-7834-4853-87dc-cd68e3cdad3a-kube-api-access-5q6km\") pod \"kube-storage-version-migrator-operator-6c8f84f897-r8mp6\" (UID: \"cdc3f7b4-7834-4853-87dc-cd68e3cdad3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-r8mp6"
Sep 17 15:30:20.210629 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.210558    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:20.269943 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.269904    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:20.272138 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.272109    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dfe98000-7851-4180-b13f-bff7553a880b-config\") pod \"service-ca-operator-cf7c776b7-nb2mr\" (UID: \"dfe98000-7851-4180-b13f-bff7553a880b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nb2mr"
Sep 17 15:30:20.272284 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.272222    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b73fab2-e530-4b68-acf0-c71d1943cb7c-metrics-certs\") pod \"router-default-85cf78879b-z2qnj\" (UID: \"8b73fab2-e530-4b68-acf0-c71d1943cb7c\") " pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:20.272344 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.272293    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhqdj\" (UniqueName: \"kubernetes.io/projected/8b73fab2-e530-4b68-acf0-c71d1943cb7c-kube-api-access-bhqdj\") pod \"router-default-85cf78879b-z2qnj\" (UID: \"8b73fab2-e530-4b68-acf0-c71d1943cb7c\") " pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:20.272398 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.272341    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b73fab2-e530-4b68-acf0-c71d1943cb7c-service-ca-bundle\") pod \"router-default-85cf78879b-z2qnj\" (UID: \"8b73fab2-e530-4b68-acf0-c71d1943cb7c\") " pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:20.272445 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.272420    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2c069c9e-0856-46b0-ba10-67829c05ac94-serving-cert\") pod \"console-operator-696884fb9c-45zt7\" (UID: \"2c069c9e-0856-46b0-ba10-67829c05ac94\") " pod="openshift-console-operator/console-operator-696884fb9c-45zt7"
Sep 17 15:30:20.272492 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.272455    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2jsz\" (UniqueName: \"kubernetes.io/projected/95eb306e-4e4f-48ed-9b49-14471d4b4ddf-kube-api-access-x2jsz\") pod \"insights-operator-5844b856db-xqb4f\" (UID: \"95eb306e-4e4f-48ed-9b49-14471d4b4ddf\") " pod="openshift-insights/insights-operator-5844b856db-xqb4f"
Sep 17 15:30:20.272545 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.272529    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs7qx\" (UniqueName: \"kubernetes.io/projected/2c069c9e-0856-46b0-ba10-67829c05ac94-kube-api-access-qs7qx\") pod \"console-operator-696884fb9c-45zt7\" (UID: \"2c069c9e-0856-46b0-ba10-67829c05ac94\") " pod="openshift-console-operator/console-operator-696884fb9c-45zt7"
Sep 17 15:30:20.272595 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.272577    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdr86\" (UniqueName: \"kubernetes.io/projected/8107c705-1f01-4194-bcde-679259e3ea51-kube-api-access-wdr86\") pod \"cluster-monitoring-operator-6cd954b897-t58k8\" (UID: \"8107c705-1f01-4194-bcde-679259e3ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-t58k8"
Sep 17 15:30:20.272642 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.272608    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dfe98000-7851-4180-b13f-bff7553a880b-serving-cert\") pod \"service-ca-operator-cf7c776b7-nb2mr\" (UID: \"dfe98000-7851-4180-b13f-bff7553a880b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nb2mr"
Sep 17 15:30:20.272691 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.272659    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8107c705-1f01-4194-bcde-679259e3ea51-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-t58k8\" (UID: \"8107c705-1f01-4194-bcde-679259e3ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-t58k8"
Sep 17 15:30:20.272797 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.272692    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95eb306e-4e4f-48ed-9b49-14471d4b4ddf-trusted-ca-bundle\") pod \"insights-operator-5844b856db-xqb4f\" (UID: \"95eb306e-4e4f-48ed-9b49-14471d4b4ddf\") " pod="openshift-insights/insights-operator-5844b856db-xqb4f"
Sep 17 15:30:20.272797 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.272755    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/95eb306e-4e4f-48ed-9b49-14471d4b4ddf-serving-cert\") pod \"insights-operator-5844b856db-xqb4f\" (UID: \"95eb306e-4e4f-48ed-9b49-14471d4b4ddf\") " pod="openshift-insights/insights-operator-5844b856db-xqb4f"
Sep 17 15:30:20.272893 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.272802    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fb3a087-736b-47bd-99ee-2acd40aa017e-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-cnf4j\" (UID: \"7fb3a087-736b-47bd-99ee-2acd40aa017e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-cnf4j"
Sep 17 15:30:20.272893 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.272833    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c069c9e-0856-46b0-ba10-67829c05ac94-trusted-ca\") pod \"console-operator-696884fb9c-45zt7\" (UID: \"2c069c9e-0856-46b0-ba10-67829c05ac94\") " pod="openshift-console-operator/console-operator-696884fb9c-45zt7"
Sep 17 15:30:20.272979 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.272896    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8b73fab2-e530-4b68-acf0-c71d1943cb7c-default-certificate\") pod \"router-default-85cf78879b-z2qnj\" (UID: \"8b73fab2-e530-4b68-acf0-c71d1943cb7c\") " pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:20.272979 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.272945    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8b73fab2-e530-4b68-acf0-c71d1943cb7c-stats-auth\") pod \"router-default-85cf78879b-z2qnj\" (UID: \"8b73fab2-e530-4b68-acf0-c71d1943cb7c\") " pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:20.272979 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.272969    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/95eb306e-4e4f-48ed-9b49-14471d4b4ddf-snapshots\") pod \"insights-operator-5844b856db-xqb4f\" (UID: \"95eb306e-4e4f-48ed-9b49-14471d4b4ddf\") " pod="openshift-insights/insights-operator-5844b856db-xqb4f"
Sep 17 15:30:20.273127 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.273030    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghzmt\" (UniqueName: \"kubernetes.io/projected/c9d7e7c9-d70d-4fd3-86df-ada691b1d830-kube-api-access-ghzmt\") pod \"volume-data-source-validator-86695b7cc7-8pwxw\" (UID: \"c9d7e7c9-d70d-4fd3-86df-ada691b1d830\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8pwxw"
Sep 17 15:30:20.273127 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.273088    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8107c705-1f01-4194-bcde-679259e3ea51-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-t58k8\" (UID: \"8107c705-1f01-4194-bcde-679259e3ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-t58k8"
Sep 17 15:30:20.273127 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.273121    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64zmg\" (UniqueName: \"kubernetes.io/projected/7fb3a087-736b-47bd-99ee-2acd40aa017e-kube-api-access-64zmg\") pod \"cluster-samples-operator-86499c89c4-cnf4j\" (UID: \"7fb3a087-736b-47bd-99ee-2acd40aa017e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-cnf4j"
Sep 17 15:30:20.273269 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.273164    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69jbb\" (UniqueName: \"kubernetes.io/projected/dfe98000-7851-4180-b13f-bff7553a880b-kube-api-access-69jbb\") pod \"service-ca-operator-cf7c776b7-nb2mr\" (UID: \"dfe98000-7851-4180-b13f-bff7553a880b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nb2mr"
Sep 17 15:30:20.273269 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.273193    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2c069c9e-0856-46b0-ba10-67829c05ac94-config\") pod \"console-operator-696884fb9c-45zt7\" (UID: \"2c069c9e-0856-46b0-ba10-67829c05ac94\") " pod="openshift-console-operator/console-operator-696884fb9c-45zt7"
Sep 17 15:30:20.273269 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.273233    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95eb306e-4e4f-48ed-9b49-14471d4b4ddf-service-ca-bundle\") pod \"insights-operator-5844b856db-xqb4f\" (UID: \"95eb306e-4e4f-48ed-9b49-14471d4b4ddf\") " pod="openshift-insights/insights-operator-5844b856db-xqb4f"
Sep 17 15:30:20.273269 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.273260    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95eb306e-4e4f-48ed-9b49-14471d4b4ddf-tmp\") pod \"insights-operator-5844b856db-xqb4f\" (UID: \"95eb306e-4e4f-48ed-9b49-14471d4b4ddf\") " pod="openshift-insights/insights-operator-5844b856db-xqb4f"
Sep 17 15:30:20.273663 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:20.273645    2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 15:30:20.273907 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:20.273896    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fb3a087-736b-47bd-99ee-2acd40aa017e-samples-operator-tls podName:7fb3a087-736b-47bd-99ee-2acd40aa017e nodeName:}" failed. No retries permitted until 2026-09-17 15:30:20.773854616 +0000 UTC m=+43.153872351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7fb3a087-736b-47bd-99ee-2acd40aa017e-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-cnf4j" (UID: "7fb3a087-736b-47bd-99ee-2acd40aa017e") : secret "samples-operator-tls" not found
Sep 17 15:30:20.282657 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.282636    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64zmg\" (UniqueName: \"kubernetes.io/projected/7fb3a087-736b-47bd-99ee-2acd40aa017e-kube-api-access-64zmg\") pod \"cluster-samples-operator-86499c89c4-cnf4j\" (UID: \"7fb3a087-736b-47bd-99ee-2acd40aa017e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-cnf4j"
Sep 17 15:30:20.282893 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.282870    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghzmt\" (UniqueName: \"kubernetes.io/projected/c9d7e7c9-d70d-4fd3-86df-ada691b1d830-kube-api-access-ghzmt\") pod \"volume-data-source-validator-86695b7cc7-8pwxw\" (UID: \"c9d7e7c9-d70d-4fd3-86df-ada691b1d830\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8pwxw"
Sep 17 15:30:20.291529 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.291507    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-r8mp6"
Sep 17 15:30:20.374645 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.374611    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b73fab2-e530-4b68-acf0-c71d1943cb7c-service-ca-bundle\") pod \"router-default-85cf78879b-z2qnj\" (UID: \"8b73fab2-e530-4b68-acf0-c71d1943cb7c\") " pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:20.374779 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.374665    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2c069c9e-0856-46b0-ba10-67829c05ac94-serving-cert\") pod \"console-operator-696884fb9c-45zt7\" (UID: \"2c069c9e-0856-46b0-ba10-67829c05ac94\") " pod="openshift-console-operator/console-operator-696884fb9c-45zt7"
Sep 17 15:30:20.374843 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:20.374806    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8b73fab2-e530-4b68-acf0-c71d1943cb7c-service-ca-bundle podName:8b73fab2-e530-4b68-acf0-c71d1943cb7c nodeName:}" failed. No retries permitted until 2026-09-17 15:30:20.874786603 +0000 UTC m=+43.254804361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8b73fab2-e530-4b68-acf0-c71d1943cb7c-service-ca-bundle") pod "router-default-85cf78879b-z2qnj" (UID: "8b73fab2-e530-4b68-acf0-c71d1943cb7c") : configmap references non-existent config key: service-ca.crt
Sep 17 15:30:20.374843 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.374835    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2jsz\" (UniqueName: \"kubernetes.io/projected/95eb306e-4e4f-48ed-9b49-14471d4b4ddf-kube-api-access-x2jsz\") pod \"insights-operator-5844b856db-xqb4f\" (UID: \"95eb306e-4e4f-48ed-9b49-14471d4b4ddf\") " pod="openshift-insights/insights-operator-5844b856db-xqb4f"
Sep 17 15:30:20.374949 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.374881    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs7qx\" (UniqueName: \"kubernetes.io/projected/2c069c9e-0856-46b0-ba10-67829c05ac94-kube-api-access-qs7qx\") pod \"console-operator-696884fb9c-45zt7\" (UID: \"2c069c9e-0856-46b0-ba10-67829c05ac94\") " pod="openshift-console-operator/console-operator-696884fb9c-45zt7"
Sep 17 15:30:20.374949 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.374902    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdr86\" (UniqueName: \"kubernetes.io/projected/8107c705-1f01-4194-bcde-679259e3ea51-kube-api-access-wdr86\") pod \"cluster-monitoring-operator-6cd954b897-t58k8\" (UID: \"8107c705-1f01-4194-bcde-679259e3ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-t58k8"
Sep 17 15:30:20.374949 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.374920    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dfe98000-7851-4180-b13f-bff7553a880b-serving-cert\") pod \"service-ca-operator-cf7c776b7-nb2mr\" (UID: \"dfe98000-7851-4180-b13f-bff7553a880b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nb2mr"
Sep 17 15:30:20.374949 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.374945    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8107c705-1f01-4194-bcde-679259e3ea51-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-t58k8\" (UID: \"8107c705-1f01-4194-bcde-679259e3ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-t58k8"
Sep 17 15:30:20.375150 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.374966    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95eb306e-4e4f-48ed-9b49-14471d4b4ddf-trusted-ca-bundle\") pod \"insights-operator-5844b856db-xqb4f\" (UID: \"95eb306e-4e4f-48ed-9b49-14471d4b4ddf\") " pod="openshift-insights/insights-operator-5844b856db-xqb4f"
Sep 17 15:30:20.375150 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.374989    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/95eb306e-4e4f-48ed-9b49-14471d4b4ddf-serving-cert\") pod \"insights-operator-5844b856db-xqb4f\" (UID: \"95eb306e-4e4f-48ed-9b49-14471d4b4ddf\") " pod="openshift-insights/insights-operator-5844b856db-xqb4f"
Sep 17 15:30:20.375150 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.375013    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c069c9e-0856-46b0-ba10-67829c05ac94-trusted-ca\") pod \"console-operator-696884fb9c-45zt7\" (UID: \"2c069c9e-0856-46b0-ba10-67829c05ac94\") " pod="openshift-console-operator/console-operator-696884fb9c-45zt7"
Sep 17 15:30:20.375150 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.375037    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8b73fab2-e530-4b68-acf0-c71d1943cb7c-default-certificate\") pod \"router-default-85cf78879b-z2qnj\" (UID: \"8b73fab2-e530-4b68-acf0-c71d1943cb7c\") " pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:20.375150 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.375070    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8b73fab2-e530-4b68-acf0-c71d1943cb7c-stats-auth\") pod \"router-default-85cf78879b-z2qnj\" (UID: \"8b73fab2-e530-4b68-acf0-c71d1943cb7c\") " pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:20.375150 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.375097    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/95eb306e-4e4f-48ed-9b49-14471d4b4ddf-snapshots\") pod \"insights-operator-5844b856db-xqb4f\" (UID: \"95eb306e-4e4f-48ed-9b49-14471d4b4ddf\") " pod="openshift-insights/insights-operator-5844b856db-xqb4f"
Sep 17 15:30:20.375150 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.375151    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8107c705-1f01-4194-bcde-679259e3ea51-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-t58k8\" (UID: \"8107c705-1f01-4194-bcde-679259e3ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-t58k8"
Sep 17 15:30:20.375492 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.375183    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69jbb\" (UniqueName: \"kubernetes.io/projected/dfe98000-7851-4180-b13f-bff7553a880b-kube-api-access-69jbb\") pod \"service-ca-operator-cf7c776b7-nb2mr\" (UID: \"dfe98000-7851-4180-b13f-bff7553a880b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nb2mr"
Sep 17 15:30:20.375492 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:20.375452    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 15:30:20.375592 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:20.375515    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8107c705-1f01-4194-bcde-679259e3ea51-cluster-monitoring-operator-tls podName:8107c705-1f01-4194-bcde-679259e3ea51 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:20.875493309 +0000 UTC m=+43.255511043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8107c705-1f01-4194-bcde-679259e3ea51-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-t58k8" (UID: "8107c705-1f01-4194-bcde-679259e3ea51") : secret "cluster-monitoring-operator-tls" not found
Sep 17 15:30:20.375873 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.375806    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/95eb306e-4e4f-48ed-9b49-14471d4b4ddf-snapshots\") pod \"insights-operator-5844b856db-xqb4f\" (UID: \"95eb306e-4e4f-48ed-9b49-14471d4b4ddf\") " pod="openshift-insights/insights-operator-5844b856db-xqb4f"
Sep 17 15:30:20.375873 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.375806    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8107c705-1f01-4194-bcde-679259e3ea51-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-t58k8\" (UID: \"8107c705-1f01-4194-bcde-679259e3ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-t58k8"
Sep 17 15:30:20.376611 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.376106    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2c069c9e-0856-46b0-ba10-67829c05ac94-config\") pod \"console-operator-696884fb9c-45zt7\" (UID: \"2c069c9e-0856-46b0-ba10-67829c05ac94\") " pod="openshift-console-operator/console-operator-696884fb9c-45zt7"
Sep 17 15:30:20.376611 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.376152    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95eb306e-4e4f-48ed-9b49-14471d4b4ddf-service-ca-bundle\") pod \"insights-operator-5844b856db-xqb4f\" (UID: \"95eb306e-4e4f-48ed-9b49-14471d4b4ddf\") " pod="openshift-insights/insights-operator-5844b856db-xqb4f"
Sep 17 15:30:20.376611 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.376183    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95eb306e-4e4f-48ed-9b49-14471d4b4ddf-tmp\") pod \"insights-operator-5844b856db-xqb4f\" (UID: \"95eb306e-4e4f-48ed-9b49-14471d4b4ddf\") " pod="openshift-insights/insights-operator-5844b856db-xqb4f"
Sep 17 15:30:20.376611 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.376222    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dfe98000-7851-4180-b13f-bff7553a880b-config\") pod \"service-ca-operator-cf7c776b7-nb2mr\" (UID: \"dfe98000-7851-4180-b13f-bff7553a880b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nb2mr"
Sep 17 15:30:20.376611 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.376252    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b73fab2-e530-4b68-acf0-c71d1943cb7c-metrics-certs\") pod \"router-default-85cf78879b-z2qnj\" (UID: \"8b73fab2-e530-4b68-acf0-c71d1943cb7c\") " pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:20.376611 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.376266    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c069c9e-0856-46b0-ba10-67829c05ac94-trusted-ca\") pod \"console-operator-696884fb9c-45zt7\" (UID: \"2c069c9e-0856-46b0-ba10-67829c05ac94\") " pod="openshift-console-operator/console-operator-696884fb9c-45zt7"
Sep 17 15:30:20.376611 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.376278    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhqdj\" (UniqueName: \"kubernetes.io/projected/8b73fab2-e530-4b68-acf0-c71d1943cb7c-kube-api-access-bhqdj\") pod \"router-default-85cf78879b-z2qnj\" (UID: \"8b73fab2-e530-4b68-acf0-c71d1943cb7c\") " pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:20.376611 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.376363    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95eb306e-4e4f-48ed-9b49-14471d4b4ddf-trusted-ca-bundle\") pod \"insights-operator-5844b856db-xqb4f\" (UID: \"95eb306e-4e4f-48ed-9b49-14471d4b4ddf\") " pod="openshift-insights/insights-operator-5844b856db-xqb4f"
Sep 17 15:30:20.376611 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:20.376537    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 15:30:20.376611 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:20.376584    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b73fab2-e530-4b68-acf0-c71d1943cb7c-metrics-certs podName:8b73fab2-e530-4b68-acf0-c71d1943cb7c nodeName:}" failed. No retries permitted until 2026-09-17 15:30:20.876567999 +0000 UTC m=+43.256585733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b73fab2-e530-4b68-acf0-c71d1943cb7c-metrics-certs") pod "router-default-85cf78879b-z2qnj" (UID: "8b73fab2-e530-4b68-acf0-c71d1943cb7c") : secret "router-metrics-certs-default" not found
Sep 17 15:30:20.377213 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.376637    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95eb306e-4e4f-48ed-9b49-14471d4b4ddf-tmp\") pod \"insights-operator-5844b856db-xqb4f\" (UID: \"95eb306e-4e4f-48ed-9b49-14471d4b4ddf\") " pod="openshift-insights/insights-operator-5844b856db-xqb4f"
Sep 17 15:30:20.377213 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.376670    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95eb306e-4e4f-48ed-9b49-14471d4b4ddf-service-ca-bundle\") pod \"insights-operator-5844b856db-xqb4f\" (UID: \"95eb306e-4e4f-48ed-9b49-14471d4b4ddf\") " pod="openshift-insights/insights-operator-5844b856db-xqb4f"
Sep 17 15:30:20.377213 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.376862    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dfe98000-7851-4180-b13f-bff7553a880b-config\") pod \"service-ca-operator-cf7c776b7-nb2mr\" (UID: \"dfe98000-7851-4180-b13f-bff7553a880b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nb2mr"
Sep 17 15:30:20.377213 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.377018    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2c069c9e-0856-46b0-ba10-67829c05ac94-config\") pod \"console-operator-696884fb9c-45zt7\" (UID: \"2c069c9e-0856-46b0-ba10-67829c05ac94\") " pod="openshift-console-operator/console-operator-696884fb9c-45zt7"
Sep 17 15:30:20.378030 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.378008    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8b73fab2-e530-4b68-acf0-c71d1943cb7c-default-certificate\") pod \"router-default-85cf78879b-z2qnj\" (UID: \"8b73fab2-e530-4b68-acf0-c71d1943cb7c\") " pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:20.378335 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.378284    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/95eb306e-4e4f-48ed-9b49-14471d4b4ddf-serving-cert\") pod \"insights-operator-5844b856db-xqb4f\" (UID: \"95eb306e-4e4f-48ed-9b49-14471d4b4ddf\") " pod="openshift-insights/insights-operator-5844b856db-xqb4f"
Sep 17 15:30:20.378426 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.378350    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8b73fab2-e530-4b68-acf0-c71d1943cb7c-stats-auth\") pod \"router-default-85cf78879b-z2qnj\" (UID: \"8b73fab2-e530-4b68-acf0-c71d1943cb7c\") " pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:20.378559 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.378540    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2c069c9e-0856-46b0-ba10-67829c05ac94-serving-cert\") pod \"console-operator-696884fb9c-45zt7\" (UID: \"2c069c9e-0856-46b0-ba10-67829c05ac94\") " pod="openshift-console-operator/console-operator-696884fb9c-45zt7"
Sep 17 15:30:20.378728 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.378712    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dfe98000-7851-4180-b13f-bff7553a880b-serving-cert\") pod \"service-ca-operator-cf7c776b7-nb2mr\" (UID: \"dfe98000-7851-4180-b13f-bff7553a880b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nb2mr"
Sep 17 15:30:20.385319 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.385286    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69jbb\" (UniqueName: \"kubernetes.io/projected/dfe98000-7851-4180-b13f-bff7553a880b-kube-api-access-69jbb\") pod \"service-ca-operator-cf7c776b7-nb2mr\" (UID: \"dfe98000-7851-4180-b13f-bff7553a880b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nb2mr"
Sep 17 15:30:20.385853 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.385833    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2jsz\" (UniqueName: \"kubernetes.io/projected/95eb306e-4e4f-48ed-9b49-14471d4b4ddf-kube-api-access-x2jsz\") pod \"insights-operator-5844b856db-xqb4f\" (UID: \"95eb306e-4e4f-48ed-9b49-14471d4b4ddf\") " pod="openshift-insights/insights-operator-5844b856db-xqb4f"
Sep 17 15:30:20.385936 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.385876    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdr86\" (UniqueName: \"kubernetes.io/projected/8107c705-1f01-4194-bcde-679259e3ea51-kube-api-access-wdr86\") pod \"cluster-monitoring-operator-6cd954b897-t58k8\" (UID: \"8107c705-1f01-4194-bcde-679259e3ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-t58k8"
Sep 17 15:30:20.386004 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.385979    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs7qx\" (UniqueName: \"kubernetes.io/projected/2c069c9e-0856-46b0-ba10-67829c05ac94-kube-api-access-qs7qx\") pod \"console-operator-696884fb9c-45zt7\" (UID: \"2c069c9e-0856-46b0-ba10-67829c05ac94\") " pod="openshift-console-operator/console-operator-696884fb9c-45zt7"
Sep 17 15:30:20.386700 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.386681    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhqdj\" (UniqueName: \"kubernetes.io/projected/8b73fab2-e530-4b68-acf0-c71d1943cb7c-kube-api-access-bhqdj\") pod \"router-default-85cf78879b-z2qnj\" (UID: \"8b73fab2-e530-4b68-acf0-c71d1943cb7c\") " pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:20.398821 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.398803    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8pwxw"
Sep 17 15:30:20.430163 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.430133    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nb2mr"
Sep 17 15:30:20.447948 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.447919    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-45zt7"
Sep 17 15:30:20.487253 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.487212    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-xqb4f"
Sep 17 15:30:20.578507 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.578329    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-r8mp6"]
Sep 17 15:30:20.584392 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:30:20.584131    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc3f7b4_7834_4853_87dc_cd68e3cdad3a.slice/crio-8df430cac88ffec9c7ab2d5810b15fdc91f40b49030968ee36c4b6528aed86c4 WatchSource:0}: Error finding container 8df430cac88ffec9c7ab2d5810b15fdc91f40b49030968ee36c4b6528aed86c4: Status 404 returned error can't find the container with id 8df430cac88ffec9c7ab2d5810b15fdc91f40b49030968ee36c4b6528aed86c4
Sep 17 15:30:20.706322 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.706291    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8pwxw"]
Sep 17 15:30:20.709441 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:30:20.709417    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d7e7c9_d70d_4fd3_86df_ada691b1d830.slice/crio-122e4bc188552fa55a14b20a7f33b8a039becd79650d0787d2c1b598beca8ab9 WatchSource:0}: Error finding container 122e4bc188552fa55a14b20a7f33b8a039becd79650d0787d2c1b598beca8ab9: Status 404 returned error can't find the container with id 122e4bc188552fa55a14b20a7f33b8a039becd79650d0787d2c1b598beca8ab9
Sep 17 15:30:20.733032 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.732955    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-nb2mr"]
Sep 17 15:30:20.735465 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:30:20.735430    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe98000_7851_4180_b13f_bff7553a880b.slice/crio-d49c3793f8446319c44b60225ddfe5682d1a8e80ae843d21042f13e5e7dfec34 WatchSource:0}: Error finding container d49c3793f8446319c44b60225ddfe5682d1a8e80ae843d21042f13e5e7dfec34: Status 404 returned error can't find the container with id d49c3793f8446319c44b60225ddfe5682d1a8e80ae843d21042f13e5e7dfec34
Sep 17 15:30:20.774408 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.774377    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-45zt7"]
Sep 17 15:30:20.777416 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:30:20.777384    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c069c9e_0856_46b0_ba10_67829c05ac94.slice/crio-e646612a693f82daab091d043f50bc866e084d3a0afd4a6744eeec57a364ae11 WatchSource:0}: Error finding container e646612a693f82daab091d043f50bc866e084d3a0afd4a6744eeec57a364ae11: Status 404 returned error can't find the container with id e646612a693f82daab091d043f50bc866e084d3a0afd4a6744eeec57a364ae11
Sep 17 15:30:20.779591 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.779565    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fb3a087-736b-47bd-99ee-2acd40aa017e-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-cnf4j\" (UID: \"7fb3a087-736b-47bd-99ee-2acd40aa017e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-cnf4j"
Sep 17 15:30:20.779741 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:20.779725    2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 15:30:20.779800 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:20.779789    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fb3a087-736b-47bd-99ee-2acd40aa017e-samples-operator-tls podName:7fb3a087-736b-47bd-99ee-2acd40aa017e nodeName:}" failed. No retries permitted until 2026-09-17 15:30:21.779771626 +0000 UTC m=+44.159789380 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7fb3a087-736b-47bd-99ee-2acd40aa017e-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-cnf4j" (UID: "7fb3a087-736b-47bd-99ee-2acd40aa017e") : secret "samples-operator-tls" not found
Sep 17 15:30:20.805593 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.805565    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-xqb4f"]
Sep 17 15:30:20.808023 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:30:20.807997    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95eb306e_4e4f_48ed_9b49_14471d4b4ddf.slice/crio-edf28f06d6fe96cc42cd236438bcfe763e6747e208114497c02ff6f5dcfa474b WatchSource:0}: Error finding container edf28f06d6fe96cc42cd236438bcfe763e6747e208114497c02ff6f5dcfa474b: Status 404 returned error can't find the container with id edf28f06d6fe96cc42cd236438bcfe763e6747e208114497c02ff6f5dcfa474b
Sep 17 15:30:20.880245 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.880204    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8107c705-1f01-4194-bcde-679259e3ea51-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-t58k8\" (UID: \"8107c705-1f01-4194-bcde-679259e3ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-t58k8"
Sep 17 15:30:20.880445 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.880261    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b73fab2-e530-4b68-acf0-c71d1943cb7c-metrics-certs\") pod \"router-default-85cf78879b-z2qnj\" (UID: \"8b73fab2-e530-4b68-acf0-c71d1943cb7c\") " pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:20.880445 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:20.880363    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 15:30:20.880445 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:20.880370    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 15:30:20.880445 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:20.880382    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b73fab2-e530-4b68-acf0-c71d1943cb7c-service-ca-bundle\") pod \"router-default-85cf78879b-z2qnj\" (UID: \"8b73fab2-e530-4b68-acf0-c71d1943cb7c\") " pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:20.880445 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:20.880424    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b73fab2-e530-4b68-acf0-c71d1943cb7c-metrics-certs podName:8b73fab2-e530-4b68-acf0-c71d1943cb7c nodeName:}" failed. No retries permitted until 2026-09-17 15:30:21.880405521 +0000 UTC m=+44.260423255 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b73fab2-e530-4b68-acf0-c71d1943cb7c-metrics-certs") pod "router-default-85cf78879b-z2qnj" (UID: "8b73fab2-e530-4b68-acf0-c71d1943cb7c") : secret "router-metrics-certs-default" not found
Sep 17 15:30:20.880644 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:20.880473    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8b73fab2-e530-4b68-acf0-c71d1943cb7c-service-ca-bundle podName:8b73fab2-e530-4b68-acf0-c71d1943cb7c nodeName:}" failed. No retries permitted until 2026-09-17 15:30:21.880462075 +0000 UTC m=+44.260479808 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8b73fab2-e530-4b68-acf0-c71d1943cb7c-service-ca-bundle") pod "router-default-85cf78879b-z2qnj" (UID: "8b73fab2-e530-4b68-acf0-c71d1943cb7c") : configmap references non-existent config key: service-ca.crt
Sep 17 15:30:20.880644 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:20.880552    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8107c705-1f01-4194-bcde-679259e3ea51-cluster-monitoring-operator-tls podName:8107c705-1f01-4194-bcde-679259e3ea51 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:21.880545101 +0000 UTC m=+44.260562834 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8107c705-1f01-4194-bcde-679259e3ea51-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-t58k8" (UID: "8107c705-1f01-4194-bcde-679259e3ea51") : secret "cluster-monitoring-operator-tls" not found
Sep 17 15:30:21.502407 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:21.502367    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-xqb4f" event={"ID":"95eb306e-4e4f-48ed-9b49-14471d4b4ddf","Type":"ContainerStarted","Data":"edf28f06d6fe96cc42cd236438bcfe763e6747e208114497c02ff6f5dcfa474b"}
Sep 17 15:30:21.504394 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:21.504360    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nb2mr" event={"ID":"dfe98000-7851-4180-b13f-bff7553a880b","Type":"ContainerStarted","Data":"d49c3793f8446319c44b60225ddfe5682d1a8e80ae843d21042f13e5e7dfec34"}
Sep 17 15:30:21.506314 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:21.506206    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8pwxw" event={"ID":"c9d7e7c9-d70d-4fd3-86df-ada691b1d830","Type":"ContainerStarted","Data":"122e4bc188552fa55a14b20a7f33b8a039becd79650d0787d2c1b598beca8ab9"}
Sep 17 15:30:21.508256 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:21.508186    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-r8mp6" event={"ID":"cdc3f7b4-7834-4853-87dc-cd68e3cdad3a","Type":"ContainerStarted","Data":"8df430cac88ffec9c7ab2d5810b15fdc91f40b49030968ee36c4b6528aed86c4"}
Sep 17 15:30:21.510847 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:21.510821    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-45zt7" event={"ID":"2c069c9e-0856-46b0-ba10-67829c05ac94","Type":"ContainerStarted","Data":"e646612a693f82daab091d043f50bc866e084d3a0afd4a6744eeec57a364ae11"}
Sep 17 15:30:21.788920 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:21.788830    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fb3a087-736b-47bd-99ee-2acd40aa017e-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-cnf4j\" (UID: \"7fb3a087-736b-47bd-99ee-2acd40aa017e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-cnf4j"
Sep 17 15:30:21.789230 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:21.789029    2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 15:30:21.789230 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:21.789110    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fb3a087-736b-47bd-99ee-2acd40aa017e-samples-operator-tls podName:7fb3a087-736b-47bd-99ee-2acd40aa017e nodeName:}" failed. No retries permitted until 2026-09-17 15:30:23.789090317 +0000 UTC m=+46.169108068 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7fb3a087-736b-47bd-99ee-2acd40aa017e-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-cnf4j" (UID: "7fb3a087-736b-47bd-99ee-2acd40aa017e") : secret "samples-operator-tls" not found
Sep 17 15:30:21.890186 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:21.889916    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8107c705-1f01-4194-bcde-679259e3ea51-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-t58k8\" (UID: \"8107c705-1f01-4194-bcde-679259e3ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-t58k8"
Sep 17 15:30:21.890186 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:21.889987    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b73fab2-e530-4b68-acf0-c71d1943cb7c-metrics-certs\") pod \"router-default-85cf78879b-z2qnj\" (UID: \"8b73fab2-e530-4b68-acf0-c71d1943cb7c\") " pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:21.890186 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:21.890014    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b73fab2-e530-4b68-acf0-c71d1943cb7c-service-ca-bundle\") pod \"router-default-85cf78879b-z2qnj\" (UID: \"8b73fab2-e530-4b68-acf0-c71d1943cb7c\") " pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:21.890186 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:21.890102    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 15:30:21.890186 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:21.890153    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 15:30:21.890186 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:21.890187    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8107c705-1f01-4194-bcde-679259e3ea51-cluster-monitoring-operator-tls podName:8107c705-1f01-4194-bcde-679259e3ea51 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:23.890167138 +0000 UTC m=+46.270184877 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8107c705-1f01-4194-bcde-679259e3ea51-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-t58k8" (UID: "8107c705-1f01-4194-bcde-679259e3ea51") : secret "cluster-monitoring-operator-tls" not found
Sep 17 15:30:21.890631 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:21.890207    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8b73fab2-e530-4b68-acf0-c71d1943cb7c-service-ca-bundle podName:8b73fab2-e530-4b68-acf0-c71d1943cb7c nodeName:}" failed. No retries permitted until 2026-09-17 15:30:23.890197115 +0000 UTC m=+46.270214853 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8b73fab2-e530-4b68-acf0-c71d1943cb7c-service-ca-bundle") pod "router-default-85cf78879b-z2qnj" (UID: "8b73fab2-e530-4b68-acf0-c71d1943cb7c") : configmap references non-existent config key: service-ca.crt
Sep 17 15:30:21.890631 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:21.890223    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b73fab2-e530-4b68-acf0-c71d1943cb7c-metrics-certs podName:8b73fab2-e530-4b68-acf0-c71d1943cb7c nodeName:}" failed. No retries permitted until 2026-09-17 15:30:23.890214566 +0000 UTC m=+46.270232308 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b73fab2-e530-4b68-acf0-c71d1943cb7c-metrics-certs") pod "router-default-85cf78879b-z2qnj" (UID: "8b73fab2-e530-4b68-acf0-c71d1943cb7c") : secret "router-metrics-certs-default" not found
Sep 17 15:30:23.149186 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:23.149141    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-qjvgh"]
Sep 17 15:30:23.162439 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:23.162406    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-qjvgh"]
Sep 17 15:30:23.162635 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:23.162539    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-qjvgh"
Sep 17 15:30:23.165031 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:23.165004    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-xv9hq\""
Sep 17 15:30:23.301935 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:23.301893    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n659b\" (UniqueName: \"kubernetes.io/projected/fa176539-bbe8-4970-8a21-269b3c2b52e9-kube-api-access-n659b\") pod \"network-check-source-6f8b768455-qjvgh\" (UID: \"fa176539-bbe8-4970-8a21-269b3c2b52e9\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-qjvgh"
Sep 17 15:30:23.403553 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:23.403466    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n659b\" (UniqueName: \"kubernetes.io/projected/fa176539-bbe8-4970-8a21-269b3c2b52e9-kube-api-access-n659b\") pod \"network-check-source-6f8b768455-qjvgh\" (UID: \"fa176539-bbe8-4970-8a21-269b3c2b52e9\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-qjvgh"
Sep 17 15:30:23.411408 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:23.411382    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n659b\" (UniqueName: \"kubernetes.io/projected/fa176539-bbe8-4970-8a21-269b3c2b52e9-kube-api-access-n659b\") pod \"network-check-source-6f8b768455-qjvgh\" (UID: \"fa176539-bbe8-4970-8a21-269b3c2b52e9\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-qjvgh"
Sep 17 15:30:23.473029 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:23.472986    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-qjvgh"
Sep 17 15:30:23.808310 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:23.808217    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fb3a087-736b-47bd-99ee-2acd40aa017e-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-cnf4j\" (UID: \"7fb3a087-736b-47bd-99ee-2acd40aa017e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-cnf4j"
Sep 17 15:30:23.808460 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:23.808370    2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 15:30:23.808460 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:23.808451    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fb3a087-736b-47bd-99ee-2acd40aa017e-samples-operator-tls podName:7fb3a087-736b-47bd-99ee-2acd40aa017e nodeName:}" failed. No retries permitted until 2026-09-17 15:30:27.808429533 +0000 UTC m=+50.188447270 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7fb3a087-736b-47bd-99ee-2acd40aa017e-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-cnf4j" (UID: "7fb3a087-736b-47bd-99ee-2acd40aa017e") : secret "samples-operator-tls" not found
Sep 17 15:30:23.909188 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:23.909144    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8107c705-1f01-4194-bcde-679259e3ea51-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-t58k8\" (UID: \"8107c705-1f01-4194-bcde-679259e3ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-t58k8"
Sep 17 15:30:23.909379 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:23.909217    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b73fab2-e530-4b68-acf0-c71d1943cb7c-metrics-certs\") pod \"router-default-85cf78879b-z2qnj\" (UID: \"8b73fab2-e530-4b68-acf0-c71d1943cb7c\") " pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:23.909379 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:23.909245    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b73fab2-e530-4b68-acf0-c71d1943cb7c-service-ca-bundle\") pod \"router-default-85cf78879b-z2qnj\" (UID: \"8b73fab2-e530-4b68-acf0-c71d1943cb7c\") " pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:23.909379 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:23.909334    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 15:30:23.909504 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:23.909396    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 15:30:23.909504 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:23.909410    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8107c705-1f01-4194-bcde-679259e3ea51-cluster-monitoring-operator-tls podName:8107c705-1f01-4194-bcde-679259e3ea51 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:27.909393153 +0000 UTC m=+50.289410888 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8107c705-1f01-4194-bcde-679259e3ea51-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-t58k8" (UID: "8107c705-1f01-4194-bcde-679259e3ea51") : secret "cluster-monitoring-operator-tls" not found
Sep 17 15:30:23.909504 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:23.909468    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8b73fab2-e530-4b68-acf0-c71d1943cb7c-service-ca-bundle podName:8b73fab2-e530-4b68-acf0-c71d1943cb7c nodeName:}" failed. No retries permitted until 2026-09-17 15:30:27.909451626 +0000 UTC m=+50.289469370 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8b73fab2-e530-4b68-acf0-c71d1943cb7c-service-ca-bundle") pod "router-default-85cf78879b-z2qnj" (UID: "8b73fab2-e530-4b68-acf0-c71d1943cb7c") : configmap references non-existent config key: service-ca.crt
Sep 17 15:30:23.909504 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:23.909484    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b73fab2-e530-4b68-acf0-c71d1943cb7c-metrics-certs podName:8b73fab2-e530-4b68-acf0-c71d1943cb7c nodeName:}" failed. No retries permitted until 2026-09-17 15:30:27.909475162 +0000 UTC m=+50.289492896 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b73fab2-e530-4b68-acf0-c71d1943cb7c-metrics-certs") pod "router-default-85cf78879b-z2qnj" (UID: "8b73fab2-e530-4b68-acf0-c71d1943cb7c") : secret "router-metrics-certs-default" not found
Sep 17 15:30:25.824819 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:25.824786    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de6d8a3f-074e-4f18-9650-0ca1cb32100f-metrics-tls\") pod \"dns-default-k7kbh\" (UID: \"de6d8a3f-074e-4f18-9650-0ca1cb32100f\") " pod="openshift-dns/dns-default-k7kbh"
Sep 17 15:30:25.825243 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:25.824834    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-registry-tls\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:25.825243 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:25.824936    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:30:25.825243 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:25.824947    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69d6bcf58c-bq4rc: secret "image-registry-tls" not found
Sep 17 15:30:25.825243 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:25.824936    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:30:25.825243 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:25.824992    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-registry-tls podName:2581da35-8109-4df3-a553-c8992a31b8d1 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:41.824979736 +0000 UTC m=+64.204997497 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-registry-tls") pod "image-registry-69d6bcf58c-bq4rc" (UID: "2581da35-8109-4df3-a553-c8992a31b8d1") : secret "image-registry-tls" not found
Sep 17 15:30:25.825243 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:25.825090    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de6d8a3f-074e-4f18-9650-0ca1cb32100f-metrics-tls podName:de6d8a3f-074e-4f18-9650-0ca1cb32100f nodeName:}" failed. No retries permitted until 2026-09-17 15:30:41.825048349 +0000 UTC m=+64.205066083 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/de6d8a3f-074e-4f18-9650-0ca1cb32100f-metrics-tls") pod "dns-default-k7kbh" (UID: "de6d8a3f-074e-4f18-9650-0ca1cb32100f") : secret "dns-default-metrics-tls" not found
Sep 17 15:30:26.261500 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:26.261359    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-qjvgh"]
Sep 17 15:30:26.264906 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:30:26.264878    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa176539_bbe8_4970_8a21_269b3c2b52e9.slice/crio-6566070981b8e52279741eaa228f1c0926ec04e11aab7782d079ac9478fe1a8e WatchSource:0}: Error finding container 6566070981b8e52279741eaa228f1c0926ec04e11aab7782d079ac9478fe1a8e: Status 404 returned error can't find the container with id 6566070981b8e52279741eaa228f1c0926ec04e11aab7782d079ac9478fe1a8e
Sep 17 15:30:26.530910 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:26.530803    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-xqb4f" event={"ID":"95eb306e-4e4f-48ed-9b49-14471d4b4ddf","Type":"ContainerStarted","Data":"4980d8d8e9a874a76f78ee00372288097a8d6ae38c49baa54e1daaa62a41633e"}
Sep 17 15:30:26.532571 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:26.532540    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nb2mr" event={"ID":"dfe98000-7851-4180-b13f-bff7553a880b","Type":"ContainerStarted","Data":"1eb5fd9672adcd138e7e341d9ba811ec1b0771abc8d162c96b54c8f744e15816"}
Sep 17 15:30:26.534293 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:26.534206    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8pwxw" event={"ID":"c9d7e7c9-d70d-4fd3-86df-ada691b1d830","Type":"ContainerStarted","Data":"6ae761e1f8b727d0a55ad38fac85841d1e3db94c2b2a41732162c8c94570cc7a"}
Sep 17 15:30:26.535951 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:26.535922    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-r8mp6" event={"ID":"cdc3f7b4-7834-4853-87dc-cd68e3cdad3a","Type":"ContainerStarted","Data":"1189b46dc69c9ecdd64e18befad743f06878d82a0c3f563792b3d60a1d8d7b2f"}
Sep 17 15:30:26.537242 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:26.537208    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-qjvgh" event={"ID":"fa176539-bbe8-4970-8a21-269b3c2b52e9","Type":"ContainerStarted","Data":"6566070981b8e52279741eaa228f1c0926ec04e11aab7782d079ac9478fe1a8e"}
Sep 17 15:30:26.538847 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:26.538830    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-45zt7_2c069c9e-0856-46b0-ba10-67829c05ac94/console-operator/0.log"
Sep 17 15:30:26.538931 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:26.538863    2556 generic.go:356] "Generic (PLEG): container finished" podID="2c069c9e-0856-46b0-ba10-67829c05ac94" containerID="06b0ff9d4774d96e26c82c55db69542ee1ef9b47ab317b2d21849bc8575d091b" exitCode=255
Sep 17 15:30:26.538931 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:26.538888    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-45zt7" event={"ID":"2c069c9e-0856-46b0-ba10-67829c05ac94","Type":"ContainerDied","Data":"06b0ff9d4774d96e26c82c55db69542ee1ef9b47ab317b2d21849bc8575d091b"}
Sep 17 15:30:26.539154 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:26.539136    2556 scope.go:117] "RemoveContainer" containerID="06b0ff9d4774d96e26c82c55db69542ee1ef9b47ab317b2d21849bc8575d091b"
Sep 17 15:30:26.548560 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:26.548517    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-xqb4f" podStartSLOduration=1.366648276 podStartE2EDuration="6.54850156s" totalImagesPullingTime="5.181853284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:26.547479295 +0000 UTC m=+48.927497043" watchObservedRunningTime="2026-09-17 15:30:26.54850156 +0000 UTC m=+48.928519316"
Sep 17 15:30:26.580242 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:26.580137    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-r8mp6" podStartSLOduration=2.174139105 podStartE2EDuration="7.580120176s" totalImagesPullingTime="5.405981071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:26.563314588 +0000 UTC m=+48.943332346" watchObservedRunningTime="2026-09-17 15:30:26.580120176 +0000 UTC m=+48.960137929"
Sep 17 15:30:26.580242 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:26.580222    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-8pwxw" podStartSLOduration=1.299947882 podStartE2EDuration="6.580216717s" totalImagesPullingTime="5.280268835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:26.57983865 +0000 UTC m=+48.959856398" watchObservedRunningTime="2026-09-17 15:30:26.580216717 +0000 UTC m=+48.960234473"
Sep 17 15:30:26.596133 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:26.595600    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nb2mr" podStartSLOduration=1.34120605 podStartE2EDuration="6.595576355s" totalImagesPullingTime="5.254370305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:26.594362963 +0000 UTC m=+48.974380722" watchObservedRunningTime="2026-09-17 15:30:26.595576355 +0000 UTC m=+48.975594168"
Sep 17 15:30:26.706378 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:26.706343    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:26.777204 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:26.777168    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:27.543843 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.543809    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-45zt7_2c069c9e-0856-46b0-ba10-67829c05ac94/console-operator/1.log"
Sep 17 15:30:27.544333 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.544317    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-45zt7_2c069c9e-0856-46b0-ba10-67829c05ac94/console-operator/0.log"
Sep 17 15:30:27.544391 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.544353    2556 generic.go:356] "Generic (PLEG): container finished" podID="2c069c9e-0856-46b0-ba10-67829c05ac94" containerID="9a48ce25a8582aec34845decddde8ea678240d5c9c2d4aec5d0fda1e06402a60" exitCode=255
Sep 17 15:30:27.544442 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.544385    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-45zt7" event={"ID":"2c069c9e-0856-46b0-ba10-67829c05ac94","Type":"ContainerDied","Data":"9a48ce25a8582aec34845decddde8ea678240d5c9c2d4aec5d0fda1e06402a60"}
Sep 17 15:30:27.544492 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.544473    2556 scope.go:117] "RemoveContainer" containerID="06b0ff9d4774d96e26c82c55db69542ee1ef9b47ab317b2d21849bc8575d091b"
Sep 17 15:30:27.544737 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.544718    2556 scope.go:117] "RemoveContainer" containerID="9a48ce25a8582aec34845decddde8ea678240d5c9c2d4aec5d0fda1e06402a60"
Sep 17 15:30:27.545196 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:27.544930    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-45zt7_openshift-console-operator(2c069c9e-0856-46b0-ba10-67829c05ac94)\"" pod="openshift-console-operator/console-operator-696884fb9c-45zt7" podUID="2c069c9e-0856-46b0-ba10-67829c05ac94"
Sep 17 15:30:27.680471 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.679675    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-n7dnd"]
Sep 17 15:30:27.683077 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.682987    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n7dnd"
Sep 17 15:30:27.685620 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.685595    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 15:30:27.685840 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.685819    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hk45w\""
Sep 17 15:30:27.686129 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.686112    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 15:30:27.697576 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.697552    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n7dnd"]
Sep 17 15:30:27.704860 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.704692    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:27.742178 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.742144    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82788049-f54a-43a9-903a-f59be68b859a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n7dnd\" (UID: \"82788049-f54a-43a9-903a-f59be68b859a\") " pod="openshift-insights/insights-runtime-extractor-n7dnd"
Sep 17 15:30:27.742342 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.742221    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82788049-f54a-43a9-903a-f59be68b859a-data-volume\") pod \"insights-runtime-extractor-n7dnd\" (UID: \"82788049-f54a-43a9-903a-f59be68b859a\") " pod="openshift-insights/insights-runtime-extractor-n7dnd"
Sep 17 15:30:27.742342 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.742276    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82788049-f54a-43a9-903a-f59be68b859a-crio-socket\") pod \"insights-runtime-extractor-n7dnd\" (UID: \"82788049-f54a-43a9-903a-f59be68b859a\") " pod="openshift-insights/insights-runtime-extractor-n7dnd"
Sep 17 15:30:27.742413 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.742400    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82788049-f54a-43a9-903a-f59be68b859a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n7dnd\" (UID: \"82788049-f54a-43a9-903a-f59be68b859a\") " pod="openshift-insights/insights-runtime-extractor-n7dnd"
Sep 17 15:30:27.742444 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.742428    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr2nk\" (UniqueName: \"kubernetes.io/projected/82788049-f54a-43a9-903a-f59be68b859a-kube-api-access-nr2nk\") pod \"insights-runtime-extractor-n7dnd\" (UID: \"82788049-f54a-43a9-903a-f59be68b859a\") " pod="openshift-insights/insights-runtime-extractor-n7dnd"
Sep 17 15:30:27.843770 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.843679    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82788049-f54a-43a9-903a-f59be68b859a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n7dnd\" (UID: \"82788049-f54a-43a9-903a-f59be68b859a\") " pod="openshift-insights/insights-runtime-extractor-n7dnd"
Sep 17 15:30:27.843770 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.843726    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr2nk\" (UniqueName: \"kubernetes.io/projected/82788049-f54a-43a9-903a-f59be68b859a-kube-api-access-nr2nk\") pod \"insights-runtime-extractor-n7dnd\" (UID: \"82788049-f54a-43a9-903a-f59be68b859a\") " pod="openshift-insights/insights-runtime-extractor-n7dnd"
Sep 17 15:30:27.843977 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:27.843827    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:30:27.843977 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.843853    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fb3a087-736b-47bd-99ee-2acd40aa017e-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-cnf4j\" (UID: \"7fb3a087-736b-47bd-99ee-2acd40aa017e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-cnf4j"
Sep 17 15:30:27.843977 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:27.843902    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82788049-f54a-43a9-903a-f59be68b859a-kube-rbac-proxy-cm podName:82788049-f54a-43a9-903a-f59be68b859a nodeName:}" failed. No retries permitted until 2026-09-17 15:30:28.343880396 +0000 UTC m=+50.723898134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82788049-f54a-43a9-903a-f59be68b859a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n7dnd" (UID: "82788049-f54a-43a9-903a-f59be68b859a") : configmap "kube-rbac-proxy" not found
Sep 17 15:30:27.843977 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.843923    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82788049-f54a-43a9-903a-f59be68b859a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n7dnd\" (UID: \"82788049-f54a-43a9-903a-f59be68b859a\") " pod="openshift-insights/insights-runtime-extractor-n7dnd"
Sep 17 15:30:27.843977 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:27.843961    2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 15:30:27.844207 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.843990    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82788049-f54a-43a9-903a-f59be68b859a-data-volume\") pod \"insights-runtime-extractor-n7dnd\" (UID: \"82788049-f54a-43a9-903a-f59be68b859a\") " pod="openshift-insights/insights-runtime-extractor-n7dnd"
Sep 17 15:30:27.844207 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:27.844002    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fb3a087-736b-47bd-99ee-2acd40aa017e-samples-operator-tls podName:7fb3a087-736b-47bd-99ee-2acd40aa017e nodeName:}" failed. No retries permitted until 2026-09-17 15:30:35.84399032 +0000 UTC m=+58.224008074 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7fb3a087-736b-47bd-99ee-2acd40aa017e-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-cnf4j" (UID: "7fb3a087-736b-47bd-99ee-2acd40aa017e") : secret "samples-operator-tls" not found
Sep 17 15:30:27.844207 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:27.844088    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:30:27.844207 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:27.844136    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82788049-f54a-43a9-903a-f59be68b859a-insights-runtime-extractor-tls podName:82788049-f54a-43a9-903a-f59be68b859a nodeName:}" failed. No retries permitted until 2026-09-17 15:30:28.344123388 +0000 UTC m=+50.724141125 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82788049-f54a-43a9-903a-f59be68b859a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n7dnd" (UID: "82788049-f54a-43a9-903a-f59be68b859a") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:30:27.844207 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.844088    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82788049-f54a-43a9-903a-f59be68b859a-crio-socket\") pod \"insights-runtime-extractor-n7dnd\" (UID: \"82788049-f54a-43a9-903a-f59be68b859a\") " pod="openshift-insights/insights-runtime-extractor-n7dnd"
Sep 17 15:30:27.844207 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.844143    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82788049-f54a-43a9-903a-f59be68b859a-crio-socket\") pod \"insights-runtime-extractor-n7dnd\" (UID: \"82788049-f54a-43a9-903a-f59be68b859a\") " pod="openshift-insights/insights-runtime-extractor-n7dnd"
Sep 17 15:30:27.844430 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.844302    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82788049-f54a-43a9-903a-f59be68b859a-data-volume\") pod \"insights-runtime-extractor-n7dnd\" (UID: \"82788049-f54a-43a9-903a-f59be68b859a\") " pod="openshift-insights/insights-runtime-extractor-n7dnd"
Sep 17 15:30:27.853828 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.853803    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr2nk\" (UniqueName: \"kubernetes.io/projected/82788049-f54a-43a9-903a-f59be68b859a-kube-api-access-nr2nk\") pod \"insights-runtime-extractor-n7dnd\" (UID: \"82788049-f54a-43a9-903a-f59be68b859a\") " pod="openshift-insights/insights-runtime-extractor-n7dnd"
Sep 17 15:30:27.945366 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.945329    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8107c705-1f01-4194-bcde-679259e3ea51-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-t58k8\" (UID: \"8107c705-1f01-4194-bcde-679259e3ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-t58k8"
Sep 17 15:30:27.945560 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.945378    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b73fab2-e530-4b68-acf0-c71d1943cb7c-metrics-certs\") pod \"router-default-85cf78879b-z2qnj\" (UID: \"8b73fab2-e530-4b68-acf0-c71d1943cb7c\") " pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:27.945560 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.945397    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b73fab2-e530-4b68-acf0-c71d1943cb7c-service-ca-bundle\") pod \"router-default-85cf78879b-z2qnj\" (UID: \"8b73fab2-e530-4b68-acf0-c71d1943cb7c\") " pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:27.945560 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:27.945497    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 15:30:27.945560 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:27.945502    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 15:30:27.945726 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:27.945581    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8107c705-1f01-4194-bcde-679259e3ea51-cluster-monitoring-operator-tls podName:8107c705-1f01-4194-bcde-679259e3ea51 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:35.945560161 +0000 UTC m=+58.325577898 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8107c705-1f01-4194-bcde-679259e3ea51-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-t58k8" (UID: "8107c705-1f01-4194-bcde-679259e3ea51") : secret "cluster-monitoring-operator-tls" not found
Sep 17 15:30:27.945726 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:27.945602    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b73fab2-e530-4b68-acf0-c71d1943cb7c-metrics-certs podName:8b73fab2-e530-4b68-acf0-c71d1943cb7c nodeName:}" failed. No retries permitted until 2026-09-17 15:30:35.945592355 +0000 UTC m=+58.325610089 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b73fab2-e530-4b68-acf0-c71d1943cb7c-metrics-certs") pod "router-default-85cf78879b-z2qnj" (UID: "8b73fab2-e530-4b68-acf0-c71d1943cb7c") : secret "router-metrics-certs-default" not found
Sep 17 15:30:27.945726 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:27.945631    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8b73fab2-e530-4b68-acf0-c71d1943cb7c-service-ca-bundle podName:8b73fab2-e530-4b68-acf0-c71d1943cb7c nodeName:}" failed. No retries permitted until 2026-09-17 15:30:35.945612514 +0000 UTC m=+58.325630269 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8b73fab2-e530-4b68-acf0-c71d1943cb7c-service-ca-bundle") pod "router-default-85cf78879b-z2qnj" (UID: "8b73fab2-e530-4b68-acf0-c71d1943cb7c") : configmap references non-existent config key: service-ca.crt
Sep 17 15:30:27.976669 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:27.976633    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:28.014589 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:28.014167    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-m7bx8"]
Sep 17 15:30:28.016911 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:28.016888    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-m7bx8"
Sep 17 15:30:28.019247 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:28.019212    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 15:30:28.019379 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:28.019215    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 15:30:28.019635 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:28.019599    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-txkzb\""
Sep 17 15:30:28.026450 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:28.026424    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-m7bx8"]
Sep 17 15:30:28.041985 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:28.041916    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:28.046404 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:28.046378    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzdrb\" (UniqueName: \"kubernetes.io/projected/dcdeb8a6-35bd-41e8-b1be-a926f3bd4a7a-kube-api-access-wzdrb\") pod \"migrator-8b4d697c5-m7bx8\" (UID: \"dcdeb8a6-35bd-41e8-b1be-a926f3bd4a7a\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-m7bx8"
Sep 17 15:30:28.103795 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:28.103758    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:28.147575 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:28.147528    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzdrb\" (UniqueName: \"kubernetes.io/projected/dcdeb8a6-35bd-41e8-b1be-a926f3bd4a7a-kube-api-access-wzdrb\") pod \"migrator-8b4d697c5-m7bx8\" (UID: \"dcdeb8a6-35bd-41e8-b1be-a926f3bd4a7a\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-m7bx8"
Sep 17 15:30:28.155913 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:28.155877    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzdrb\" (UniqueName: \"kubernetes.io/projected/dcdeb8a6-35bd-41e8-b1be-a926f3bd4a7a-kube-api-access-wzdrb\") pod \"migrator-8b4d697c5-m7bx8\" (UID: \"dcdeb8a6-35bd-41e8-b1be-a926f3bd4a7a\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-m7bx8"
Sep 17 15:30:28.330528 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:28.330488    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-m7bx8"
Sep 17 15:30:28.349593 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:28.349562    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82788049-f54a-43a9-903a-f59be68b859a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n7dnd\" (UID: \"82788049-f54a-43a9-903a-f59be68b859a\") " pod="openshift-insights/insights-runtime-extractor-n7dnd"
Sep 17 15:30:28.349734 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:28.349625    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82788049-f54a-43a9-903a-f59be68b859a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n7dnd\" (UID: \"82788049-f54a-43a9-903a-f59be68b859a\") " pod="openshift-insights/insights-runtime-extractor-n7dnd"
Sep 17 15:30:28.349796 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:28.349763    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:30:28.349869 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:28.349849    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:30:28.349915 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:28.349852    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82788049-f54a-43a9-903a-f59be68b859a-insights-runtime-extractor-tls podName:82788049-f54a-43a9-903a-f59be68b859a nodeName:}" failed. No retries permitted until 2026-09-17 15:30:29.34982991 +0000 UTC m=+51.729847657 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82788049-f54a-43a9-903a-f59be68b859a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n7dnd" (UID: "82788049-f54a-43a9-903a-f59be68b859a") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:30:28.349980 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:28.349966    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82788049-f54a-43a9-903a-f59be68b859a-kube-rbac-proxy-cm podName:82788049-f54a-43a9-903a-f59be68b859a nodeName:}" failed. No retries permitted until 2026-09-17 15:30:29.349945759 +0000 UTC m=+51.729963504 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82788049-f54a-43a9-903a-f59be68b859a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n7dnd" (UID: "82788049-f54a-43a9-903a-f59be68b859a") : configmap "kube-rbac-proxy" not found
Sep 17 15:30:28.476224 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:28.476190    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:28.550464 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:28.550432    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-45zt7_2c069c9e-0856-46b0-ba10-67829c05ac94/console-operator/1.log"
Sep 17 15:30:28.550894 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:28.550878    2556 scope.go:117] "RemoveContainer" containerID="9a48ce25a8582aec34845decddde8ea678240d5c9c2d4aec5d0fda1e06402a60"
Sep 17 15:30:28.551146 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:28.551115    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-45zt7_openshift-console-operator(2c069c9e-0856-46b0-ba10-67829c05ac94)\"" pod="openshift-console-operator/console-operator-696884fb9c-45zt7" podUID="2c069c9e-0856-46b0-ba10-67829c05ac94"
Sep 17 15:30:28.602155 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:28.602119    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:28.618769 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:28.618740    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-m7bx8"]
Sep 17 15:30:28.622779 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:30:28.622748    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcdeb8a6_35bd_41e8_b1be_a926f3bd4a7a.slice/crio-97b561d5a9f30018e60cab9eafac0dc2b6448d2c57c89d3ecf267d3a83223532 WatchSource:0}: Error finding container 97b561d5a9f30018e60cab9eafac0dc2b6448d2c57c89d3ecf267d3a83223532: Status 404 returned error can't find the container with id 97b561d5a9f30018e60cab9eafac0dc2b6448d2c57c89d3ecf267d3a83223532
Sep 17 15:30:28.738619 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:28.738438    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:28.874445 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:28.874414    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:29.023946 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:29.023838    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:29.043623 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:29.043598    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xfzrp_593418fc-096d-445c-9863-9ed9f57ee092/dns-node-resolver/0.log"
Sep 17 15:30:29.360817 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:29.360761    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82788049-f54a-43a9-903a-f59be68b859a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n7dnd\" (UID: \"82788049-f54a-43a9-903a-f59be68b859a\") " pod="openshift-insights/insights-runtime-extractor-n7dnd"
Sep 17 15:30:29.361001 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:29.360855    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82788049-f54a-43a9-903a-f59be68b859a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n7dnd\" (UID: \"82788049-f54a-43a9-903a-f59be68b859a\") " pod="openshift-insights/insights-runtime-extractor-n7dnd"
Sep 17 15:30:29.361001 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:29.360909    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:30:29.361001 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:29.360984    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82788049-f54a-43a9-903a-f59be68b859a-kube-rbac-proxy-cm podName:82788049-f54a-43a9-903a-f59be68b859a nodeName:}" failed. No retries permitted until 2026-09-17 15:30:31.360962208 +0000 UTC m=+53.740979962 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82788049-f54a-43a9-903a-f59be68b859a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n7dnd" (UID: "82788049-f54a-43a9-903a-f59be68b859a") : configmap "kube-rbac-proxy" not found
Sep 17 15:30:29.361201 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:29.361037    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:30:29.361201 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:29.361108    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82788049-f54a-43a9-903a-f59be68b859a-insights-runtime-extractor-tls podName:82788049-f54a-43a9-903a-f59be68b859a nodeName:}" failed. No retries permitted until 2026-09-17 15:30:31.361093324 +0000 UTC m=+53.741111063 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82788049-f54a-43a9-903a-f59be68b859a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n7dnd" (UID: "82788049-f54a-43a9-903a-f59be68b859a") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:30:29.556776 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:29.556739    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-m7bx8" event={"ID":"dcdeb8a6-35bd-41e8-b1be-a926f3bd4a7a","Type":"ContainerStarted","Data":"97b561d5a9f30018e60cab9eafac0dc2b6448d2c57c89d3ecf267d3a83223532"}
Sep 17 15:30:29.812515 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:29.812457    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:29.912953 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:29.912913    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:30.042677 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.042591    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s9svr_fca9ba31-c254-4a6e-8607-fb6a4af8efc2/node-ca/0.log"
Sep 17 15:30:30.335654 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.335561    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-lgdm7"]
Sep 17 15:30:30.338785 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.338763    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-lgdm7"
Sep 17 15:30:30.341247 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.341209    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 15:30:30.342045 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.341995    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 15:30:30.342173 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.342116    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 15:30:30.342173 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.342139    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-dqmf4\""
Sep 17 15:30:30.342284 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.342172    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 15:30:30.352425 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.352399    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-lgdm7"]
Sep 17 15:30:30.372409 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.372374    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/580a9d92-206d-4257-8279-5b54a81132ab-signing-key\") pod \"service-ca-568869fdc5-lgdm7\" (UID: \"580a9d92-206d-4257-8279-5b54a81132ab\") " pod="openshift-service-ca/service-ca-568869fdc5-lgdm7"
Sep 17 15:30:30.372409 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.372417    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrhk5\" (UniqueName: \"kubernetes.io/projected/580a9d92-206d-4257-8279-5b54a81132ab-kube-api-access-xrhk5\") pod \"service-ca-568869fdc5-lgdm7\" (UID: \"580a9d92-206d-4257-8279-5b54a81132ab\") " pod="openshift-service-ca/service-ca-568869fdc5-lgdm7"
Sep 17 15:30:30.372650 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.372559    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/580a9d92-206d-4257-8279-5b54a81132ab-signing-cabundle\") pod \"service-ca-568869fdc5-lgdm7\" (UID: \"580a9d92-206d-4257-8279-5b54a81132ab\") " pod="openshift-service-ca/service-ca-568869fdc5-lgdm7"
Sep 17 15:30:30.388547 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.388499    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:30.445230 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.445196    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r975h_ac3c1f9c-dbfd-4097-b8c5-50d3fa2962c5/serve-healthcheck-canary/0.log"
Sep 17 15:30:30.448080 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.448024    2556 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-45zt7"
Sep 17 15:30:30.448217 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.448099    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-45zt7"
Sep 17 15:30:30.448217 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.448175    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:30.448562 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.448546    2556 scope.go:117] "RemoveContainer" containerID="9a48ce25a8582aec34845decddde8ea678240d5c9c2d4aec5d0fda1e06402a60"
Sep 17 15:30:30.448791 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:30.448774    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-45zt7_openshift-console-operator(2c069c9e-0856-46b0-ba10-67829c05ac94)\"" pod="openshift-console-operator/console-operator-696884fb9c-45zt7" podUID="2c069c9e-0856-46b0-ba10-67829c05ac94"
Sep 17 15:30:30.473519 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.473488    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/580a9d92-206d-4257-8279-5b54a81132ab-signing-key\") pod \"service-ca-568869fdc5-lgdm7\" (UID: \"580a9d92-206d-4257-8279-5b54a81132ab\") " pod="openshift-service-ca/service-ca-568869fdc5-lgdm7"
Sep 17 15:30:30.473634 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.473534    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrhk5\" (UniqueName: \"kubernetes.io/projected/580a9d92-206d-4257-8279-5b54a81132ab-kube-api-access-xrhk5\") pod \"service-ca-568869fdc5-lgdm7\" (UID: \"580a9d92-206d-4257-8279-5b54a81132ab\") " pod="openshift-service-ca/service-ca-568869fdc5-lgdm7"
Sep 17 15:30:30.473694 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.473671    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/580a9d92-206d-4257-8279-5b54a81132ab-signing-cabundle\") pod \"service-ca-568869fdc5-lgdm7\" (UID: \"580a9d92-206d-4257-8279-5b54a81132ab\") " pod="openshift-service-ca/service-ca-568869fdc5-lgdm7"
Sep 17 15:30:30.474387 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.474363    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/580a9d92-206d-4257-8279-5b54a81132ab-signing-cabundle\") pod \"service-ca-568869fdc5-lgdm7\" (UID: \"580a9d92-206d-4257-8279-5b54a81132ab\") " pod="openshift-service-ca/service-ca-568869fdc5-lgdm7"
Sep 17 15:30:30.476546 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.476525    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/580a9d92-206d-4257-8279-5b54a81132ab-signing-key\") pod \"service-ca-568869fdc5-lgdm7\" (UID: \"580a9d92-206d-4257-8279-5b54a81132ab\") " pod="openshift-service-ca/service-ca-568869fdc5-lgdm7"
Sep 17 15:30:30.480999 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.480977    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrhk5\" (UniqueName: \"kubernetes.io/projected/580a9d92-206d-4257-8279-5b54a81132ab-kube-api-access-xrhk5\") pod \"service-ca-568869fdc5-lgdm7\" (UID: \"580a9d92-206d-4257-8279-5b54a81132ab\") " pod="openshift-service-ca/service-ca-568869fdc5-lgdm7"
Sep 17 15:30:30.561340 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.561300    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-qjvgh" event={"ID":"fa176539-bbe8-4970-8a21-269b3c2b52e9","Type":"ContainerStarted","Data":"952dc9be022d82b2b5ee3691a62b5ac5138e40854e0be46ae298c135dc6a8fce"}
Sep 17 15:30:30.562907 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.562883    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-m7bx8" event={"ID":"dcdeb8a6-35bd-41e8-b1be-a926f3bd4a7a","Type":"ContainerStarted","Data":"011f1583dd693e0c2c8f502f34c0ba22eb13b972a1d392329dfb9889a66c5127"}
Sep 17 15:30:30.577135 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.577083    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-qjvgh" podStartSLOduration=4.165795856 podStartE2EDuration="7.577040802s" totalImagesPullingTime="3.411244946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:30.576695796 +0000 UTC m=+52.956713552" watchObservedRunningTime="2026-09-17 15:30:30.577040802 +0000 UTC m=+52.957058559"
Sep 17 15:30:30.650137 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.650092    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-lgdm7"
Sep 17 15:30:30.757805 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.757762    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:30.831052 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.831008    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:30.931400 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:30.931312    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-lgdm7"]
Sep 17 15:30:30.935034 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:30:30.935001    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580a9d92_206d_4257_8279_5b54a81132ab.slice/crio-5d774622df8a4cbf439e4bb65757f6728cc07af68d592300f2e7695dd94a8739 WatchSource:0}: Error finding container 5d774622df8a4cbf439e4bb65757f6728cc07af68d592300f2e7695dd94a8739: Status 404 returned error can't find the container with id 5d774622df8a4cbf439e4bb65757f6728cc07af68d592300f2e7695dd94a8739
Sep 17 15:30:31.328737 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:31.328647    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:31.382940 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:31.382905    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82788049-f54a-43a9-903a-f59be68b859a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n7dnd\" (UID: \"82788049-f54a-43a9-903a-f59be68b859a\") " pod="openshift-insights/insights-runtime-extractor-n7dnd"
Sep 17 15:30:31.383145 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:31.383048    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82788049-f54a-43a9-903a-f59be68b859a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n7dnd\" (UID: \"82788049-f54a-43a9-903a-f59be68b859a\") " pod="openshift-insights/insights-runtime-extractor-n7dnd"
Sep 17 15:30:31.383145 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:31.383086    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:30:31.383145 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:31.383137    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:30:31.383316 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:31.383168    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82788049-f54a-43a9-903a-f59be68b859a-insights-runtime-extractor-tls podName:82788049-f54a-43a9-903a-f59be68b859a nodeName:}" failed. No retries permitted until 2026-09-17 15:30:35.38314667 +0000 UTC m=+57.763164404 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82788049-f54a-43a9-903a-f59be68b859a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n7dnd" (UID: "82788049-f54a-43a9-903a-f59be68b859a") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:30:31.383316 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:31.383193    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82788049-f54a-43a9-903a-f59be68b859a-kube-rbac-proxy-cm podName:82788049-f54a-43a9-903a-f59be68b859a nodeName:}" failed. No retries permitted until 2026-09-17 15:30:35.38317912 +0000 UTC m=+57.763196857 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82788049-f54a-43a9-903a-f59be68b859a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n7dnd" (UID: "82788049-f54a-43a9-903a-f59be68b859a") : configmap "kube-rbac-proxy" not found
Sep 17 15:30:31.567859 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:31.567822    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-m7bx8" event={"ID":"dcdeb8a6-35bd-41e8-b1be-a926f3bd4a7a","Type":"ContainerStarted","Data":"fad5ccbb9d609d6c9081a6e5b0e5ae2fbbcab7b0b5bf68359794f739b722d523"}
Sep 17 15:30:31.569422 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:31.569396    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-lgdm7" event={"ID":"580a9d92-206d-4257-8279-5b54a81132ab","Type":"ContainerStarted","Data":"1113569cc73cf44f71b786837de87e1647db108cf76508d78912f5eacaa1182b"}
Sep 17 15:30:31.569561 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:31.569437    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-lgdm7" event={"ID":"580a9d92-206d-4257-8279-5b54a81132ab","Type":"ContainerStarted","Data":"5d774622df8a4cbf439e4bb65757f6728cc07af68d592300f2e7695dd94a8739"}
Sep 17 15:30:31.628777 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:31.628720    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-lgdm7" podStartSLOduration=1.628701593 podStartE2EDuration="1.628701593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:31.627896261 +0000 UTC m=+54.007914017" watchObservedRunningTime="2026-09-17 15:30:31.628701593 +0000 UTC m=+54.008719387"
Sep 17 15:30:31.629004 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:31.628944    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-m7bx8" podStartSLOduration=2.79234408 podStartE2EDuration="4.628938823s" totalImagesPullingTime="1.836594743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:31.596130873 +0000 UTC m=+53.976148632" watchObservedRunningTime="2026-09-17 15:30:31.628938823 +0000 UTC m=+54.008956631"
Sep 17 15:30:32.217079 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:32.217020    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:32.317520 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:32.317464    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:32.444518 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:32.444470    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:32.559979 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:32.559887    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:32.698929 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:32.698882    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:32.809582 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:32.809545    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:32.902332 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:32.902291    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:33.274657 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:33.274563    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:33.945093 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:33.945036    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:34.290298 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:34.290189    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:34.498555 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:34.498494    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log" limit=1024
Sep 17 15:30:35.419783 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:35.419739    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82788049-f54a-43a9-903a-f59be68b859a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n7dnd\" (UID: \"82788049-f54a-43a9-903a-f59be68b859a\") " pod="openshift-insights/insights-runtime-extractor-n7dnd"
Sep 17 15:30:35.420205 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:35.419815    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82788049-f54a-43a9-903a-f59be68b859a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n7dnd\" (UID: \"82788049-f54a-43a9-903a-f59be68b859a\") " pod="openshift-insights/insights-runtime-extractor-n7dnd"
Sep 17 15:30:35.420205 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:35.419859    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:30:35.420205 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:35.419935    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82788049-f54a-43a9-903a-f59be68b859a-kube-rbac-proxy-cm podName:82788049-f54a-43a9-903a-f59be68b859a nodeName:}" failed. No retries permitted until 2026-09-17 15:30:43.419919392 +0000 UTC m=+65.799937125 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82788049-f54a-43a9-903a-f59be68b859a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n7dnd" (UID: "82788049-f54a-43a9-903a-f59be68b859a") : configmap "kube-rbac-proxy" not found
Sep 17 15:30:35.420205 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:35.419965    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:30:35.420205 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:35.420033    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82788049-f54a-43a9-903a-f59be68b859a-insights-runtime-extractor-tls podName:82788049-f54a-43a9-903a-f59be68b859a nodeName:}" failed. No retries permitted until 2026-09-17 15:30:43.42002101 +0000 UTC m=+65.800038744 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82788049-f54a-43a9-903a-f59be68b859a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n7dnd" (UID: "82788049-f54a-43a9-903a-f59be68b859a") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:30:35.924682 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:35.924633    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fb3a087-736b-47bd-99ee-2acd40aa017e-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-cnf4j\" (UID: \"7fb3a087-736b-47bd-99ee-2acd40aa017e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-cnf4j"
Sep 17 15:30:35.927203 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:35.927178    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fb3a087-736b-47bd-99ee-2acd40aa017e-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-cnf4j\" (UID: \"7fb3a087-736b-47bd-99ee-2acd40aa017e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-cnf4j"
Sep 17 15:30:36.026010 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:36.025972    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8107c705-1f01-4194-bcde-679259e3ea51-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-t58k8\" (UID: \"8107c705-1f01-4194-bcde-679259e3ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-t58k8"
Sep 17 15:30:36.026190 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:36.026025    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b73fab2-e530-4b68-acf0-c71d1943cb7c-metrics-certs\") pod \"router-default-85cf78879b-z2qnj\" (UID: \"8b73fab2-e530-4b68-acf0-c71d1943cb7c\") " pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:36.026190 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:36.026043    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b73fab2-e530-4b68-acf0-c71d1943cb7c-service-ca-bundle\") pod \"router-default-85cf78879b-z2qnj\" (UID: \"8b73fab2-e530-4b68-acf0-c71d1943cb7c\") " pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:36.026190 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:36.026166    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 15:30:36.026314 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:36.026212    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 15:30:36.026314 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:36.026170    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8b73fab2-e530-4b68-acf0-c71d1943cb7c-service-ca-bundle podName:8b73fab2-e530-4b68-acf0-c71d1943cb7c nodeName:}" failed. No retries permitted until 2026-09-17 15:30:52.026156263 +0000 UTC m=+74.406173997 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8b73fab2-e530-4b68-acf0-c71d1943cb7c-service-ca-bundle") pod "router-default-85cf78879b-z2qnj" (UID: "8b73fab2-e530-4b68-acf0-c71d1943cb7c") : configmap references non-existent config key: service-ca.crt
Sep 17 15:30:36.026314 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:36.026259    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8107c705-1f01-4194-bcde-679259e3ea51-cluster-monitoring-operator-tls podName:8107c705-1f01-4194-bcde-679259e3ea51 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:52.026246157 +0000 UTC m=+74.406263894 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8107c705-1f01-4194-bcde-679259e3ea51-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-t58k8" (UID: "8107c705-1f01-4194-bcde-679259e3ea51") : secret "cluster-monitoring-operator-tls" not found
Sep 17 15:30:36.026314 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:30:36.026281    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b73fab2-e530-4b68-acf0-c71d1943cb7c-metrics-certs podName:8b73fab2-e530-4b68-acf0-c71d1943cb7c nodeName:}" failed. No retries permitted until 2026-09-17 15:30:52.026271977 +0000 UTC m=+74.406289716 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b73fab2-e530-4b68-acf0-c71d1943cb7c-metrics-certs") pod "router-default-85cf78879b-z2qnj" (UID: "8b73fab2-e530-4b68-acf0-c71d1943cb7c") : secret "router-metrics-certs-default" not found
Sep 17 15:30:36.206525 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:36.206439    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-cnf4j"
Sep 17 15:30:36.477937 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:36.477851    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-cnf4j"]
Sep 17 15:30:36.586801 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:36.586761    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-cnf4j" event={"ID":"7fb3a087-736b-47bd-99ee-2acd40aa017e","Type":"ContainerStarted","Data":"498bc4a748cb37e96ae97e1260d1a2fc258efaf671539af164bb6a4153a24ba4"}
Sep 17 15:30:38.595195 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:38.595096    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-cnf4j" event={"ID":"7fb3a087-736b-47bd-99ee-2acd40aa017e","Type":"ContainerStarted","Data":"10fe55b0f9da59f8bfeaf5ee37a7bb21a4c01d70a4d3a4e9f6ada8587963d014"}
Sep 17 15:30:38.595195 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:38.595162    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-cnf4j" event={"ID":"7fb3a087-736b-47bd-99ee-2acd40aa017e","Type":"ContainerStarted","Data":"a9d564de06f6a5228a84c53fb4b866974d4841a5a43e75ec6e1a8fc80b8283e4"}
Sep 17 15:30:38.612257 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:38.612201    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-cnf4j" podStartSLOduration=17.859024618 podStartE2EDuration="19.612184751s" totalImagesPullingTime="1.753160133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:38.610553114 +0000 UTC m=+60.990570872" watchObservedRunningTime="2026-09-17 15:30:38.612184751 +0000 UTC m=+60.992202507"
Sep 17 15:30:41.880672 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:41.880630    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de6d8a3f-074e-4f18-9650-0ca1cb32100f-metrics-tls\") pod \"dns-default-k7kbh\" (UID: \"de6d8a3f-074e-4f18-9650-0ca1cb32100f\") " pod="openshift-dns/dns-default-k7kbh"
Sep 17 15:30:41.881129 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:41.880693    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-registry-tls\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:41.883129 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:41.883106    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de6d8a3f-074e-4f18-9650-0ca1cb32100f-metrics-tls\") pod \"dns-default-k7kbh\" (UID: \"de6d8a3f-074e-4f18-9650-0ca1cb32100f\") " pod="openshift-dns/dns-default-k7kbh"
Sep 17 15:30:41.883184 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:41.883155    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-registry-tls\") pod \"image-registry-69d6bcf58c-bq4rc\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") " pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:42.102529 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:42.102494    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lzxqs\""
Sep 17 15:30:42.109010 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:42.108988    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:42.118090 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:42.118072    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l7hr4\""
Sep 17 15:30:42.123324 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:42.123301    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k7kbh"
Sep 17 15:30:42.398857 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:42.398775    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69d6bcf58c-bq4rc"]
Sep 17 15:30:42.402882 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:30:42.402851    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2581da35_8109_4df3_a553_c8992a31b8d1.slice/crio-afc21b3bab1abcbac2dc1accde26954cadd917e98337d0b985befded6e639a48 WatchSource:0}: Error finding container afc21b3bab1abcbac2dc1accde26954cadd917e98337d0b985befded6e639a48: Status 404 returned error can't find the container with id afc21b3bab1abcbac2dc1accde26954cadd917e98337d0b985befded6e639a48
Sep 17 15:30:42.413046 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:42.413017    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k7kbh"]
Sep 17 15:30:42.416296 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:30:42.416268    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde6d8a3f_074e_4f18_9650_0ca1cb32100f.slice/crio-bb0dfc014aed101caf7143241bb0a6932e10f7de60996fe8e3113bacf0d62167 WatchSource:0}: Error finding container bb0dfc014aed101caf7143241bb0a6932e10f7de60996fe8e3113bacf0d62167: Status 404 returned error can't find the container with id bb0dfc014aed101caf7143241bb0a6932e10f7de60996fe8e3113bacf0d62167
Sep 17 15:30:42.609786 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:42.609747    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k7kbh" event={"ID":"de6d8a3f-074e-4f18-9650-0ca1cb32100f","Type":"ContainerStarted","Data":"bb0dfc014aed101caf7143241bb0a6932e10f7de60996fe8e3113bacf0d62167"}
Sep 17 15:30:42.611408 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:42.611382    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc" event={"ID":"2581da35-8109-4df3-a553-c8992a31b8d1","Type":"ContainerStarted","Data":"7076e2af0305555d99c8810b58b12283cd6caa0d052715734b03a4c34179437c"}
Sep 17 15:30:42.611556 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:42.611421    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc" event={"ID":"2581da35-8109-4df3-a553-c8992a31b8d1","Type":"ContainerStarted","Data":"afc21b3bab1abcbac2dc1accde26954cadd917e98337d0b985befded6e639a48"}
Sep 17 15:30:42.611556 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:42.611524    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:30:42.634709 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:42.634653    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc" podStartSLOduration=64.634638056 podStartE2EDuration="1m4.634638056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:42.633465246 +0000 UTC m=+65.013483003" watchObservedRunningTime="2026-09-17 15:30:42.634638056 +0000 UTC m=+65.014655812"
Sep 17 15:30:42.889599 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:42.889563    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plpr7\" (UniqueName: \"kubernetes.io/projected/352c01b0-7c02-4da9-a311-816815cd5e1e-kube-api-access-plpr7\") pod \"network-check-target-l2rxt\" (UID: \"352c01b0-7c02-4da9-a311-816815cd5e1e\") " pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:30:42.890015 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:42.889758    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84728120-5cde-4b79-944d-7554db40cf41-original-pull-secret\") pod \"global-pull-secret-syncer-r5jgb\" (UID: \"84728120-5cde-4b79-944d-7554db40cf41\") " pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:30:42.892032 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:42.892010    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plpr7\" (UniqueName: \"kubernetes.io/projected/352c01b0-7c02-4da9-a311-816815cd5e1e-kube-api-access-plpr7\") pod \"network-check-target-l2rxt\" (UID: \"352c01b0-7c02-4da9-a311-816815cd5e1e\") " pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:30:42.892372 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:42.892357    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 15:30:42.902399 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:42.902379    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/84728120-5cde-4b79-944d-7554db40cf41-original-pull-secret\") pod \"global-pull-secret-syncer-r5jgb\" (UID: \"84728120-5cde-4b79-944d-7554db40cf41\") " pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:30:42.942803 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:42.942769    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5jgb"
Sep 17 15:30:42.951722 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:42.951698    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zrbtw\""
Sep 17 15:30:42.959971 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:42.959943    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:30:42.990506 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:42.990461    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eae675d6-96cb-4843-9deb-31182313d8ac-metrics-certs\") pod \"network-metrics-daemon-hn89v\" (UID: \"eae675d6-96cb-4843-9deb-31182313d8ac\") " pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:30:42.993108 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:42.993022    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 15:30:43.004051 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:43.003966    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eae675d6-96cb-4843-9deb-31182313d8ac-metrics-certs\") pod \"network-metrics-daemon-hn89v\" (UID: \"eae675d6-96cb-4843-9deb-31182313d8ac\") " pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:30:43.129844 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:43.129813    2556 scope.go:117] "RemoveContainer" containerID="9a48ce25a8582aec34845decddde8ea678240d5c9c2d4aec5d0fda1e06402a60"
Sep 17 15:30:43.219156 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:43.219126    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r5jgb"]
Sep 17 15:30:43.222296 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:30:43.222263    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84728120_5cde_4b79_944d_7554db40cf41.slice/crio-2bbb569d9284087bb38630da294e6dcaf8eb7c78b42407afe4cdec8312f54d69 WatchSource:0}: Error finding container 2bbb569d9284087bb38630da294e6dcaf8eb7c78b42407afe4cdec8312f54d69: Status 404 returned error can't find the container with id 2bbb569d9284087bb38630da294e6dcaf8eb7c78b42407afe4cdec8312f54d69
Sep 17 15:30:43.245238 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:43.245166    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l2rxt"]
Sep 17 15:30:43.248632 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:30:43.248600    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod352c01b0_7c02_4da9_a311_816815cd5e1e.slice/crio-39f4a498096161c1592e102504b132b5933f9629eeaf2ec1af5dc468f663fd04 WatchSource:0}: Error finding container 39f4a498096161c1592e102504b132b5933f9629eeaf2ec1af5dc468f663fd04: Status 404 returned error can't find the container with id 39f4a498096161c1592e102504b132b5933f9629eeaf2ec1af5dc468f663fd04
Sep 17 15:30:43.258344 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:43.258317    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pmg9m\""
Sep 17 15:30:43.266478 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:43.266450    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hn89v"
Sep 17 15:30:43.495101 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:43.495002    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82788049-f54a-43a9-903a-f59be68b859a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n7dnd\" (UID: \"82788049-f54a-43a9-903a-f59be68b859a\") " pod="openshift-insights/insights-runtime-extractor-n7dnd"
Sep 17 15:30:43.495288 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:43.495249    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82788049-f54a-43a9-903a-f59be68b859a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n7dnd\" (UID: \"82788049-f54a-43a9-903a-f59be68b859a\") " pod="openshift-insights/insights-runtime-extractor-n7dnd"
Sep 17 15:30:43.495741 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:43.495715    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82788049-f54a-43a9-903a-f59be68b859a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n7dnd\" (UID: \"82788049-f54a-43a9-903a-f59be68b859a\") " pod="openshift-insights/insights-runtime-extractor-n7dnd"
Sep 17 15:30:43.497649 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:43.497612    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82788049-f54a-43a9-903a-f59be68b859a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n7dnd\" (UID: \"82788049-f54a-43a9-903a-f59be68b859a\") " pod="openshift-insights/insights-runtime-extractor-n7dnd"
Sep 17 15:30:43.555261 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:43.555220    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hn89v"]
Sep 17 15:30:43.559829 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:30:43.559793    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae675d6_96cb_4843_9deb_31182313d8ac.slice/crio-5c529bc1effbfaab80d01a10bdd9fe1fbdf9fb41e2b5402a9ef0275cebc4afc0 WatchSource:0}: Error finding container 5c529bc1effbfaab80d01a10bdd9fe1fbdf9fb41e2b5402a9ef0275cebc4afc0: Status 404 returned error can't find the container with id 5c529bc1effbfaab80d01a10bdd9fe1fbdf9fb41e2b5402a9ef0275cebc4afc0
Sep 17 15:30:43.598312 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:43.598283    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hk45w\""
Sep 17 15:30:43.606414 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:43.606373    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n7dnd"
Sep 17 15:30:43.616891 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:43.616809    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hn89v" event={"ID":"eae675d6-96cb-4843-9deb-31182313d8ac","Type":"ContainerStarted","Data":"5c529bc1effbfaab80d01a10bdd9fe1fbdf9fb41e2b5402a9ef0275cebc4afc0"}
Sep 17 15:30:43.618793 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:43.618752    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l2rxt" event={"ID":"352c01b0-7c02-4da9-a311-816815cd5e1e","Type":"ContainerStarted","Data":"e6b5635f3efe8c21f6190c5cc71709a771c7fd74fd5ef352108c1066325f0dee"}
Sep 17 15:30:43.618918 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:43.618809    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l2rxt" event={"ID":"352c01b0-7c02-4da9-a311-816815cd5e1e","Type":"ContainerStarted","Data":"39f4a498096161c1592e102504b132b5933f9629eeaf2ec1af5dc468f663fd04"}
Sep 17 15:30:43.618918 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:43.618867    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:30:43.620435 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:43.620339    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r5jgb" event={"ID":"84728120-5cde-4b79-944d-7554db40cf41","Type":"ContainerStarted","Data":"2bbb569d9284087bb38630da294e6dcaf8eb7c78b42407afe4cdec8312f54d69"}
Sep 17 15:30:43.623600 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:43.623578    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-45zt7_2c069c9e-0856-46b0-ba10-67829c05ac94/console-operator/1.log"
Sep 17 15:30:43.623796 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:43.623772    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-45zt7" event={"ID":"2c069c9e-0856-46b0-ba10-67829c05ac94","Type":"ContainerStarted","Data":"18f79e56ce4de648048d02123fd6dcb461cd0f891287ba1c91bcc0c157fd737f"}
Sep 17 15:30:43.624128 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:43.624097    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-45zt7"
Sep 17 15:30:43.637634 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:43.637581    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-l2rxt" podStartSLOduration=65.637565653 podStartE2EDuration="1m5.637565653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:43.636249645 +0000 UTC m=+66.016267402" watchObservedRunningTime="2026-09-17 15:30:43.637565653 +0000 UTC m=+66.017583401"
Sep 17 15:30:43.656816 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:43.656752    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-45zt7" podStartSLOduration=18.43732892 podStartE2EDuration="23.65673438s" totalImagesPullingTime="5.21940546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:43.656434458 +0000 UTC m=+66.036452219" watchObservedRunningTime="2026-09-17 15:30:43.65673438 +0000 UTC m=+66.036752523"
Sep 17 15:30:43.891025 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:43.890987    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n7dnd"]
Sep 17 15:30:43.894992 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:30:43.894903    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82788049_f54a_43a9_903a_f59be68b859a.slice/crio-ad3440eeaebb6e0bc4d5da24e9cc2b2e16029ac29d6e5286b33a61d1d388cc60 WatchSource:0}: Error finding container ad3440eeaebb6e0bc4d5da24e9cc2b2e16029ac29d6e5286b33a61d1d388cc60: Status 404 returned error can't find the container with id ad3440eeaebb6e0bc4d5da24e9cc2b2e16029ac29d6e5286b33a61d1d388cc60
Sep 17 15:30:44.487658 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:44.487442    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-45zt7"
Sep 17 15:30:44.635269 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:44.635137    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n7dnd" event={"ID":"82788049-f54a-43a9-903a-f59be68b859a","Type":"ContainerStarted","Data":"09fe3c6352e7085010ed3e547604392ae32167b2bcd7e3640832f00c441d0d98"}
Sep 17 15:30:44.635269 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:44.635221    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n7dnd" event={"ID":"82788049-f54a-43a9-903a-f59be68b859a","Type":"ContainerStarted","Data":"ad3440eeaebb6e0bc4d5da24e9cc2b2e16029ac29d6e5286b33a61d1d388cc60"}
Sep 17 15:30:45.640874 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:45.640835    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n7dnd" event={"ID":"82788049-f54a-43a9-903a-f59be68b859a","Type":"ContainerStarted","Data":"6ed1c637f53b6b518ad10c4f50887455de907cee568d8129f21b25a856cd1536"}
Sep 17 15:30:45.643137 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:45.643110    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hn89v" event={"ID":"eae675d6-96cb-4843-9deb-31182313d8ac","Type":"ContainerStarted","Data":"9aab310c3f287bd015cf216f04d6636e7ebba1f5436f91f4a28f496d7d2f963b"}
Sep 17 15:30:45.643277 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:45.643160    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hn89v" event={"ID":"eae675d6-96cb-4843-9deb-31182313d8ac","Type":"ContainerStarted","Data":"b838f1722691f3492ccb8b294d6a24432e7dbb5ce7790f6b7734a27e7978c0d6"}
Sep 17 15:30:45.645390 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:45.645352    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k7kbh" event={"ID":"de6d8a3f-074e-4f18-9650-0ca1cb32100f","Type":"ContainerStarted","Data":"73c6d20e71168365ff2a5f7ad3be30bdb641fdad9d7566cdab41b4f28ef82c5c"}
Sep 17 15:30:45.645524 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:45.645407    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k7kbh" event={"ID":"de6d8a3f-074e-4f18-9650-0ca1cb32100f","Type":"ContainerStarted","Data":"adb9298ca4016caa5d7b78f0c4cd3104125cd16e4fd793a92f9535a85ffbbcc3"}
Sep 17 15:30:45.660698 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:45.660639    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hn89v" podStartSLOduration=66.067149986 podStartE2EDuration="1m7.660622484s" totalImagesPullingTime="1.593472498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:45.659801513 +0000 UTC m=+68.039819269" watchObservedRunningTime="2026-09-17 15:30:45.660622484 +0000 UTC m=+68.040640239"
Sep 17 15:30:45.676408 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:45.676325    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-k7kbh" podStartSLOduration=33.938509632 podStartE2EDuration="36.676306985s" totalImagesPullingTime="2.737797353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:45.675680525 +0000 UTC m=+68.055698294" watchObservedRunningTime="2026-09-17 15:30:45.676306985 +0000 UTC m=+68.056324743"
Sep 17 15:30:46.650822 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:46.650758    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-k7kbh"
Sep 17 15:30:48.661478 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:48.661440    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r5jgb" event={"ID":"84728120-5cde-4b79-944d-7554db40cf41","Type":"ContainerStarted","Data":"11cc1d2b2b4dc4f096c3c2ac46a539e553fda8ffd74cb600148ca5bd5f0bcd2f"}
Sep 17 15:30:48.663604 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:48.663576    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n7dnd" event={"ID":"82788049-f54a-43a9-903a-f59be68b859a","Type":"ContainerStarted","Data":"2f2cbdc1c50bad2176f5b629fe319f82a9eb71f44f4a784f870c81a68d4d9827"}
Sep 17 15:30:48.676145 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:48.676081    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-r5jgb" podStartSLOduration=65.660164954 podStartE2EDuration="1m10.676047257s" totalImagesPullingTime="5.015882303s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:48.675771244 +0000 UTC m=+71.055789002" watchObservedRunningTime="2026-09-17 15:30:48.676047257 +0000 UTC m=+71.056065015"
Sep 17 15:30:48.693500 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:48.693448    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-n7dnd" podStartSLOduration=17.562598326 podStartE2EDuration="21.693430537s" totalImagesPullingTime="4.130832211s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:30:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:48.692460007 +0000 UTC m=+71.072477764" watchObservedRunningTime="2026-09-17 15:30:48.693430537 +0000 UTC m=+71.073448291"
Sep 17 15:30:51.838076 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.838040    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-65bdt"]
Sep 17 15:30:51.840451 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.840427    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-65bdt"
Sep 17 15:30:51.841724 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.841697    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bdbb6cf68-4wxvj"]
Sep 17 15:30:51.843584 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.843568    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bdbb6cf68-4wxvj"
Sep 17 15:30:51.844204 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.844185    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-kcpbr\""
Sep 17 15:30:51.844358 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.844343    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 15:30:51.844497 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.844482    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 15:30:51.846124 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.846110    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 15:30:51.846196 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.846129    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 15:30:51.846884 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.846867    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 15:30:51.847609 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.847590    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 15:30:51.858864 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.858843    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8648ff86b4-6rbwd"]
Sep 17 15:30:51.861051 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.861033    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8648ff86b4-6rbwd"
Sep 17 15:30:51.862631 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.862614    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6876f7bf84-s47zp"]
Sep 17 15:30:51.863942 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.863926    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-q95zh\""
Sep 17 15:30:51.864218 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.864202    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 15:30:51.864764 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.864744    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6876f7bf84-s47zp"
Sep 17 15:30:51.871003 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.869989    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 15:30:51.871003 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.870289    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 15:30:51.871003 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.870643    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 15:30:51.871003 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.870836    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 15:30:51.871500 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.871483    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bdbb6cf68-4wxvj"]
Sep 17 15:30:51.872695 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.872667    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-65bdt"]
Sep 17 15:30:51.880027 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.879963    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8648ff86b4-6rbwd"]
Sep 17 15:30:51.881892 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.881866    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6876f7bf84-s47zp"]
Sep 17 15:30:51.946641 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.946612    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-5rsgl"]
Sep 17 15:30:51.948896 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.948880    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-5rsgl"
Sep 17 15:30:51.951243 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.951215    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 15:30:51.951243 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.951232    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 15:30:51.951583 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.951567    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-74mb6\""
Sep 17 15:30:51.969578 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.969555    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/723ac148-79f4-4dbd-8324-faa3a81ac318-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6876f7bf84-s47zp\" (UID: \"723ac148-79f4-4dbd-8324-faa3a81ac318\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6876f7bf84-s47zp"
Sep 17 15:30:51.969700 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.969589    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ss6n\" (UniqueName: \"kubernetes.io/projected/c353dc2a-c345-4f5e-8f51-2fa024651665-kube-api-access-5ss6n\") pod \"klusterlet-addon-workmgr-6bdbb6cf68-4wxvj\" (UID: \"c353dc2a-c345-4f5e-8f51-2fa024651665\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bdbb6cf68-4wxvj"
Sep 17 15:30:51.969700 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.969619    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8097c742-7087-4432-b73b-d3c369735e3f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8648ff86b4-6rbwd\" (UID: \"8097c742-7087-4432-b73b-d3c369735e3f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8648ff86b4-6rbwd"
Sep 17 15:30:51.969700 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.969677    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/723ac148-79f4-4dbd-8324-faa3a81ac318-ca\") pod \"cluster-proxy-proxy-agent-6876f7bf84-s47zp\" (UID: \"723ac148-79f4-4dbd-8324-faa3a81ac318\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6876f7bf84-s47zp"
Sep 17 15:30:51.969827 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.969751    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4knx\" (UniqueName: \"kubernetes.io/projected/723ac148-79f4-4dbd-8324-faa3a81ac318-kube-api-access-r4knx\") pod \"cluster-proxy-proxy-agent-6876f7bf84-s47zp\" (UID: \"723ac148-79f4-4dbd-8324-faa3a81ac318\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6876f7bf84-s47zp"
Sep 17 15:30:51.969827 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.969784    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/12788045-cf19-4320-8007-927007d2eb7c-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-65bdt\" (UID: \"12788045-cf19-4320-8007-927007d2eb7c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-65bdt"
Sep 17 15:30:51.969827 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.969805    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8wfr\" (UniqueName: \"kubernetes.io/projected/8097c742-7087-4432-b73b-d3c369735e3f-kube-api-access-x8wfr\") pod \"managed-serviceaccount-addon-agent-8648ff86b4-6rbwd\" (UID: \"8097c742-7087-4432-b73b-d3c369735e3f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8648ff86b4-6rbwd"
Sep 17 15:30:51.969827 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.969824    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c353dc2a-c345-4f5e-8f51-2fa024651665-tmp\") pod \"klusterlet-addon-workmgr-6bdbb6cf68-4wxvj\" (UID: \"c353dc2a-c345-4f5e-8f51-2fa024651665\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bdbb6cf68-4wxvj"
Sep 17 15:30:51.969990 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.969841    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/723ac148-79f4-4dbd-8324-faa3a81ac318-hub\") pod \"cluster-proxy-proxy-agent-6876f7bf84-s47zp\" (UID: \"723ac148-79f4-4dbd-8324-faa3a81ac318\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6876f7bf84-s47zp"
Sep 17 15:30:51.969990 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.969860    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12788045-cf19-4320-8007-927007d2eb7c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-65bdt\" (UID: \"12788045-cf19-4320-8007-927007d2eb7c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-65bdt"
Sep 17 15:30:51.969990 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.969889    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/723ac148-79f4-4dbd-8324-faa3a81ac318-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6876f7bf84-s47zp\" (UID: \"723ac148-79f4-4dbd-8324-faa3a81ac318\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6876f7bf84-s47zp"
Sep 17 15:30:51.969990 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.969910    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/723ac148-79f4-4dbd-8324-faa3a81ac318-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6876f7bf84-s47zp\" (UID: \"723ac148-79f4-4dbd-8324-faa3a81ac318\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6876f7bf84-s47zp"
Sep 17 15:30:51.969990 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.969940    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c353dc2a-c345-4f5e-8f51-2fa024651665-klusterlet-config\") pod \"klusterlet-addon-workmgr-6bdbb6cf68-4wxvj\" (UID: \"c353dc2a-c345-4f5e-8f51-2fa024651665\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bdbb6cf68-4wxvj"
Sep 17 15:30:51.974952 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:51.974928    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-5rsgl"]
Sep 17 15:30:52.071214 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.071183    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/723ac148-79f4-4dbd-8324-faa3a81ac318-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6876f7bf84-s47zp\" (UID: \"723ac148-79f4-4dbd-8324-faa3a81ac318\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6876f7bf84-s47zp"
Sep 17 15:30:52.071214 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.071223    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8107c705-1f01-4194-bcde-679259e3ea51-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-t58k8\" (UID: \"8107c705-1f01-4194-bcde-679259e3ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-t58k8"
Sep 17 15:30:52.071502 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.071246    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c353dc2a-c345-4f5e-8f51-2fa024651665-klusterlet-config\") pod \"klusterlet-addon-workmgr-6bdbb6cf68-4wxvj\" (UID: \"c353dc2a-c345-4f5e-8f51-2fa024651665\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bdbb6cf68-4wxvj"
Sep 17 15:30:52.071502 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.071266    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/723ac148-79f4-4dbd-8324-faa3a81ac318-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6876f7bf84-s47zp\" (UID: \"723ac148-79f4-4dbd-8324-faa3a81ac318\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6876f7bf84-s47zp"
Sep 17 15:30:52.071502 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.071302    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ss6n\" (UniqueName: \"kubernetes.io/projected/c353dc2a-c345-4f5e-8f51-2fa024651665-kube-api-access-5ss6n\") pod \"klusterlet-addon-workmgr-6bdbb6cf68-4wxvj\" (UID: \"c353dc2a-c345-4f5e-8f51-2fa024651665\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bdbb6cf68-4wxvj"
Sep 17 15:30:52.071654 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.071552    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8097c742-7087-4432-b73b-d3c369735e3f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8648ff86b4-6rbwd\" (UID: \"8097c742-7087-4432-b73b-d3c369735e3f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8648ff86b4-6rbwd"
Sep 17 15:30:52.071654 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.071604    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b73fab2-e530-4b68-acf0-c71d1943cb7c-metrics-certs\") pod \"router-default-85cf78879b-z2qnj\" (UID: \"8b73fab2-e530-4b68-acf0-c71d1943cb7c\") " pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:52.071654 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.071637    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b73fab2-e530-4b68-acf0-c71d1943cb7c-service-ca-bundle\") pod \"router-default-85cf78879b-z2qnj\" (UID: \"8b73fab2-e530-4b68-acf0-c71d1943cb7c\") " pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:52.071810 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.071672    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/723ac148-79f4-4dbd-8324-faa3a81ac318-ca\") pod \"cluster-proxy-proxy-agent-6876f7bf84-s47zp\" (UID: \"723ac148-79f4-4dbd-8324-faa3a81ac318\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6876f7bf84-s47zp"
Sep 17 15:30:52.071810 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.071727    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxfq4\" (UniqueName: \"kubernetes.io/projected/3baff753-aee2-4eb4-a85f-4d01a2deda3d-kube-api-access-nxfq4\") pod \"downloads-76f99584f4-5rsgl\" (UID: \"3baff753-aee2-4eb4-a85f-4d01a2deda3d\") " pod="openshift-console/downloads-76f99584f4-5rsgl"
Sep 17 15:30:52.071810 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.071766    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4knx\" (UniqueName: \"kubernetes.io/projected/723ac148-79f4-4dbd-8324-faa3a81ac318-kube-api-access-r4knx\") pod \"cluster-proxy-proxy-agent-6876f7bf84-s47zp\" (UID: \"723ac148-79f4-4dbd-8324-faa3a81ac318\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6876f7bf84-s47zp"
Sep 17 15:30:52.071951 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.071808    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/12788045-cf19-4320-8007-927007d2eb7c-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-65bdt\" (UID: \"12788045-cf19-4320-8007-927007d2eb7c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-65bdt"
Sep 17 15:30:52.071951 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.071839    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8wfr\" (UniqueName: \"kubernetes.io/projected/8097c742-7087-4432-b73b-d3c369735e3f-kube-api-access-x8wfr\") pod \"managed-serviceaccount-addon-agent-8648ff86b4-6rbwd\" (UID: \"8097c742-7087-4432-b73b-d3c369735e3f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8648ff86b4-6rbwd"
Sep 17 15:30:52.071951 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.071875    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c353dc2a-c345-4f5e-8f51-2fa024651665-tmp\") pod \"klusterlet-addon-workmgr-6bdbb6cf68-4wxvj\" (UID: \"c353dc2a-c345-4f5e-8f51-2fa024651665\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bdbb6cf68-4wxvj"
Sep 17 15:30:52.071951 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.071900    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/723ac148-79f4-4dbd-8324-faa3a81ac318-hub\") pod \"cluster-proxy-proxy-agent-6876f7bf84-s47zp\" (UID: \"723ac148-79f4-4dbd-8324-faa3a81ac318\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6876f7bf84-s47zp"
Sep 17 15:30:52.071951 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.071923    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12788045-cf19-4320-8007-927007d2eb7c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-65bdt\" (UID: \"12788045-cf19-4320-8007-927007d2eb7c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-65bdt"
Sep 17 15:30:52.072279 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.071980    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/723ac148-79f4-4dbd-8324-faa3a81ac318-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6876f7bf84-s47zp\" (UID: \"723ac148-79f4-4dbd-8324-faa3a81ac318\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6876f7bf84-s47zp"
Sep 17 15:30:52.072407 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.072383    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c353dc2a-c345-4f5e-8f51-2fa024651665-tmp\") pod \"klusterlet-addon-workmgr-6bdbb6cf68-4wxvj\" (UID: \"c353dc2a-c345-4f5e-8f51-2fa024651665\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bdbb6cf68-4wxvj"
Sep 17 15:30:52.072842 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.072816    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/723ac148-79f4-4dbd-8324-faa3a81ac318-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6876f7bf84-s47zp\" (UID: \"723ac148-79f4-4dbd-8324-faa3a81ac318\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6876f7bf84-s47zp"
Sep 17 15:30:52.072952 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.072926    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b73fab2-e530-4b68-acf0-c71d1943cb7c-service-ca-bundle\") pod \"router-default-85cf78879b-z2qnj\" (UID: \"8b73fab2-e530-4b68-acf0-c71d1943cb7c\") " pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:52.073093 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.073051    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/12788045-cf19-4320-8007-927007d2eb7c-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-65bdt\" (UID: \"12788045-cf19-4320-8007-927007d2eb7c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-65bdt"
Sep 17 15:30:52.075056 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.075012    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/723ac148-79f4-4dbd-8324-faa3a81ac318-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6876f7bf84-s47zp\" (UID: \"723ac148-79f4-4dbd-8324-faa3a81ac318\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6876f7bf84-s47zp"
Sep 17 15:30:52.075184 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.075109    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8107c705-1f01-4194-bcde-679259e3ea51-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-t58k8\" (UID: \"8107c705-1f01-4194-bcde-679259e3ea51\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-t58k8"
Sep 17 15:30:52.075883 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.075598    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c353dc2a-c345-4f5e-8f51-2fa024651665-klusterlet-config\") pod \"klusterlet-addon-workmgr-6bdbb6cf68-4wxvj\" (UID: \"c353dc2a-c345-4f5e-8f51-2fa024651665\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bdbb6cf68-4wxvj"
Sep 17 15:30:52.075883 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.075606    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/723ac148-79f4-4dbd-8324-faa3a81ac318-hub\") pod \"cluster-proxy-proxy-agent-6876f7bf84-s47zp\" (UID: \"723ac148-79f4-4dbd-8324-faa3a81ac318\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6876f7bf84-s47zp"
Sep 17 15:30:52.076051 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.076039    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8097c742-7087-4432-b73b-d3c369735e3f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8648ff86b4-6rbwd\" (UID: \"8097c742-7087-4432-b73b-d3c369735e3f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8648ff86b4-6rbwd"
Sep 17 15:30:52.076139 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.076055    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/723ac148-79f4-4dbd-8324-faa3a81ac318-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6876f7bf84-s47zp\" (UID: \"723ac148-79f4-4dbd-8324-faa3a81ac318\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6876f7bf84-s47zp"
Sep 17 15:30:52.076139 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.076039    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/12788045-cf19-4320-8007-927007d2eb7c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-65bdt\" (UID: \"12788045-cf19-4320-8007-927007d2eb7c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-65bdt"
Sep 17 15:30:52.076139 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.076126    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b73fab2-e530-4b68-acf0-c71d1943cb7c-metrics-certs\") pod \"router-default-85cf78879b-z2qnj\" (UID: \"8b73fab2-e530-4b68-acf0-c71d1943cb7c\") " pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:52.076604 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.076587    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/723ac148-79f4-4dbd-8324-faa3a81ac318-ca\") pod \"cluster-proxy-proxy-agent-6876f7bf84-s47zp\" (UID: \"723ac148-79f4-4dbd-8324-faa3a81ac318\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6876f7bf84-s47zp"
Sep 17 15:30:52.082903 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.082884    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ss6n\" (UniqueName: \"kubernetes.io/projected/c353dc2a-c345-4f5e-8f51-2fa024651665-kube-api-access-5ss6n\") pod \"klusterlet-addon-workmgr-6bdbb6cf68-4wxvj\" (UID: \"c353dc2a-c345-4f5e-8f51-2fa024651665\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bdbb6cf68-4wxvj"
Sep 17 15:30:52.086535 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.086513    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4knx\" (UniqueName: \"kubernetes.io/projected/723ac148-79f4-4dbd-8324-faa3a81ac318-kube-api-access-r4knx\") pod \"cluster-proxy-proxy-agent-6876f7bf84-s47zp\" (UID: \"723ac148-79f4-4dbd-8324-faa3a81ac318\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6876f7bf84-s47zp"
Sep 17 15:30:52.088455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.088411    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8wfr\" (UniqueName: \"kubernetes.io/projected/8097c742-7087-4432-b73b-d3c369735e3f-kube-api-access-x8wfr\") pod \"managed-serviceaccount-addon-agent-8648ff86b4-6rbwd\" (UID: \"8097c742-7087-4432-b73b-d3c369735e3f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8648ff86b4-6rbwd"
Sep 17 15:30:52.151186 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.151149    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-65bdt"
Sep 17 15:30:52.155995 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.155974    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bdbb6cf68-4wxvj"
Sep 17 15:30:52.173346 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.173299    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxfq4\" (UniqueName: \"kubernetes.io/projected/3baff753-aee2-4eb4-a85f-4d01a2deda3d-kube-api-access-nxfq4\") pod \"downloads-76f99584f4-5rsgl\" (UID: \"3baff753-aee2-4eb4-a85f-4d01a2deda3d\") " pod="openshift-console/downloads-76f99584f4-5rsgl"
Sep 17 15:30:52.181290 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.181261    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8648ff86b4-6rbwd"
Sep 17 15:30:52.185917 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.185889    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxfq4\" (UniqueName: \"kubernetes.io/projected/3baff753-aee2-4eb4-a85f-4d01a2deda3d-kube-api-access-nxfq4\") pod \"downloads-76f99584f4-5rsgl\" (UID: \"3baff753-aee2-4eb4-a85f-4d01a2deda3d\") " pod="openshift-console/downloads-76f99584f4-5rsgl"
Sep 17 15:30:52.188785 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.188716    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6876f7bf84-s47zp"
Sep 17 15:30:52.217622 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.217543    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-nvj49\""
Sep 17 15:30:52.225296 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.225263    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-t58k8"
Sep 17 15:30:52.269097 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.268793    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wz88k\""
Sep 17 15:30:52.269809 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.269475    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-5rsgl"
Sep 17 15:30:52.275969 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.275946    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:52.434201 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.434157    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bdbb6cf68-4wxvj"]
Sep 17 15:30:52.436326 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.436295    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-65bdt"]
Sep 17 15:30:52.437409 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:30:52.437385    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc353dc2a_c345_4f5e_8f51_2fa024651665.slice/crio-553fd45b5d73dd7714d5695fdd18e324cdf12ff3f92e048bca3d3e2faa28db76 WatchSource:0}: Error finding container 553fd45b5d73dd7714d5695fdd18e324cdf12ff3f92e048bca3d3e2faa28db76: Status 404 returned error can't find the container with id 553fd45b5d73dd7714d5695fdd18e324cdf12ff3f92e048bca3d3e2faa28db76
Sep 17 15:30:52.438762 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:30:52.438740    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12788045_cf19_4320_8007_927007d2eb7c.slice/crio-b9474fc53299c5a0d0e620c1a5f947ac73149dc780079e713bc30a5ee1cc2b22 WatchSource:0}: Error finding container b9474fc53299c5a0d0e620c1a5f947ac73149dc780079e713bc30a5ee1cc2b22: Status 404 returned error can't find the container with id b9474fc53299c5a0d0e620c1a5f947ac73149dc780079e713bc30a5ee1cc2b22
Sep 17 15:30:52.492179 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.492146    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8648ff86b4-6rbwd"]
Sep 17 15:30:52.495460 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:30:52.495431    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8097c742_7087_4432_b73b_d3c369735e3f.slice/crio-81bb7fcf329dd59872c7090bbba2b090d2729fed493a37c4b10d52c6cdc20b2c WatchSource:0}: Error finding container 81bb7fcf329dd59872c7090bbba2b090d2729fed493a37c4b10d52c6cdc20b2c: Status 404 returned error can't find the container with id 81bb7fcf329dd59872c7090bbba2b090d2729fed493a37c4b10d52c6cdc20b2c
Sep 17 15:30:52.499646 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.499623    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6876f7bf84-s47zp"]
Sep 17 15:30:52.501432 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:30:52.501409    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod723ac148_79f4_4dbd_8324_faa3a81ac318.slice/crio-985610d78089f69bf9b9bce19eb510d164f94e3192da5149b2b142f23ed5a672 WatchSource:0}: Error finding container 985610d78089f69bf9b9bce19eb510d164f94e3192da5149b2b142f23ed5a672: Status 404 returned error can't find the container with id 985610d78089f69bf9b9bce19eb510d164f94e3192da5149b2b142f23ed5a672
Sep 17 15:30:52.571246 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.571221    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-t58k8"]
Sep 17 15:30:52.574218 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:30:52.574187    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8107c705_1f01_4194_bcde_679259e3ea51.slice/crio-d46872fba5865562a1dd87b27b11f45e5d4a7c42a5bd7b0a338e34e5ba67fb5c WatchSource:0}: Error finding container d46872fba5865562a1dd87b27b11f45e5d4a7c42a5bd7b0a338e34e5ba67fb5c: Status 404 returned error can't find the container with id d46872fba5865562a1dd87b27b11f45e5d4a7c42a5bd7b0a338e34e5ba67fb5c
Sep 17 15:30:52.588725 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.588693    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-5rsgl"]
Sep 17 15:30:52.592223 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:30:52.592196    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3baff753_aee2_4eb4_a85f_4d01a2deda3d.slice/crio-4d632c707a2c6b775edb29d0d0b5f7b11236b00dfb932a681c95075a2d085d3d WatchSource:0}: Error finding container 4d632c707a2c6b775edb29d0d0b5f7b11236b00dfb932a681c95075a2d085d3d: Status 404 returned error can't find the container with id 4d632c707a2c6b775edb29d0d0b5f7b11236b00dfb932a681c95075a2d085d3d
Sep 17 15:30:52.596783 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.596759    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-85cf78879b-z2qnj"]
Sep 17 15:30:52.599621 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:30:52.599590    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b73fab2_e530_4b68_acf0_c71d1943cb7c.slice/crio-79916b33e2c745fcd28402b2784a8dafc1c9e1279ffc9ad18384f4579d3e8eb2 WatchSource:0}: Error finding container 79916b33e2c745fcd28402b2784a8dafc1c9e1279ffc9ad18384f4579d3e8eb2: Status 404 returned error can't find the container with id 79916b33e2c745fcd28402b2784a8dafc1c9e1279ffc9ad18384f4579d3e8eb2
Sep 17 15:30:52.685886 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.685849    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-5rsgl" event={"ID":"3baff753-aee2-4eb4-a85f-4d01a2deda3d","Type":"ContainerStarted","Data":"4d632c707a2c6b775edb29d0d0b5f7b11236b00dfb932a681c95075a2d085d3d"}
Sep 17 15:30:52.687601 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.687568    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-85cf78879b-z2qnj" event={"ID":"8b73fab2-e530-4b68-acf0-c71d1943cb7c","Type":"ContainerStarted","Data":"d88a86d34ac9079d18fa4b0a726dbc337e8ff2182e510e2c7063eab07666ff1b"}
Sep 17 15:30:52.687753 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.687619    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-85cf78879b-z2qnj" event={"ID":"8b73fab2-e530-4b68-acf0-c71d1943cb7c","Type":"ContainerStarted","Data":"79916b33e2c745fcd28402b2784a8dafc1c9e1279ffc9ad18384f4579d3e8eb2"}
Sep 17 15:30:52.688860 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.688830    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8648ff86b4-6rbwd" event={"ID":"8097c742-7087-4432-b73b-d3c369735e3f","Type":"ContainerStarted","Data":"81bb7fcf329dd59872c7090bbba2b090d2729fed493a37c4b10d52c6cdc20b2c"}
Sep 17 15:30:52.690036 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.690010    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6876f7bf84-s47zp" event={"ID":"723ac148-79f4-4dbd-8324-faa3a81ac318","Type":"ContainerStarted","Data":"985610d78089f69bf9b9bce19eb510d164f94e3192da5149b2b142f23ed5a672"}
Sep 17 15:30:52.691244 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.691224    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-65bdt" event={"ID":"12788045-cf19-4320-8007-927007d2eb7c","Type":"ContainerStarted","Data":"b9474fc53299c5a0d0e620c1a5f947ac73149dc780079e713bc30a5ee1cc2b22"}
Sep 17 15:30:52.692627 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.692607    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-t58k8" event={"ID":"8107c705-1f01-4194-bcde-679259e3ea51","Type":"ContainerStarted","Data":"d46872fba5865562a1dd87b27b11f45e5d4a7c42a5bd7b0a338e34e5ba67fb5c"}
Sep 17 15:30:52.693822 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.693803    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bdbb6cf68-4wxvj" event={"ID":"c353dc2a-c345-4f5e-8f51-2fa024651665","Type":"ContainerStarted","Data":"553fd45b5d73dd7714d5695fdd18e324cdf12ff3f92e048bca3d3e2faa28db76"}
Sep 17 15:30:52.708474 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:52.708415    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-85cf78879b-z2qnj" podStartSLOduration=32.708395073 podStartE2EDuration="32.708395073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:52.707013772 +0000 UTC m=+75.087031535" watchObservedRunningTime="2026-09-17 15:30:52.708395073 +0000 UTC m=+75.088412831"
Sep 17 15:30:53.277536 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:53.277490    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:53.281285 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:53.281257    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:53.702768 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:53.702570    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:53.704004 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:53.703977    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-85cf78879b-z2qnj"
Sep 17 15:30:56.659315 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:30:56.659271    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-k7kbh"
Sep 17 15:31:01.744026 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:01.743987    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8648ff86b4-6rbwd" event={"ID":"8097c742-7087-4432-b73b-d3c369735e3f","Type":"ContainerStarted","Data":"696afa9ecad4dc28271067bd45a2290731eb63c6205983907b999a230c40f921"}
Sep 17 15:31:01.746346 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:01.746315    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6876f7bf84-s47zp" event={"ID":"723ac148-79f4-4dbd-8324-faa3a81ac318","Type":"ContainerStarted","Data":"b84bf61e3b431660eacad9ce89a0a09ad901fa441580cc0a0d685ab3a7722b53"}
Sep 17 15:31:01.746501 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:01.746359    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6876f7bf84-s47zp" event={"ID":"723ac148-79f4-4dbd-8324-faa3a81ac318","Type":"ContainerStarted","Data":"5020a894885c322c97fb0e8a46c9bd8602a0273044a9e9529fc97257565693a6"}
Sep 17 15:31:01.746501 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:01.746378    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6876f7bf84-s47zp" event={"ID":"723ac148-79f4-4dbd-8324-faa3a81ac318","Type":"ContainerStarted","Data":"1d5ae357c07d46020c984230e61d44d9025b15ac7cebe7cc19aa52fc1c749e62"}
Sep 17 15:31:01.748269 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:01.748238    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-65bdt" event={"ID":"12788045-cf19-4320-8007-927007d2eb7c","Type":"ContainerStarted","Data":"24044f83e9030d9edfb529a454c091fec0695076a85687af699ed95658e19ce0"}
Sep 17 15:31:01.749785 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:01.749759    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-t58k8" event={"ID":"8107c705-1f01-4194-bcde-679259e3ea51","Type":"ContainerStarted","Data":"d1d4c7045094888749aa9ddfd3084a7769a364848853ebdea91b9cb166a7e983"}
Sep 17 15:31:01.751179 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:01.751159    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bdbb6cf68-4wxvj" event={"ID":"c353dc2a-c345-4f5e-8f51-2fa024651665","Type":"ContainerStarted","Data":"174185c4def3ff6553b88208bfda447a0a6cc50f882fa3451624c18a23c7fc94"}
Sep 17 15:31:01.751364 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:01.751348    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bdbb6cf68-4wxvj"
Sep 17 15:31:01.753197 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:01.753176    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bdbb6cf68-4wxvj"
Sep 17 15:31:01.780495 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:01.780422    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8648ff86b4-6rbwd" podStartSLOduration=2.5917894500000003 podStartE2EDuration="10.780398384s" totalImagesPullingTime="8.188608934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:01.76220885 +0000 UTC m=+84.142226602" watchObservedRunningTime="2026-09-17 15:31:01.780398384 +0000 UTC m=+84.160416141"
Sep 17 15:31:01.780940 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:01.780902    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-65bdt" podStartSLOduration=2.567063411 podStartE2EDuration="10.780891553s" totalImagesPullingTime="8.213828142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:01.779813205 +0000 UTC m=+84.159830964" watchObservedRunningTime="2026-09-17 15:31:01.780891553 +0000 UTC m=+84.160909308"
Sep 17 15:31:01.798690 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:01.798625    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bdbb6cf68-4wxvj" podStartSLOduration=2.5519515139999998 podStartE2EDuration="10.798603776s" totalImagesPullingTime="8.246652262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:01.795848612 +0000 UTC m=+84.175866370" watchObservedRunningTime="2026-09-17 15:31:01.798603776 +0000 UTC m=+84.178621532"
Sep 17 15:31:01.816820 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:01.816756    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6876f7bf84-s47zp" podStartSLOduration=2.6168354320000002 podStartE2EDuration="10.816736716s" totalImagesPullingTime="8.199901284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:01.816190395 +0000 UTC m=+84.196208152" watchObservedRunningTime="2026-09-17 15:31:01.816736716 +0000 UTC m=+84.196754472"
Sep 17 15:31:01.843160 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:01.843097    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-t58k8" podStartSLOduration=33.760476598 podStartE2EDuration="41.843054272s" totalImagesPullingTime="8.082577674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:01.842584796 +0000 UTC m=+84.222602552" watchObservedRunningTime="2026-09-17 15:31:01.843054272 +0000 UTC m=+84.223072030"
Sep 17 15:31:02.113418 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:02.113362    2556 patch_prober.go:28] interesting pod/image-registry-69d6bcf58c-bq4rc container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 15:31:02.113607 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:02.113430    2556 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc" podUID="2581da35-8109-4df3-a553-c8992a31b8d1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:31:03.630576 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:03.630542    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:31:09.732422 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:09.732379    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kprs2"]
Sep 17 15:31:09.768619 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:09.768587    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:09.772740 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:09.772681    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 15:31:09.774269 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:09.774013    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 15:31:09.774844 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:09.774474    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 15:31:09.774844 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:09.774696    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jw4fs\""
Sep 17 15:31:09.781558 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:09.781536    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 15:31:09.939508 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:09.939342    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ae480d7-2e41-41e3-9923-656e5e9e82ae-sys\") pod \"node-exporter-kprs2\" (UID: \"9ae480d7-2e41-41e3-9923-656e5e9e82ae\") " pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:09.939508 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:09.939391    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ae480d7-2e41-41e3-9923-656e5e9e82ae-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kprs2\" (UID: \"9ae480d7-2e41-41e3-9923-656e5e9e82ae\") " pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:09.939508 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:09.939430    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9ae480d7-2e41-41e3-9923-656e5e9e82ae-node-exporter-textfile\") pod \"node-exporter-kprs2\" (UID: \"9ae480d7-2e41-41e3-9923-656e5e9e82ae\") " pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:09.939508 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:09.939460    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9ae480d7-2e41-41e3-9923-656e5e9e82ae-node-exporter-accelerators-collector-config\") pod \"node-exporter-kprs2\" (UID: \"9ae480d7-2e41-41e3-9923-656e5e9e82ae\") " pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:09.940021 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:09.939566    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ae480d7-2e41-41e3-9923-656e5e9e82ae-metrics-client-ca\") pod \"node-exporter-kprs2\" (UID: \"9ae480d7-2e41-41e3-9923-656e5e9e82ae\") " pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:09.940021 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:09.939634    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9ae480d7-2e41-41e3-9923-656e5e9e82ae-root\") pod \"node-exporter-kprs2\" (UID: \"9ae480d7-2e41-41e3-9923-656e5e9e82ae\") " pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:09.940021 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:09.939667    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9ae480d7-2e41-41e3-9923-656e5e9e82ae-node-exporter-wtmp\") pod \"node-exporter-kprs2\" (UID: \"9ae480d7-2e41-41e3-9923-656e5e9e82ae\") " pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:09.940021 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:09.939704    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9ae480d7-2e41-41e3-9923-656e5e9e82ae-node-exporter-tls\") pod \"node-exporter-kprs2\" (UID: \"9ae480d7-2e41-41e3-9923-656e5e9e82ae\") " pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:09.940021 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:09.939732    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqmct\" (UniqueName: \"kubernetes.io/projected/9ae480d7-2e41-41e3-9923-656e5e9e82ae-kube-api-access-kqmct\") pod \"node-exporter-kprs2\" (UID: \"9ae480d7-2e41-41e3-9923-656e5e9e82ae\") " pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:10.042019 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:10.041049    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9ae480d7-2e41-41e3-9923-656e5e9e82ae-node-exporter-tls\") pod \"node-exporter-kprs2\" (UID: \"9ae480d7-2e41-41e3-9923-656e5e9e82ae\") " pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:10.042019 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:10.041114    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqmct\" (UniqueName: \"kubernetes.io/projected/9ae480d7-2e41-41e3-9923-656e5e9e82ae-kube-api-access-kqmct\") pod \"node-exporter-kprs2\" (UID: \"9ae480d7-2e41-41e3-9923-656e5e9e82ae\") " pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:10.042019 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:10.041163    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ae480d7-2e41-41e3-9923-656e5e9e82ae-sys\") pod \"node-exporter-kprs2\" (UID: \"9ae480d7-2e41-41e3-9923-656e5e9e82ae\") " pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:10.042019 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:10.041190    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ae480d7-2e41-41e3-9923-656e5e9e82ae-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kprs2\" (UID: \"9ae480d7-2e41-41e3-9923-656e5e9e82ae\") " pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:10.042019 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:10.041228    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9ae480d7-2e41-41e3-9923-656e5e9e82ae-node-exporter-textfile\") pod \"node-exporter-kprs2\" (UID: \"9ae480d7-2e41-41e3-9923-656e5e9e82ae\") " pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:10.042019 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:10.041255    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9ae480d7-2e41-41e3-9923-656e5e9e82ae-node-exporter-accelerators-collector-config\") pod \"node-exporter-kprs2\" (UID: \"9ae480d7-2e41-41e3-9923-656e5e9e82ae\") " pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:10.042019 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:10.041303    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ae480d7-2e41-41e3-9923-656e5e9e82ae-metrics-client-ca\") pod \"node-exporter-kprs2\" (UID: \"9ae480d7-2e41-41e3-9923-656e5e9e82ae\") " pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:10.042019 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:10.041355    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9ae480d7-2e41-41e3-9923-656e5e9e82ae-root\") pod \"node-exporter-kprs2\" (UID: \"9ae480d7-2e41-41e3-9923-656e5e9e82ae\") " pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:10.042019 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:10.041385    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9ae480d7-2e41-41e3-9923-656e5e9e82ae-node-exporter-wtmp\") pod \"node-exporter-kprs2\" (UID: \"9ae480d7-2e41-41e3-9923-656e5e9e82ae\") " pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:10.042019 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:10.041569    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9ae480d7-2e41-41e3-9923-656e5e9e82ae-node-exporter-wtmp\") pod \"node-exporter-kprs2\" (UID: \"9ae480d7-2e41-41e3-9923-656e5e9e82ae\") " pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:10.042019 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:10.041625    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ae480d7-2e41-41e3-9923-656e5e9e82ae-sys\") pod \"node-exporter-kprs2\" (UID: \"9ae480d7-2e41-41e3-9923-656e5e9e82ae\") " pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:10.043650 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:10.042382    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9ae480d7-2e41-41e3-9923-656e5e9e82ae-node-exporter-textfile\") pod \"node-exporter-kprs2\" (UID: \"9ae480d7-2e41-41e3-9923-656e5e9e82ae\") " pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:10.043650 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:10.042454    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9ae480d7-2e41-41e3-9923-656e5e9e82ae-root\") pod \"node-exporter-kprs2\" (UID: \"9ae480d7-2e41-41e3-9923-656e5e9e82ae\") " pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:10.043650 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:10.042630    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9ae480d7-2e41-41e3-9923-656e5e9e82ae-node-exporter-accelerators-collector-config\") pod \"node-exporter-kprs2\" (UID: \"9ae480d7-2e41-41e3-9923-656e5e9e82ae\") " pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:10.043650 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:10.042923    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ae480d7-2e41-41e3-9923-656e5e9e82ae-metrics-client-ca\") pod \"node-exporter-kprs2\" (UID: \"9ae480d7-2e41-41e3-9923-656e5e9e82ae\") " pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:10.046082 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:10.045637    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9ae480d7-2e41-41e3-9923-656e5e9e82ae-node-exporter-tls\") pod \"node-exporter-kprs2\" (UID: \"9ae480d7-2e41-41e3-9923-656e5e9e82ae\") " pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:10.046399 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:10.046374    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ae480d7-2e41-41e3-9923-656e5e9e82ae-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kprs2\" (UID: \"9ae480d7-2e41-41e3-9923-656e5e9e82ae\") " pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:10.050856 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:10.050832    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqmct\" (UniqueName: \"kubernetes.io/projected/9ae480d7-2e41-41e3-9923-656e5e9e82ae-kube-api-access-kqmct\") pod \"node-exporter-kprs2\" (UID: \"9ae480d7-2e41-41e3-9923-656e5e9e82ae\") " pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:10.083318 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:10.082915    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kprs2"
Sep 17 15:31:11.455450 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:31:11.455417    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae480d7_2e41_41e3_9923_656e5e9e82ae.slice/crio-7aba0f431e293b03bb2aa49a2e77ca25b47b81c3b0ebe61c4f9365fdbb0915a4 WatchSource:0}: Error finding container 7aba0f431e293b03bb2aa49a2e77ca25b47b81c3b0ebe61c4f9365fdbb0915a4: Status 404 returned error can't find the container with id 7aba0f431e293b03bb2aa49a2e77ca25b47b81c3b0ebe61c4f9365fdbb0915a4
Sep 17 15:31:11.801437 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:11.801354    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kprs2" event={"ID":"9ae480d7-2e41-41e3-9923-656e5e9e82ae","Type":"ContainerStarted","Data":"7aba0f431e293b03bb2aa49a2e77ca25b47b81c3b0ebe61c4f9365fdbb0915a4"}
Sep 17 15:31:12.809466 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:12.809423    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-5rsgl" event={"ID":"3baff753-aee2-4eb4-a85f-4d01a2deda3d","Type":"ContainerStarted","Data":"7fd5fd0dc0a2d001a68dac9f5ba3d3da766871cbe33f0b8f328c8a54864830a2"}
Sep 17 15:31:12.810604 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:12.810575    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-5rsgl"
Sep 17 15:31:12.823773 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:12.823744    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-5rsgl"
Sep 17 15:31:12.830036 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:12.829971    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-5rsgl" podStartSLOduration=2.619720675 podStartE2EDuration="21.829922794s" totalImagesPullingTime="19.210202119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:12.829282951 +0000 UTC m=+95.209300708" watchObservedRunningTime="2026-09-17 15:31:12.829922794 +0000 UTC m=+95.209940553"
Sep 17 15:31:13.817846 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:13.817812    2556 generic.go:356] "Generic (PLEG): container finished" podID="9ae480d7-2e41-41e3-9923-656e5e9e82ae" containerID="c6403a295826d317cc476bf94609a11ca30e77f53f2c0a5f36b158b95460e174" exitCode=0
Sep 17 15:31:13.818377 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:13.817908    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kprs2" event={"ID":"9ae480d7-2e41-41e3-9923-656e5e9e82ae","Type":"ContainerDied","Data":"c6403a295826d317cc476bf94609a11ca30e77f53f2c0a5f36b158b95460e174"}
Sep 17 15:31:14.638663 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:14.638627    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-l2rxt"
Sep 17 15:31:14.826290 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:14.826250    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kprs2" event={"ID":"9ae480d7-2e41-41e3-9923-656e5e9e82ae","Type":"ContainerStarted","Data":"3c65f471cebf5d2c3ce2c79764776d52d20fc30f71d9c22ede534d64c94177b3"}
Sep 17 15:31:14.826767 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:14.826337    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kprs2" event={"ID":"9ae480d7-2e41-41e3-9923-656e5e9e82ae","Type":"ContainerStarted","Data":"73351632f7576a5c5c8062b7c2cbbc4c97ba7b6cc1a949ee37f81ffb95af465a"}
Sep 17 15:31:14.853294 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:14.853229    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kprs2" podStartSLOduration=4.429877637 podStartE2EDuration="5.85320381s" totalImagesPullingTime="1.323044324s" totalInitContainerRuntime="100.281849ms" isStatefulPod=true podCreationTimestamp="2026-09-17 15:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:14.851204868 +0000 UTC m=+97.231222625" watchObservedRunningTime="2026-09-17 15:31:14.85320381 +0000 UTC m=+97.233221567"
Sep 17 15:31:19.933451 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:19.933413    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69d6bcf58c-bq4rc"]
Sep 17 15:31:37.920393 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:37.920353    2556 generic.go:356] "Generic (PLEG): container finished" podID="dfe98000-7851-4180-b13f-bff7553a880b" containerID="1eb5fd9672adcd138e7e341d9ba811ec1b0771abc8d162c96b54c8f744e15816" exitCode=0
Sep 17 15:31:37.920903 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:37.920426    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nb2mr" event={"ID":"dfe98000-7851-4180-b13f-bff7553a880b","Type":"ContainerDied","Data":"1eb5fd9672adcd138e7e341d9ba811ec1b0771abc8d162c96b54c8f744e15816"}
Sep 17 15:31:37.920903 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:37.920808    2556 scope.go:117] "RemoveContainer" containerID="1eb5fd9672adcd138e7e341d9ba811ec1b0771abc8d162c96b54c8f744e15816"
Sep 17 15:31:37.922114 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:37.922091    2556 generic.go:356] "Generic (PLEG): container finished" podID="cdc3f7b4-7834-4853-87dc-cd68e3cdad3a" containerID="1189b46dc69c9ecdd64e18befad743f06878d82a0c3f563792b3d60a1d8d7b2f" exitCode=0
Sep 17 15:31:37.922210 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:37.922116    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-r8mp6" event={"ID":"cdc3f7b4-7834-4853-87dc-cd68e3cdad3a","Type":"ContainerDied","Data":"1189b46dc69c9ecdd64e18befad743f06878d82a0c3f563792b3d60a1d8d7b2f"}
Sep 17 15:31:37.922958 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:37.922925    2556 scope.go:117] "RemoveContainer" containerID="1189b46dc69c9ecdd64e18befad743f06878d82a0c3f563792b3d60a1d8d7b2f"
Sep 17 15:31:38.928997 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:38.928960    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-nb2mr" event={"ID":"dfe98000-7851-4180-b13f-bff7553a880b","Type":"ContainerStarted","Data":"88eb63cea1a2db2a8d7330c8780e0b2f90342d1b24460565c89878e2755a8b3f"}
Sep 17 15:31:38.930738 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:38.930714    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-r8mp6" event={"ID":"cdc3f7b4-7834-4853-87dc-cd68e3cdad3a","Type":"ContainerStarted","Data":"980a4b8c7b4cbc794572290a374b06e59636eb30afe8053c3ec7eb4670eabc4e"}
Sep 17 15:31:44.965054 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:44.964986    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc" podUID="2581da35-8109-4df3-a553-c8992a31b8d1" containerName="registry" containerID="cri-o://7076e2af0305555d99c8810b58b12283cd6caa0d052715734b03a4c34179437c" gracePeriod=30
Sep 17 15:31:45.236533 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.236457    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:31:45.357630 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.357594    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-registry-tls\") pod \"2581da35-8109-4df3-a553-c8992a31b8d1\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") "
Sep 17 15:31:45.357823 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.357650    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2581da35-8109-4df3-a553-c8992a31b8d1-installation-pull-secrets\") pod \"2581da35-8109-4df3-a553-c8992a31b8d1\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") "
Sep 17 15:31:45.357823 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.357680    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjzgp\" (UniqueName: \"kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-kube-api-access-wjzgp\") pod \"2581da35-8109-4df3-a553-c8992a31b8d1\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") "
Sep 17 15:31:45.357823 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.357698    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2581da35-8109-4df3-a553-c8992a31b8d1-ca-trust-extracted\") pod \"2581da35-8109-4df3-a553-c8992a31b8d1\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") "
Sep 17 15:31:45.357823 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.357751    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2581da35-8109-4df3-a553-c8992a31b8d1-registry-certificates\") pod \"2581da35-8109-4df3-a553-c8992a31b8d1\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") "
Sep 17 15:31:45.357823 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.357773    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2581da35-8109-4df3-a553-c8992a31b8d1-trusted-ca\") pod \"2581da35-8109-4df3-a553-c8992a31b8d1\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") "
Sep 17 15:31:45.357823 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.357802    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2581da35-8109-4df3-a553-c8992a31b8d1-image-registry-private-configuration\") pod \"2581da35-8109-4df3-a553-c8992a31b8d1\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") "
Sep 17 15:31:45.358134 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.357838    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-bound-sa-token\") pod \"2581da35-8109-4df3-a553-c8992a31b8d1\" (UID: \"2581da35-8109-4df3-a553-c8992a31b8d1\") "
Sep 17 15:31:45.358568 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.358497    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2581da35-8109-4df3-a553-c8992a31b8d1-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2581da35-8109-4df3-a553-c8992a31b8d1" (UID: "2581da35-8109-4df3-a553-c8992a31b8d1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:31:45.358568 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.358551    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2581da35-8109-4df3-a553-c8992a31b8d1-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2581da35-8109-4df3-a553-c8992a31b8d1" (UID: "2581da35-8109-4df3-a553-c8992a31b8d1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:31:45.360603 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.360573    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2581da35-8109-4df3-a553-c8992a31b8d1-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2581da35-8109-4df3-a553-c8992a31b8d1" (UID: "2581da35-8109-4df3-a553-c8992a31b8d1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:31:45.360761 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.360737    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2581da35-8109-4df3-a553-c8992a31b8d1" (UID: "2581da35-8109-4df3-a553-c8992a31b8d1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:31:45.360841 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.360743    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2581da35-8109-4df3-a553-c8992a31b8d1-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2581da35-8109-4df3-a553-c8992a31b8d1" (UID: "2581da35-8109-4df3-a553-c8992a31b8d1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:31:45.361221 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.361198    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2581da35-8109-4df3-a553-c8992a31b8d1" (UID: "2581da35-8109-4df3-a553-c8992a31b8d1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:31:45.361310 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.361270    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-kube-api-access-wjzgp" (OuterVolumeSpecName: "kube-api-access-wjzgp") pod "2581da35-8109-4df3-a553-c8992a31b8d1" (UID: "2581da35-8109-4df3-a553-c8992a31b8d1"). InnerVolumeSpecName "kube-api-access-wjzgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:31:45.369376 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.369345    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2581da35-8109-4df3-a553-c8992a31b8d1-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2581da35-8109-4df3-a553-c8992a31b8d1" (UID: "2581da35-8109-4df3-a553-c8992a31b8d1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:31:45.459148 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.459118    2556 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-registry-tls\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:31:45.459148 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.459147    2556 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2581da35-8109-4df3-a553-c8992a31b8d1-installation-pull-secrets\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:31:45.459370 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.459162    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjzgp\" (UniqueName: \"kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-kube-api-access-wjzgp\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:31:45.459370 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.459176    2556 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2581da35-8109-4df3-a553-c8992a31b8d1-ca-trust-extracted\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:31:45.459370 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.459188    2556 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2581da35-8109-4df3-a553-c8992a31b8d1-registry-certificates\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:31:45.459370 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.459199    2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2581da35-8109-4df3-a553-c8992a31b8d1-trusted-ca\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:31:45.459370 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.459210    2556 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2581da35-8109-4df3-a553-c8992a31b8d1-image-registry-private-configuration\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:31:45.459370 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.459223    2556 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2581da35-8109-4df3-a553-c8992a31b8d1-bound-sa-token\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:31:45.960528 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.960494    2556 generic.go:356] "Generic (PLEG): container finished" podID="2581da35-8109-4df3-a553-c8992a31b8d1" containerID="7076e2af0305555d99c8810b58b12283cd6caa0d052715734b03a4c34179437c" exitCode=0
Sep 17 15:31:45.960704 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.960546    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc"
Sep 17 15:31:45.960704 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.960583    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc" event={"ID":"2581da35-8109-4df3-a553-c8992a31b8d1","Type":"ContainerDied","Data":"7076e2af0305555d99c8810b58b12283cd6caa0d052715734b03a4c34179437c"}
Sep 17 15:31:45.960704 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.960634    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69d6bcf58c-bq4rc" event={"ID":"2581da35-8109-4df3-a553-c8992a31b8d1","Type":"ContainerDied","Data":"afc21b3bab1abcbac2dc1accde26954cadd917e98337d0b985befded6e639a48"}
Sep 17 15:31:45.960704 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.960660    2556 scope.go:117] "RemoveContainer" containerID="7076e2af0305555d99c8810b58b12283cd6caa0d052715734b03a4c34179437c"
Sep 17 15:31:45.970046 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.969858    2556 scope.go:117] "RemoveContainer" containerID="7076e2af0305555d99c8810b58b12283cd6caa0d052715734b03a4c34179437c"
Sep 17 15:31:45.970331 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:31:45.970172    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7076e2af0305555d99c8810b58b12283cd6caa0d052715734b03a4c34179437c\": container with ID starting with 7076e2af0305555d99c8810b58b12283cd6caa0d052715734b03a4c34179437c not found: ID does not exist" containerID="7076e2af0305555d99c8810b58b12283cd6caa0d052715734b03a4c34179437c"
Sep 17 15:31:45.970331 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.970197    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7076e2af0305555d99c8810b58b12283cd6caa0d052715734b03a4c34179437c"} err="failed to get container status \"7076e2af0305555d99c8810b58b12283cd6caa0d052715734b03a4c34179437c\": rpc error: code = NotFound desc = could not find container \"7076e2af0305555d99c8810b58b12283cd6caa0d052715734b03a4c34179437c\": container with ID starting with 7076e2af0305555d99c8810b58b12283cd6caa0d052715734b03a4c34179437c not found: ID does not exist"
Sep 17 15:31:45.982414 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.982390    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69d6bcf58c-bq4rc"]
Sep 17 15:31:45.991737 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:45.991703    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-69d6bcf58c-bq4rc"]
Sep 17 15:31:46.135511 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:31:46.135466    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2581da35-8109-4df3-a553-c8992a31b8d1" path="/var/lib/kubelet/pods/2581da35-8109-4df3-a553-c8992a31b8d1/volumes"
Sep 17 15:32:02.027011 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:32:02.026972    2556 generic.go:356] "Generic (PLEG): container finished" podID="95eb306e-4e4f-48ed-9b49-14471d4b4ddf" containerID="4980d8d8e9a874a76f78ee00372288097a8d6ae38c49baa54e1daaa62a41633e" exitCode=0
Sep 17 15:32:02.027440 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:32:02.027048    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-xqb4f" event={"ID":"95eb306e-4e4f-48ed-9b49-14471d4b4ddf","Type":"ContainerDied","Data":"4980d8d8e9a874a76f78ee00372288097a8d6ae38c49baa54e1daaa62a41633e"}
Sep 17 15:32:02.027440 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:32:02.027427    2556 scope.go:117] "RemoveContainer" containerID="4980d8d8e9a874a76f78ee00372288097a8d6ae38c49baa54e1daaa62a41633e"
Sep 17 15:32:03.032866 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:32:03.032828    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-xqb4f" event={"ID":"95eb306e-4e4f-48ed-9b49-14471d4b4ddf","Type":"ContainerStarted","Data":"53fecdb3ea6d83097be6c307a6da337b1a0dbb0402ac641b86c736f915a40271"}
Sep 17 15:34:27.276106 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:27.276054    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5df9996869-86l8g"]
Sep 17 15:34:27.276630 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:27.276534    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2581da35-8109-4df3-a553-c8992a31b8d1" containerName="registry"
Sep 17 15:34:27.276630 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:27.276556    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2581da35-8109-4df3-a553-c8992a31b8d1" containerName="registry"
Sep 17 15:34:27.276748 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:27.276642    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2581da35-8109-4df3-a553-c8992a31b8d1" containerName="registry"
Sep 17 15:34:27.278711 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:27.278689    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-86l8g"
Sep 17 15:34:27.281746 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:27.281729    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-r95mn\""
Sep 17 15:34:27.282003 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:27.281987    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 17 15:34:27.282051 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:27.282039    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 17 15:34:27.282318 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:27.282301    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 17 15:34:27.282534 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:27.282522    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 17 15:34:27.318617 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:27.318587    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5df9996869-86l8g"]
Sep 17 15:34:27.352095 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:27.352040    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fa8daaea-084c-4cb6-9c92-2b5a61aac914-manifests\") pod \"opendatahub-operator-controller-manager-5df9996869-86l8g\" (UID: \"fa8daaea-084c-4cb6-9c92-2b5a61aac914\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-86l8g"
Sep 17 15:34:27.352256 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:27.352101    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fa8daaea-084c-4cb6-9c92-2b5a61aac914-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5df9996869-86l8g\" (UID: \"fa8daaea-084c-4cb6-9c92-2b5a61aac914\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-86l8g"
Sep 17 15:34:27.352256 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:27.352123    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bjs4\" (UniqueName: \"kubernetes.io/projected/fa8daaea-084c-4cb6-9c92-2b5a61aac914-kube-api-access-8bjs4\") pod \"opendatahub-operator-controller-manager-5df9996869-86l8g\" (UID: \"fa8daaea-084c-4cb6-9c92-2b5a61aac914\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-86l8g"
Sep 17 15:34:27.352256 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:27.352179    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa8daaea-084c-4cb6-9c92-2b5a61aac914-tmp\") pod \"opendatahub-operator-controller-manager-5df9996869-86l8g\" (UID: \"fa8daaea-084c-4cb6-9c92-2b5a61aac914\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-86l8g"
Sep 17 15:34:27.352256 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:27.352199    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa8daaea-084c-4cb6-9c92-2b5a61aac914-webhook-cert\") pod \"opendatahub-operator-controller-manager-5df9996869-86l8g\" (UID: \"fa8daaea-084c-4cb6-9c92-2b5a61aac914\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-86l8g"
Sep 17 15:34:27.452743 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:27.452709    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fa8daaea-084c-4cb6-9c92-2b5a61aac914-manifests\") pod \"opendatahub-operator-controller-manager-5df9996869-86l8g\" (UID: \"fa8daaea-084c-4cb6-9c92-2b5a61aac914\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-86l8g"
Sep 17 15:34:27.452743 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:27.452747    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fa8daaea-084c-4cb6-9c92-2b5a61aac914-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5df9996869-86l8g\" (UID: \"fa8daaea-084c-4cb6-9c92-2b5a61aac914\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-86l8g"
Sep 17 15:34:27.452990 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:27.452769    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bjs4\" (UniqueName: \"kubernetes.io/projected/fa8daaea-084c-4cb6-9c92-2b5a61aac914-kube-api-access-8bjs4\") pod \"opendatahub-operator-controller-manager-5df9996869-86l8g\" (UID: \"fa8daaea-084c-4cb6-9c92-2b5a61aac914\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-86l8g"
Sep 17 15:34:27.452990 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:27.452814    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa8daaea-084c-4cb6-9c92-2b5a61aac914-tmp\") pod \"opendatahub-operator-controller-manager-5df9996869-86l8g\" (UID: \"fa8daaea-084c-4cb6-9c92-2b5a61aac914\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-86l8g"
Sep 17 15:34:27.452990 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:27.452841    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa8daaea-084c-4cb6-9c92-2b5a61aac914-webhook-cert\") pod \"opendatahub-operator-controller-manager-5df9996869-86l8g\" (UID: \"fa8daaea-084c-4cb6-9c92-2b5a61aac914\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-86l8g"
Sep 17 15:34:27.453251 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:27.453231    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fa8daaea-084c-4cb6-9c92-2b5a61aac914-manifests\") pod \"opendatahub-operator-controller-manager-5df9996869-86l8g\" (UID: \"fa8daaea-084c-4cb6-9c92-2b5a61aac914\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-86l8g"
Sep 17 15:34:27.455188 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:27.455162    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa8daaea-084c-4cb6-9c92-2b5a61aac914-tmp\") pod \"opendatahub-operator-controller-manager-5df9996869-86l8g\" (UID: \"fa8daaea-084c-4cb6-9c92-2b5a61aac914\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-86l8g"
Sep 17 15:34:27.455409 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:27.455390    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa8daaea-084c-4cb6-9c92-2b5a61aac914-webhook-cert\") pod \"opendatahub-operator-controller-manager-5df9996869-86l8g\" (UID: \"fa8daaea-084c-4cb6-9c92-2b5a61aac914\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-86l8g"
Sep 17 15:34:27.455497 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:27.455475    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fa8daaea-084c-4cb6-9c92-2b5a61aac914-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5df9996869-86l8g\" (UID: \"fa8daaea-084c-4cb6-9c92-2b5a61aac914\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-86l8g"
Sep 17 15:34:27.464642 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:27.464622    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bjs4\" (UniqueName: \"kubernetes.io/projected/fa8daaea-084c-4cb6-9c92-2b5a61aac914-kube-api-access-8bjs4\") pod \"opendatahub-operator-controller-manager-5df9996869-86l8g\" (UID: \"fa8daaea-084c-4cb6-9c92-2b5a61aac914\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-86l8g"
Sep 17 15:34:27.588194 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:27.588101    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-86l8g"
Sep 17 15:34:27.868910 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:27.868869    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5df9996869-86l8g"]
Sep 17 15:34:27.871727 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:34:27.871699    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa8daaea_084c_4cb6_9c92_2b5a61aac914.slice/crio-540a28d834edc8257a1688f09d1eefefa58640268737665fe0dd3d59690b241f WatchSource:0}: Error finding container 540a28d834edc8257a1688f09d1eefefa58640268737665fe0dd3d59690b241f: Status 404 returned error can't find the container with id 540a28d834edc8257a1688f09d1eefefa58640268737665fe0dd3d59690b241f
Sep 17 15:34:28.593723 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:28.593689    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-86l8g" event={"ID":"fa8daaea-084c-4cb6-9c92-2b5a61aac914","Type":"ContainerStarted","Data":"540a28d834edc8257a1688f09d1eefefa58640268737665fe0dd3d59690b241f"}
Sep 17 15:34:31.608605 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:31.608569    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-86l8g" event={"ID":"fa8daaea-084c-4cb6-9c92-2b5a61aac914","Type":"ContainerStarted","Data":"edeee9576e599ee75bf3a7bbb02b64600c2c5d5ac2e28d322b4666c244bc6cba"}
Sep 17 15:34:32.614368 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:32.614332    2556 generic.go:356] "Generic (PLEG): container finished" podID="fa8daaea-084c-4cb6-9c92-2b5a61aac914" containerID="edeee9576e599ee75bf3a7bbb02b64600c2c5d5ac2e28d322b4666c244bc6cba" exitCode=0
Sep 17 15:34:32.614736 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:32.614374    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-86l8g" event={"ID":"fa8daaea-084c-4cb6-9c92-2b5a61aac914","Type":"ContainerDied","Data":"edeee9576e599ee75bf3a7bbb02b64600c2c5d5ac2e28d322b4666c244bc6cba"}
Sep 17 15:34:33.621030 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:33.620995    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-86l8g" event={"ID":"fa8daaea-084c-4cb6-9c92-2b5a61aac914","Type":"ContainerStarted","Data":"26f21fdc8d6e10d42c8a772b344b9e2d91a8f66c514142b58c39a93524405290"}
Sep 17 15:34:33.621521 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:33.621122    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-86l8g"
Sep 17 15:34:33.651209 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:33.651155    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-86l8g" podStartSLOduration=2.102462326 podStartE2EDuration="6.651137926s" totalImagesPullingTime="3.531090668s" totalInitContainerRuntime="1.017584932s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:34:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:34:33.649535396 +0000 UTC m=+296.029553152" watchObservedRunningTime="2026-09-17 15:34:33.651137926 +0000 UTC m=+296.031155681"
Sep 17 15:34:38.038244 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:38.038212    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-45zt7_2c069c9e-0856-46b0-ba10-67829c05ac94/console-operator/1.log"
Sep 17 15:34:38.049252 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:38.049223    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-45zt7_2c069c9e-0856-46b0-ba10-67829c05ac94/console-operator/1.log"
Sep 17 15:34:38.053159 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:38.053141    2556 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 15:34:44.628450 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:44.628416    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-86l8g"
Sep 17 15:34:59.337643 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:59.337608    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4g2kw"]
Sep 17 15:34:59.341485 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:59.341463    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-4g2kw"
Sep 17 15:34:59.344715 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:59.344693    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 17 15:34:59.352307 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:59.352283    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4g2kw"]
Sep 17 15:34:59.431273 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:59.431234    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pr5j\" (UniqueName: \"kubernetes.io/projected/693bf480-ee10-4366-9335-ef0c2651f980-kube-api-access-6pr5j\") pod \"model-serving-api-54d89bd5fc-4g2kw\" (UID: \"693bf480-ee10-4366-9335-ef0c2651f980\") " pod="opendatahub/model-serving-api-54d89bd5fc-4g2kw"
Sep 17 15:34:59.431471 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:59.431290    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/693bf480-ee10-4366-9335-ef0c2651f980-tls-certs\") pod \"model-serving-api-54d89bd5fc-4g2kw\" (UID: \"693bf480-ee10-4366-9335-ef0c2651f980\") " pod="opendatahub/model-serving-api-54d89bd5fc-4g2kw"
Sep 17 15:34:59.531832 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:59.531789    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pr5j\" (UniqueName: \"kubernetes.io/projected/693bf480-ee10-4366-9335-ef0c2651f980-kube-api-access-6pr5j\") pod \"model-serving-api-54d89bd5fc-4g2kw\" (UID: \"693bf480-ee10-4366-9335-ef0c2651f980\") " pod="opendatahub/model-serving-api-54d89bd5fc-4g2kw"
Sep 17 15:34:59.532004 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:59.531860    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/693bf480-ee10-4366-9335-ef0c2651f980-tls-certs\") pod \"model-serving-api-54d89bd5fc-4g2kw\" (UID: \"693bf480-ee10-4366-9335-ef0c2651f980\") " pod="opendatahub/model-serving-api-54d89bd5fc-4g2kw"
Sep 17 15:34:59.532055 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:34:59.532019    2556 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 17 15:34:59.532135 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:34:59.532123    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/693bf480-ee10-4366-9335-ef0c2651f980-tls-certs podName:693bf480-ee10-4366-9335-ef0c2651f980 nodeName:}" failed. No retries permitted until 2026-09-17 15:35:00.03210041 +0000 UTC m=+322.412118161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/693bf480-ee10-4366-9335-ef0c2651f980-tls-certs") pod "model-serving-api-54d89bd5fc-4g2kw" (UID: "693bf480-ee10-4366-9335-ef0c2651f980") : secret "model-serving-api-tls" not found
Sep 17 15:34:59.540969 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:34:59.540939    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pr5j\" (UniqueName: \"kubernetes.io/projected/693bf480-ee10-4366-9335-ef0c2651f980-kube-api-access-6pr5j\") pod \"model-serving-api-54d89bd5fc-4g2kw\" (UID: \"693bf480-ee10-4366-9335-ef0c2651f980\") " pod="opendatahub/model-serving-api-54d89bd5fc-4g2kw"
Sep 17 15:35:00.036727 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:00.036682    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/693bf480-ee10-4366-9335-ef0c2651f980-tls-certs\") pod \"model-serving-api-54d89bd5fc-4g2kw\" (UID: \"693bf480-ee10-4366-9335-ef0c2651f980\") " pod="opendatahub/model-serving-api-54d89bd5fc-4g2kw"
Sep 17 15:35:00.039193 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:00.039155    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/693bf480-ee10-4366-9335-ef0c2651f980-tls-certs\") pod \"model-serving-api-54d89bd5fc-4g2kw\" (UID: \"693bf480-ee10-4366-9335-ef0c2651f980\") " pod="opendatahub/model-serving-api-54d89bd5fc-4g2kw"
Sep 17 15:35:00.252634 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:00.252600    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-4g2kw"
Sep 17 15:35:00.523184 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:00.523148    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4g2kw"]
Sep 17 15:35:00.526262 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:35:00.526229    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod693bf480_ee10_4366_9335_ef0c2651f980.slice/crio-0b30e2b5f7c14f3f3dadde9dbad597b6f8f074cddc6b6db44fd23049b89967c3 WatchSource:0}: Error finding container 0b30e2b5f7c14f3f3dadde9dbad597b6f8f074cddc6b6db44fd23049b89967c3: Status 404 returned error can't find the container with id 0b30e2b5f7c14f3f3dadde9dbad597b6f8f074cddc6b6db44fd23049b89967c3
Sep 17 15:35:00.528550 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:00.528534    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:35:00.738878 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:00.738837    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-4g2kw" event={"ID":"693bf480-ee10-4366-9335-ef0c2651f980","Type":"ContainerStarted","Data":"0b30e2b5f7c14f3f3dadde9dbad597b6f8f074cddc6b6db44fd23049b89967c3"}
Sep 17 15:35:03.757055 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:03.757013    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-4g2kw" event={"ID":"693bf480-ee10-4366-9335-ef0c2651f980","Type":"ContainerStarted","Data":"936c2e6b343ce35a9ae4a73f2a4ec5d9ecb42cf563b384edea992c5416492deb"}
Sep 17 15:35:03.757448 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:03.757184    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-4g2kw"
Sep 17 15:35:03.776020 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:03.775973    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-4g2kw" podStartSLOduration=2.215535343 podStartE2EDuration="4.77595882s" totalImagesPullingTime="2.560423477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:34:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:35:03.774715939 +0000 UTC m=+326.154733695" watchObservedRunningTime="2026-09-17 15:35:03.77595882 +0000 UTC m=+326.155976575"
Sep 17 15:35:06.094382 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:06.094337    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c7d57fb89-fc5dq"]
Sep 17 15:35:06.098327 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:06.098303    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-fc5dq"
Sep 17 15:35:06.101646 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:06.101623    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 15:35:06.102815 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:06.102797    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 15:35:06.103338 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:06.103317    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:35:06.103448 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:06.103332    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 15:35:06.103448 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:06.103335    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 15:35:06.103642 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:06.103627    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-75rsx\""
Sep 17 15:35:06.110270 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:06.110248    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c7d57fb89-fc5dq"]
Sep 17 15:35:06.192212 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:06.192181    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0ccf5b35-fb27-4838-8a3b-37a3ed4a9f39-manager-config\") pod \"lws-controller-manager-6c7d57fb89-fc5dq\" (UID: \"0ccf5b35-fb27-4838-8a3b-37a3ed4a9f39\") " pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-fc5dq"
Sep 17 15:35:06.192369 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:06.192228    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0ccf5b35-fb27-4838-8a3b-37a3ed4a9f39-metrics-cert\") pod \"lws-controller-manager-6c7d57fb89-fc5dq\" (UID: \"0ccf5b35-fb27-4838-8a3b-37a3ed4a9f39\") " pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-fc5dq"
Sep 17 15:35:06.192369 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:06.192288    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssrck\" (UniqueName: \"kubernetes.io/projected/0ccf5b35-fb27-4838-8a3b-37a3ed4a9f39-kube-api-access-ssrck\") pod \"lws-controller-manager-6c7d57fb89-fc5dq\" (UID: \"0ccf5b35-fb27-4838-8a3b-37a3ed4a9f39\") " pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-fc5dq"
Sep 17 15:35:06.192369 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:06.192358    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ccf5b35-fb27-4838-8a3b-37a3ed4a9f39-cert\") pod \"lws-controller-manager-6c7d57fb89-fc5dq\" (UID: \"0ccf5b35-fb27-4838-8a3b-37a3ed4a9f39\") " pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-fc5dq"
Sep 17 15:35:06.293580 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:06.293545    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ccf5b35-fb27-4838-8a3b-37a3ed4a9f39-cert\") pod \"lws-controller-manager-6c7d57fb89-fc5dq\" (UID: \"0ccf5b35-fb27-4838-8a3b-37a3ed4a9f39\") " pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-fc5dq"
Sep 17 15:35:06.293760 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:06.293591    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0ccf5b35-fb27-4838-8a3b-37a3ed4a9f39-manager-config\") pod \"lws-controller-manager-6c7d57fb89-fc5dq\" (UID: \"0ccf5b35-fb27-4838-8a3b-37a3ed4a9f39\") " pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-fc5dq"
Sep 17 15:35:06.293760 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:06.293635    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0ccf5b35-fb27-4838-8a3b-37a3ed4a9f39-metrics-cert\") pod \"lws-controller-manager-6c7d57fb89-fc5dq\" (UID: \"0ccf5b35-fb27-4838-8a3b-37a3ed4a9f39\") " pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-fc5dq"
Sep 17 15:35:06.293760 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:06.293666    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssrck\" (UniqueName: \"kubernetes.io/projected/0ccf5b35-fb27-4838-8a3b-37a3ed4a9f39-kube-api-access-ssrck\") pod \"lws-controller-manager-6c7d57fb89-fc5dq\" (UID: \"0ccf5b35-fb27-4838-8a3b-37a3ed4a9f39\") " pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-fc5dq"
Sep 17 15:35:06.294379 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:06.294352    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0ccf5b35-fb27-4838-8a3b-37a3ed4a9f39-manager-config\") pod \"lws-controller-manager-6c7d57fb89-fc5dq\" (UID: \"0ccf5b35-fb27-4838-8a3b-37a3ed4a9f39\") " pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-fc5dq"
Sep 17 15:35:06.296045 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:06.296020    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0ccf5b35-fb27-4838-8a3b-37a3ed4a9f39-metrics-cert\") pod \"lws-controller-manager-6c7d57fb89-fc5dq\" (UID: \"0ccf5b35-fb27-4838-8a3b-37a3ed4a9f39\") " pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-fc5dq"
Sep 17 15:35:06.296153 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:06.296137    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ccf5b35-fb27-4838-8a3b-37a3ed4a9f39-cert\") pod \"lws-controller-manager-6c7d57fb89-fc5dq\" (UID: \"0ccf5b35-fb27-4838-8a3b-37a3ed4a9f39\") " pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-fc5dq"
Sep 17 15:35:06.311041 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:06.311016    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssrck\" (UniqueName: \"kubernetes.io/projected/0ccf5b35-fb27-4838-8a3b-37a3ed4a9f39-kube-api-access-ssrck\") pod \"lws-controller-manager-6c7d57fb89-fc5dq\" (UID: \"0ccf5b35-fb27-4838-8a3b-37a3ed4a9f39\") " pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-fc5dq"
Sep 17 15:35:06.408587 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:06.408554    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-fc5dq"
Sep 17 15:35:06.697291 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:06.697103    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c7d57fb89-fc5dq"]
Sep 17 15:35:06.699799 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:35:06.699769    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ccf5b35_fb27_4838_8a3b_37a3ed4a9f39.slice/crio-867cbbf7392995e5cbc6de6e6e6f804283e540b3e2c05db7d71ab47dceff57ff WatchSource:0}: Error finding container 867cbbf7392995e5cbc6de6e6e6f804283e540b3e2c05db7d71ab47dceff57ff: Status 404 returned error can't find the container with id 867cbbf7392995e5cbc6de6e6e6f804283e540b3e2c05db7d71ab47dceff57ff
Sep 17 15:35:06.771260 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:06.771225    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-fc5dq" event={"ID":"0ccf5b35-fb27-4838-8a3b-37a3ed4a9f39","Type":"ContainerStarted","Data":"867cbbf7392995e5cbc6de6e6e6f804283e540b3e2c05db7d71ab47dceff57ff"}
Sep 17 15:35:08.710258 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:08.710193    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-lq9sm"]
Sep 17 15:35:08.720735 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:08.720709    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lq9sm"
Sep 17 15:35:08.724233 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:08.723726    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-zm427\""
Sep 17 15:35:08.724233 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:08.724054    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 17 15:35:08.736392 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:08.736364    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-lq9sm"]
Sep 17 15:35:08.917707 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:08.917671    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fjnp\" (UniqueName: \"kubernetes.io/projected/89181401-93fb-4dac-8941-57ebe347292a-kube-api-access-6fjnp\") pod \"llmisvc-controller-manager-777c6bd7d5-lq9sm\" (UID: \"89181401-93fb-4dac-8941-57ebe347292a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lq9sm"
Sep 17 15:35:08.917901 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:08.917735    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89181401-93fb-4dac-8941-57ebe347292a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lq9sm\" (UID: \"89181401-93fb-4dac-8941-57ebe347292a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lq9sm"
Sep 17 15:35:09.018760 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:09.018679    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fjnp\" (UniqueName: \"kubernetes.io/projected/89181401-93fb-4dac-8941-57ebe347292a-kube-api-access-6fjnp\") pod \"llmisvc-controller-manager-777c6bd7d5-lq9sm\" (UID: \"89181401-93fb-4dac-8941-57ebe347292a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lq9sm"
Sep 17 15:35:09.018760 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:09.018758    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89181401-93fb-4dac-8941-57ebe347292a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lq9sm\" (UID: \"89181401-93fb-4dac-8941-57ebe347292a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lq9sm"
Sep 17 15:35:09.021592 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:09.021565    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89181401-93fb-4dac-8941-57ebe347292a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lq9sm\" (UID: \"89181401-93fb-4dac-8941-57ebe347292a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lq9sm"
Sep 17 15:35:09.044243 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:09.044212    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fjnp\" (UniqueName: \"kubernetes.io/projected/89181401-93fb-4dac-8941-57ebe347292a-kube-api-access-6fjnp\") pod \"llmisvc-controller-manager-777c6bd7d5-lq9sm\" (UID: \"89181401-93fb-4dac-8941-57ebe347292a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lq9sm"
Sep 17 15:35:09.335963 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:09.335939    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lq9sm"
Sep 17 15:35:09.611659 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:09.611576    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-lq9sm"]
Sep 17 15:35:09.613693 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:35:09.613660    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod89181401_93fb_4dac_8941_57ebe347292a.slice/crio-e141e44e7f7bad05e7610af510799e5d34d9ee461e2e39f5de53e402ed056fde WatchSource:0}: Error finding container e141e44e7f7bad05e7610af510799e5d34d9ee461e2e39f5de53e402ed056fde: Status 404 returned error can't find the container with id e141e44e7f7bad05e7610af510799e5d34d9ee461e2e39f5de53e402ed056fde
Sep 17 15:35:09.788563 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:09.788524    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lq9sm" event={"ID":"89181401-93fb-4dac-8941-57ebe347292a","Type":"ContainerStarted","Data":"e141e44e7f7bad05e7610af510799e5d34d9ee461e2e39f5de53e402ed056fde"}
Sep 17 15:35:09.790213 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:09.790188    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-fc5dq" event={"ID":"0ccf5b35-fb27-4838-8a3b-37a3ed4a9f39","Type":"ContainerStarted","Data":"77b14fbfe5d0abcaa354d87df4f83d79956d00fce2d7fa2c086b58d309bf846a"}
Sep 17 15:35:09.790332 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:09.790316    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-fc5dq"
Sep 17 15:35:09.820366 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:09.820318    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-fc5dq" podStartSLOduration=1.187083608 podStartE2EDuration="3.820302879s" totalImagesPullingTime="2.633219271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:35:09.819546085 +0000 UTC m=+332.199563842" watchObservedRunningTime="2026-09-17 15:35:09.820302879 +0000 UTC m=+332.200320703"
Sep 17 15:35:12.807627 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:12.807590    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lq9sm" event={"ID":"89181401-93fb-4dac-8941-57ebe347292a","Type":"ContainerStarted","Data":"ea4460750b0553f3a34cf79362b4fd38f50bd400e806cd7b7ccdc391ac36eac8"}
Sep 17 15:35:12.808019 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:12.807721    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lq9sm"
Sep 17 15:35:12.828193 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:12.828146    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lq9sm" podStartSLOduration=1.8241775059999998 podStartE2EDuration="4.828132237s" totalImagesPullingTime="3.003954731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:35:12.826491211 +0000 UTC m=+335.206508969" watchObservedRunningTime="2026-09-17 15:35:12.828132237 +0000 UTC m=+335.208149992"
Sep 17 15:35:14.765919 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:14.765889    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-4g2kw"
Sep 17 15:35:20.797805 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:20.797772    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-fc5dq"
Sep 17 15:35:43.814863 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:35:43.814782    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lq9sm"
Sep 17 15:36:24.579488 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:36:24.579454    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-s45bq"]
Sep 17 15:36:24.582639 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:36:24.582623    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-s45bq"
Sep 17 15:36:24.588278 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:36:24.588255    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 17 15:36:24.594323 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:36:24.588452    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 15:36:24.594629 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:36:24.594602    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-v6vfz\""
Sep 17 15:36:24.594737 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:36:24.594653    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 15:36:24.599592 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:36:24.599572    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-s45bq"]
Sep 17 15:36:24.632013 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:36:24.631974    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zc8r\" (UniqueName: \"kubernetes.io/projected/e8a4c613-d01f-42ed-b771-0744a7e9f2e9-kube-api-access-4zc8r\") pod \"dns-operator-controller-manager-7b84cd7c9c-s45bq\" (UID: \"e8a4c613-d01f-42ed-b771-0744a7e9f2e9\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-s45bq"
Sep 17 15:36:24.732965 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:36:24.732922    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zc8r\" (UniqueName: \"kubernetes.io/projected/e8a4c613-d01f-42ed-b771-0744a7e9f2e9-kube-api-access-4zc8r\") pod \"dns-operator-controller-manager-7b84cd7c9c-s45bq\" (UID: \"e8a4c613-d01f-42ed-b771-0744a7e9f2e9\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-s45bq"
Sep 17 15:36:24.743463 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:36:24.743434    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zc8r\" (UniqueName: \"kubernetes.io/projected/e8a4c613-d01f-42ed-b771-0744a7e9f2e9-kube-api-access-4zc8r\") pod \"dns-operator-controller-manager-7b84cd7c9c-s45bq\" (UID: \"e8a4c613-d01f-42ed-b771-0744a7e9f2e9\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-s45bq"
Sep 17 15:36:24.898420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:36:24.898388    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-s45bq"
Sep 17 15:36:25.170912 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:36:25.170888    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-s45bq"]
Sep 17 15:36:25.173442 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:36:25.173414    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a4c613_d01f_42ed_b771_0744a7e9f2e9.slice/crio-7bfa685e104515045519479bcff10e53299022f9fd6661a53ba0b0f6b8c07931 WatchSource:0}: Error finding container 7bfa685e104515045519479bcff10e53299022f9fd6661a53ba0b0f6b8c07931: Status 404 returned error can't find the container with id 7bfa685e104515045519479bcff10e53299022f9fd6661a53ba0b0f6b8c07931
Sep 17 15:36:26.137006 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:36:26.136958    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-s45bq" event={"ID":"e8a4c613-d01f-42ed-b771-0744a7e9f2e9","Type":"ContainerStarted","Data":"7bfa685e104515045519479bcff10e53299022f9fd6661a53ba0b0f6b8c07931"}
Sep 17 15:36:28.142852 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:36:28.142819    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-s45bq" event={"ID":"e8a4c613-d01f-42ed-b771-0744a7e9f2e9","Type":"ContainerStarted","Data":"62aba4c19322fea0984d5307eb1c1f5596c3ac13395c1c6fb5f5784ecc4287e4"}
Sep 17 15:36:28.143300 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:36:28.142930    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-s45bq"
Sep 17 15:36:28.183240 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:36:28.183183    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-s45bq" podStartSLOduration=1.7866777040000001 podStartE2EDuration="4.183167544s" totalImagesPullingTime="2.39648984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:36:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:36:28.181794731 +0000 UTC m=+410.561812499" watchObservedRunningTime="2026-09-17 15:36:28.183167544 +0000 UTC m=+410.563185300"
Sep 17 15:36:39.149646 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:36:39.149613    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-s45bq"
Sep 17 15:37:32.674397 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:32.674365    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-88nxf"]
Sep 17 15:37:32.676600 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:32.676586    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-88nxf"
Sep 17 15:37:32.682193 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:32.682168    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4cmh6\""
Sep 17 15:37:32.682318 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:32.682168    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 17 15:37:32.694936 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:32.694904    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-88nxf"]
Sep 17 15:37:32.766905 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:32.766877    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-88nxf"]
Sep 17 15:37:32.776999 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:32.776975    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zbtp\" (UniqueName: \"kubernetes.io/projected/6ab80d4c-1566-4acc-a5a0-57a7aba82eb5-kube-api-access-6zbtp\") pod \"limitador-limitador-8588749977-88nxf\" (UID: \"6ab80d4c-1566-4acc-a5a0-57a7aba82eb5\") " pod="kuadrant-system/limitador-limitador-8588749977-88nxf"
Sep 17 15:37:32.777136 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:32.777023    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6ab80d4c-1566-4acc-a5a0-57a7aba82eb5-config-file\") pod \"limitador-limitador-8588749977-88nxf\" (UID: \"6ab80d4c-1566-4acc-a5a0-57a7aba82eb5\") " pod="kuadrant-system/limitador-limitador-8588749977-88nxf"
Sep 17 15:37:32.878075 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:32.878013    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zbtp\" (UniqueName: \"kubernetes.io/projected/6ab80d4c-1566-4acc-a5a0-57a7aba82eb5-kube-api-access-6zbtp\") pod \"limitador-limitador-8588749977-88nxf\" (UID: \"6ab80d4c-1566-4acc-a5a0-57a7aba82eb5\") " pod="kuadrant-system/limitador-limitador-8588749977-88nxf"
Sep 17 15:37:32.878260 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:32.878103    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6ab80d4c-1566-4acc-a5a0-57a7aba82eb5-config-file\") pod \"limitador-limitador-8588749977-88nxf\" (UID: \"6ab80d4c-1566-4acc-a5a0-57a7aba82eb5\") " pod="kuadrant-system/limitador-limitador-8588749977-88nxf"
Sep 17 15:37:32.878669 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:32.878651    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6ab80d4c-1566-4acc-a5a0-57a7aba82eb5-config-file\") pod \"limitador-limitador-8588749977-88nxf\" (UID: \"6ab80d4c-1566-4acc-a5a0-57a7aba82eb5\") " pod="kuadrant-system/limitador-limitador-8588749977-88nxf"
Sep 17 15:37:32.887870 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:32.887842    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zbtp\" (UniqueName: \"kubernetes.io/projected/6ab80d4c-1566-4acc-a5a0-57a7aba82eb5-kube-api-access-6zbtp\") pod \"limitador-limitador-8588749977-88nxf\" (UID: \"6ab80d4c-1566-4acc-a5a0-57a7aba82eb5\") " pod="kuadrant-system/limitador-limitador-8588749977-88nxf"
Sep 17 15:37:32.986375 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:32.986283    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-88nxf"
Sep 17 15:37:33.260951 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:33.260927    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-88nxf"]
Sep 17 15:37:33.262806 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:37:33.262766    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab80d4c_1566_4acc_a5a0_57a7aba82eb5.slice/crio-7cdda33d9bcac3fbba6595a082d0a58cc5c746315bbd849eb04e4e2c7643d1ae WatchSource:0}: Error finding container 7cdda33d9bcac3fbba6595a082d0a58cc5c746315bbd849eb04e4e2c7643d1ae: Status 404 returned error can't find the container with id 7cdda33d9bcac3fbba6595a082d0a58cc5c746315bbd849eb04e4e2c7643d1ae
Sep 17 15:37:33.418881 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:33.418849    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-88nxf" event={"ID":"6ab80d4c-1566-4acc-a5a0-57a7aba82eb5","Type":"ContainerStarted","Data":"7cdda33d9bcac3fbba6595a082d0a58cc5c746315bbd849eb04e4e2c7643d1ae"}
Sep 17 15:37:35.429376 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:35.429326    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-88nxf" event={"ID":"6ab80d4c-1566-4acc-a5a0-57a7aba82eb5","Type":"ContainerStarted","Data":"9eee82c4ab4fd13dde539abb7046f3d67163b20cd1bcaee05c7712a735e4484c"}
Sep 17 15:37:35.429753 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:35.429457    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-88nxf"
Sep 17 15:37:35.456079 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:35.456006    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-88nxf" podStartSLOduration=2.025748316 podStartE2EDuration="3.455986392s" totalImagesPullingTime="1.430238076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:37:35.4556516 +0000 UTC m=+477.835669357" watchObservedRunningTime="2026-09-17 15:37:35.455986392 +0000 UTC m=+477.836004146"
Sep 17 15:37:46.435384 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:46.435351    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-88nxf"
Sep 17 15:37:46.564558 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:46.564521    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-88nxf"]
Sep 17 15:37:46.564761 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:46.564736    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-88nxf" podUID="6ab80d4c-1566-4acc-a5a0-57a7aba82eb5" containerName="limitador" containerID="cri-o://9eee82c4ab4fd13dde539abb7046f3d67163b20cd1bcaee05c7712a735e4484c" gracePeriod=30
Sep 17 15:37:47.105352 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:47.105330    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-88nxf"
Sep 17 15:37:47.193727 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:47.193666    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zbtp\" (UniqueName: \"kubernetes.io/projected/6ab80d4c-1566-4acc-a5a0-57a7aba82eb5-kube-api-access-6zbtp\") pod \"6ab80d4c-1566-4acc-a5a0-57a7aba82eb5\" (UID: \"6ab80d4c-1566-4acc-a5a0-57a7aba82eb5\") "
Sep 17 15:37:47.193866 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:47.193754    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6ab80d4c-1566-4acc-a5a0-57a7aba82eb5-config-file\") pod \"6ab80d4c-1566-4acc-a5a0-57a7aba82eb5\" (UID: \"6ab80d4c-1566-4acc-a5a0-57a7aba82eb5\") "
Sep 17 15:37:47.194104 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:47.194044    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab80d4c-1566-4acc-a5a0-57a7aba82eb5-config-file" (OuterVolumeSpecName: "config-file") pod "6ab80d4c-1566-4acc-a5a0-57a7aba82eb5" (UID: "6ab80d4c-1566-4acc-a5a0-57a7aba82eb5"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:37:47.195689 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:47.195668    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab80d4c-1566-4acc-a5a0-57a7aba82eb5-kube-api-access-6zbtp" (OuterVolumeSpecName: "kube-api-access-6zbtp") pod "6ab80d4c-1566-4acc-a5a0-57a7aba82eb5" (UID: "6ab80d4c-1566-4acc-a5a0-57a7aba82eb5"). InnerVolumeSpecName "kube-api-access-6zbtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:37:47.294339 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:47.294305    2556 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6ab80d4c-1566-4acc-a5a0-57a7aba82eb5-config-file\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:37:47.294339 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:47.294334    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zbtp\" (UniqueName: \"kubernetes.io/projected/6ab80d4c-1566-4acc-a5a0-57a7aba82eb5-kube-api-access-6zbtp\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:37:47.483302 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:47.483220    2556 generic.go:356] "Generic (PLEG): container finished" podID="6ab80d4c-1566-4acc-a5a0-57a7aba82eb5" containerID="9eee82c4ab4fd13dde539abb7046f3d67163b20cd1bcaee05c7712a735e4484c" exitCode=0
Sep 17 15:37:47.483302 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:47.483264    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-88nxf"
Sep 17 15:37:47.483718 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:47.483305    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-88nxf" event={"ID":"6ab80d4c-1566-4acc-a5a0-57a7aba82eb5","Type":"ContainerDied","Data":"9eee82c4ab4fd13dde539abb7046f3d67163b20cd1bcaee05c7712a735e4484c"}
Sep 17 15:37:47.483718 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:47.483351    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-88nxf" event={"ID":"6ab80d4c-1566-4acc-a5a0-57a7aba82eb5","Type":"ContainerDied","Data":"7cdda33d9bcac3fbba6595a082d0a58cc5c746315bbd849eb04e4e2c7643d1ae"}
Sep 17 15:37:47.483718 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:47.483371    2556 scope.go:117] "RemoveContainer" containerID="9eee82c4ab4fd13dde539abb7046f3d67163b20cd1bcaee05c7712a735e4484c"
Sep 17 15:37:47.491967 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:47.491952    2556 scope.go:117] "RemoveContainer" containerID="9eee82c4ab4fd13dde539abb7046f3d67163b20cd1bcaee05c7712a735e4484c"
Sep 17 15:37:47.492268 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:37:47.492250    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eee82c4ab4fd13dde539abb7046f3d67163b20cd1bcaee05c7712a735e4484c\": container with ID starting with 9eee82c4ab4fd13dde539abb7046f3d67163b20cd1bcaee05c7712a735e4484c not found: ID does not exist" containerID="9eee82c4ab4fd13dde539abb7046f3d67163b20cd1bcaee05c7712a735e4484c"
Sep 17 15:37:47.492324 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:47.492277    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eee82c4ab4fd13dde539abb7046f3d67163b20cd1bcaee05c7712a735e4484c"} err="failed to get container status \"9eee82c4ab4fd13dde539abb7046f3d67163b20cd1bcaee05c7712a735e4484c\": rpc error: code = NotFound desc = could not find container \"9eee82c4ab4fd13dde539abb7046f3d67163b20cd1bcaee05c7712a735e4484c\": container with ID starting with 9eee82c4ab4fd13dde539abb7046f3d67163b20cd1bcaee05c7712a735e4484c not found: ID does not exist"
Sep 17 15:37:47.517266 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:47.517241    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-88nxf"]
Sep 17 15:37:47.524702 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:47.524683    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-88nxf"]
Sep 17 15:37:48.138580 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:48.138544    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ab80d4c-1566-4acc-a5a0-57a7aba82eb5" path="/var/lib/kubelet/pods/6ab80d4c-1566-4acc-a5a0-57a7aba82eb5/volumes"
Sep 17 15:37:49.354915 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:49.354885    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-2mqcs"]
Sep 17 15:37:49.355388 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:49.355370    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ab80d4c-1566-4acc-a5a0-57a7aba82eb5" containerName="limitador"
Sep 17 15:37:49.355459 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:49.355392    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ab80d4c-1566-4acc-a5a0-57a7aba82eb5" containerName="limitador"
Sep 17 15:37:49.355509 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:49.355489    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ab80d4c-1566-4acc-a5a0-57a7aba82eb5" containerName="limitador"
Sep 17 15:37:49.359680 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:49.359661    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2mqcs"
Sep 17 15:37:49.362501 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:49.362479    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-cd4bf\""
Sep 17 15:37:49.362829 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:49.362815    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 17 15:37:49.363229 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:49.363216    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 15:37:49.363282 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:49.363255    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 17 15:37:49.371295 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:49.371276    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-2mqcs"]
Sep 17 15:37:49.408594 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:49.408561    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9ba42d1c-e8db-4178-a9db-b6b04b961f72-data\") pod \"postgres-69fb4594cf-2mqcs\" (UID: \"9ba42d1c-e8db-4178-a9db-b6b04b961f72\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2mqcs"
Sep 17 15:37:49.408689 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:49.408601    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6hft\" (UniqueName: \"kubernetes.io/projected/9ba42d1c-e8db-4178-a9db-b6b04b961f72-kube-api-access-b6hft\") pod \"postgres-69fb4594cf-2mqcs\" (UID: \"9ba42d1c-e8db-4178-a9db-b6b04b961f72\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2mqcs"
Sep 17 15:37:49.509437 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:49.509403    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9ba42d1c-e8db-4178-a9db-b6b04b961f72-data\") pod \"postgres-69fb4594cf-2mqcs\" (UID: \"9ba42d1c-e8db-4178-a9db-b6b04b961f72\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2mqcs"
Sep 17 15:37:49.509535 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:49.509453    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6hft\" (UniqueName: \"kubernetes.io/projected/9ba42d1c-e8db-4178-a9db-b6b04b961f72-kube-api-access-b6hft\") pod \"postgres-69fb4594cf-2mqcs\" (UID: \"9ba42d1c-e8db-4178-a9db-b6b04b961f72\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2mqcs"
Sep 17 15:37:49.509737 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:49.509721    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9ba42d1c-e8db-4178-a9db-b6b04b961f72-data\") pod \"postgres-69fb4594cf-2mqcs\" (UID: \"9ba42d1c-e8db-4178-a9db-b6b04b961f72\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2mqcs"
Sep 17 15:37:49.519989 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:49.519965    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6hft\" (UniqueName: \"kubernetes.io/projected/9ba42d1c-e8db-4178-a9db-b6b04b961f72-kube-api-access-b6hft\") pod \"postgres-69fb4594cf-2mqcs\" (UID: \"9ba42d1c-e8db-4178-a9db-b6b04b961f72\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2mqcs"
Sep 17 15:37:49.669780 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:49.669725    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2mqcs"
Sep 17 15:37:49.948562 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:49.948485    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-2mqcs"]
Sep 17 15:37:49.953726 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:37:49.953697    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba42d1c_e8db_4178_a9db_b6b04b961f72.slice/crio-3c4bfb4561b14aba3a74a01393b6a6e0e4c0a3c9546eb48ae11d49bcf8051ff2 WatchSource:0}: Error finding container 3c4bfb4561b14aba3a74a01393b6a6e0e4c0a3c9546eb48ae11d49bcf8051ff2: Status 404 returned error can't find the container with id 3c4bfb4561b14aba3a74a01393b6a6e0e4c0a3c9546eb48ae11d49bcf8051ff2
Sep 17 15:37:50.497852 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:50.497815    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2mqcs" event={"ID":"9ba42d1c-e8db-4178-a9db-b6b04b961f72","Type":"ContainerStarted","Data":"3c4bfb4561b14aba3a74a01393b6a6e0e4c0a3c9546eb48ae11d49bcf8051ff2"}
Sep 17 15:37:55.525917 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:55.525882    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2mqcs" event={"ID":"9ba42d1c-e8db-4178-a9db-b6b04b961f72","Type":"ContainerStarted","Data":"931b137d7d11fad8319ab9e11acb8b50ca65ffb2b7eac3aa397da10a83426d85"}
Sep 17 15:37:55.526314 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:55.526016    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2mqcs"
Sep 17 15:37:55.553769 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:37:55.553722    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2mqcs" podStartSLOduration=1.652418123 podStartE2EDuration="6.55370699s" totalImagesPullingTime="4.901288867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:37:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:37:55.552454215 +0000 UTC m=+497.932471971" watchObservedRunningTime="2026-09-17 15:37:55.55370699 +0000 UTC m=+497.933724744"
Sep 17 15:38:01.557952 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:01.557923    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2mqcs"
Sep 17 15:38:14.070415 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.070382    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-759f779fcd-rdqb5"]
Sep 17 15:38:14.086004 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.085980    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-759f779fcd-rdqb5"
Sep 17 15:38:14.089865 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.089841    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 17 15:38:14.090042 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.090021    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 17 15:38:14.090146 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.090108    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mf6ws\""
Sep 17 15:38:14.090864 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.090845    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 17 15:38:14.093679 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.093656    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-759f779fcd-rdqb5"]
Sep 17 15:38:14.202036 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.202005    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc0d83f7-5d84-417b-a513-6649a7857803-metrics-certs\") pod \"maas-controller-759f779fcd-rdqb5\" (UID: \"cc0d83f7-5d84-417b-a513-6649a7857803\") " pod="opendatahub/maas-controller-759f779fcd-rdqb5"
Sep 17 15:38:14.202267 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.202242    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k26dj\" (UniqueName: \"kubernetes.io/projected/cc0d83f7-5d84-417b-a513-6649a7857803-kube-api-access-k26dj\") pod \"maas-controller-759f779fcd-rdqb5\" (UID: \"cc0d83f7-5d84-417b-a513-6649a7857803\") " pod="opendatahub/maas-controller-759f779fcd-rdqb5"
Sep 17 15:38:14.202368 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.202340    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cc0d83f7-5d84-417b-a513-6649a7857803-webhook-cert\") pod \"maas-controller-759f779fcd-rdqb5\" (UID: \"cc0d83f7-5d84-417b-a513-6649a7857803\") " pod="opendatahub/maas-controller-759f779fcd-rdqb5"
Sep 17 15:38:14.258080 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.258038    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b9f9c646d-lw4ph"]
Sep 17 15:38:14.260193 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.260179    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9f9c646d-lw4ph"
Sep 17 15:38:14.278462 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.278436    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b9f9c646d-lw4ph"]
Sep 17 15:38:14.302991 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.302969    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc0d83f7-5d84-417b-a513-6649a7857803-metrics-certs\") pod \"maas-controller-759f779fcd-rdqb5\" (UID: \"cc0d83f7-5d84-417b-a513-6649a7857803\") " pod="opendatahub/maas-controller-759f779fcd-rdqb5"
Sep 17 15:38:14.303145 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.303126    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k26dj\" (UniqueName: \"kubernetes.io/projected/cc0d83f7-5d84-417b-a513-6649a7857803-kube-api-access-k26dj\") pod \"maas-controller-759f779fcd-rdqb5\" (UID: \"cc0d83f7-5d84-417b-a513-6649a7857803\") " pod="opendatahub/maas-controller-759f779fcd-rdqb5"
Sep 17 15:38:14.303302 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.303280    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cc0d83f7-5d84-417b-a513-6649a7857803-webhook-cert\") pod \"maas-controller-759f779fcd-rdqb5\" (UID: \"cc0d83f7-5d84-417b-a513-6649a7857803\") " pod="opendatahub/maas-controller-759f779fcd-rdqb5"
Sep 17 15:38:14.303437 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:38:14.303401    2556 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 17 15:38:14.303526 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:38:14.303486    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc0d83f7-5d84-417b-a513-6649a7857803-webhook-cert podName:cc0d83f7-5d84-417b-a513-6649a7857803 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:14.803468282 +0000 UTC m=+517.183486021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/cc0d83f7-5d84-417b-a513-6649a7857803-webhook-cert") pod "maas-controller-759f779fcd-rdqb5" (UID: "cc0d83f7-5d84-417b-a513-6649a7857803") : secret "maas-controller-webhook-cert" not found
Sep 17 15:38:14.305299 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.305280    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc0d83f7-5d84-417b-a513-6649a7857803-metrics-certs\") pod \"maas-controller-759f779fcd-rdqb5\" (UID: \"cc0d83f7-5d84-417b-a513-6649a7857803\") " pod="opendatahub/maas-controller-759f779fcd-rdqb5"
Sep 17 15:38:14.313813 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.313793    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k26dj\" (UniqueName: \"kubernetes.io/projected/cc0d83f7-5d84-417b-a513-6649a7857803-kube-api-access-k26dj\") pod \"maas-controller-759f779fcd-rdqb5\" (UID: \"cc0d83f7-5d84-417b-a513-6649a7857803\") " pod="opendatahub/maas-controller-759f779fcd-rdqb5"
Sep 17 15:38:14.404217 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.404187    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgr49\" (UniqueName: \"kubernetes.io/projected/d04bbc03-6252-4a48-80b1-26aaf6956426-kube-api-access-lgr49\") pod \"maas-controller-5b9f9c646d-lw4ph\" (UID: \"d04bbc03-6252-4a48-80b1-26aaf6956426\") " pod="opendatahub/maas-controller-5b9f9c646d-lw4ph"
Sep 17 15:38:14.404330 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.404231    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d04bbc03-6252-4a48-80b1-26aaf6956426-metrics-certs\") pod \"maas-controller-5b9f9c646d-lw4ph\" (UID: \"d04bbc03-6252-4a48-80b1-26aaf6956426\") " pod="opendatahub/maas-controller-5b9f9c646d-lw4ph"
Sep 17 15:38:14.404330 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.404286    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d04bbc03-6252-4a48-80b1-26aaf6956426-webhook-cert\") pod \"maas-controller-5b9f9c646d-lw4ph\" (UID: \"d04bbc03-6252-4a48-80b1-26aaf6956426\") " pod="opendatahub/maas-controller-5b9f9c646d-lw4ph"
Sep 17 15:38:14.504797 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.504768    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d04bbc03-6252-4a48-80b1-26aaf6956426-webhook-cert\") pod \"maas-controller-5b9f9c646d-lw4ph\" (UID: \"d04bbc03-6252-4a48-80b1-26aaf6956426\") " pod="opendatahub/maas-controller-5b9f9c646d-lw4ph"
Sep 17 15:38:14.504911 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.504821    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgr49\" (UniqueName: \"kubernetes.io/projected/d04bbc03-6252-4a48-80b1-26aaf6956426-kube-api-access-lgr49\") pod \"maas-controller-5b9f9c646d-lw4ph\" (UID: \"d04bbc03-6252-4a48-80b1-26aaf6956426\") " pod="opendatahub/maas-controller-5b9f9c646d-lw4ph"
Sep 17 15:38:14.504911 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.504867    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d04bbc03-6252-4a48-80b1-26aaf6956426-metrics-certs\") pod \"maas-controller-5b9f9c646d-lw4ph\" (UID: \"d04bbc03-6252-4a48-80b1-26aaf6956426\") " pod="opendatahub/maas-controller-5b9f9c646d-lw4ph"
Sep 17 15:38:14.507263 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.507236    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d04bbc03-6252-4a48-80b1-26aaf6956426-webhook-cert\") pod \"maas-controller-5b9f9c646d-lw4ph\" (UID: \"d04bbc03-6252-4a48-80b1-26aaf6956426\") " pod="opendatahub/maas-controller-5b9f9c646d-lw4ph"
Sep 17 15:38:14.507351 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.507304    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d04bbc03-6252-4a48-80b1-26aaf6956426-metrics-certs\") pod \"maas-controller-5b9f9c646d-lw4ph\" (UID: \"d04bbc03-6252-4a48-80b1-26aaf6956426\") " pod="opendatahub/maas-controller-5b9f9c646d-lw4ph"
Sep 17 15:38:14.516586 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.516558    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgr49\" (UniqueName: \"kubernetes.io/projected/d04bbc03-6252-4a48-80b1-26aaf6956426-kube-api-access-lgr49\") pod \"maas-controller-5b9f9c646d-lw4ph\" (UID: \"d04bbc03-6252-4a48-80b1-26aaf6956426\") " pod="opendatahub/maas-controller-5b9f9c646d-lw4ph"
Sep 17 15:38:14.569744 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.569721    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9f9c646d-lw4ph"
Sep 17 15:38:14.808450 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.808346    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cc0d83f7-5d84-417b-a513-6649a7857803-webhook-cert\") pod \"maas-controller-759f779fcd-rdqb5\" (UID: \"cc0d83f7-5d84-417b-a513-6649a7857803\") " pod="opendatahub/maas-controller-759f779fcd-rdqb5"
Sep 17 15:38:14.810652 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.810632    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cc0d83f7-5d84-417b-a513-6649a7857803-webhook-cert\") pod \"maas-controller-759f779fcd-rdqb5\" (UID: \"cc0d83f7-5d84-417b-a513-6649a7857803\") " pod="opendatahub/maas-controller-759f779fcd-rdqb5"
Sep 17 15:38:14.844858 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.844836    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b9f9c646d-lw4ph"]
Sep 17 15:38:14.846078 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:38:14.846037    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd04bbc03_6252_4a48_80b1_26aaf6956426.slice/crio-d45c1ed255d4de52e5a866e74230c1f03342bc695b0848141d540f1fd437e5cf WatchSource:0}: Error finding container d45c1ed255d4de52e5a866e74230c1f03342bc695b0848141d540f1fd437e5cf: Status 404 returned error can't find the container with id d45c1ed255d4de52e5a866e74230c1f03342bc695b0848141d540f1fd437e5cf
Sep 17 15:38:14.998405 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:14.998381    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-759f779fcd-rdqb5"
Sep 17 15:38:15.270657 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:15.270588    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-759f779fcd-rdqb5"]
Sep 17 15:38:15.272786 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:38:15.272762    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0d83f7_5d84_417b_a513_6649a7857803.slice/crio-9933b47502a1b01d59a3c2e5d067958fe2cd3e90843549f87be8db739bfdf39d WatchSource:0}: Error finding container 9933b47502a1b01d59a3c2e5d067958fe2cd3e90843549f87be8db739bfdf39d: Status 404 returned error can't find the container with id 9933b47502a1b01d59a3c2e5d067958fe2cd3e90843549f87be8db739bfdf39d
Sep 17 15:38:15.627045 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:15.627007    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-759f779fcd-rdqb5" event={"ID":"cc0d83f7-5d84-417b-a513-6649a7857803","Type":"ContainerStarted","Data":"9933b47502a1b01d59a3c2e5d067958fe2cd3e90843549f87be8db739bfdf39d"}
Sep 17 15:38:15.628705 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:15.628673    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9f9c646d-lw4ph" event={"ID":"d04bbc03-6252-4a48-80b1-26aaf6956426","Type":"ContainerStarted","Data":"d45c1ed255d4de52e5a866e74230c1f03342bc695b0848141d540f1fd437e5cf"}
Sep 17 15:38:17.639370 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:17.639330    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-759f779fcd-rdqb5" event={"ID":"cc0d83f7-5d84-417b-a513-6649a7857803","Type":"ContainerStarted","Data":"07bf293a8e9ba6e4429cea0fe4136768ddc8c8abfdbcf2c01809fd737501763f"}
Sep 17 15:38:17.639880 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:17.639419    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-759f779fcd-rdqb5"
Sep 17 15:38:17.640883 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:17.640862    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9f9c646d-lw4ph" event={"ID":"d04bbc03-6252-4a48-80b1-26aaf6956426","Type":"ContainerStarted","Data":"fdfbf25f2d1880ef09a2b7a1168273ee76a7ffeb90d48c8c45723c8f5ca3c12b"}
Sep 17 15:38:17.641000 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:17.640907    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b9f9c646d-lw4ph"
Sep 17 15:38:17.748004 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:17.747910    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-759f779fcd-rdqb5" podStartSLOduration=1.536132057 podStartE2EDuration="3.747893845s" totalImagesPullingTime="2.211761788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:38:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:38:17.747702281 +0000 UTC m=+520.127720036" watchObservedRunningTime="2026-09-17 15:38:17.747893845 +0000 UTC m=+520.127911600"
Sep 17 15:38:17.821446 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:17.821389    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b9f9c646d-lw4ph" podStartSLOduration=1.189924457 podStartE2EDuration="3.821371867s" totalImagesPullingTime="2.63144741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:38:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:38:17.805694791 +0000 UTC m=+520.185712547" watchObservedRunningTime="2026-09-17 15:38:17.821371867 +0000 UTC m=+520.201389622"
Sep 17 15:38:28.647360 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:28.647325    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-759f779fcd-rdqb5"
Sep 17 15:38:28.647811 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:28.647404    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b9f9c646d-lw4ph"
Sep 17 15:38:28.738260 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:28.738229    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-759f779fcd-rdqb5"]
Sep 17 15:38:28.738441 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:28.738398    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-759f779fcd-rdqb5" podUID="cc0d83f7-5d84-417b-a513-6649a7857803" containerName="manager" containerID="cri-o://07bf293a8e9ba6e4429cea0fe4136768ddc8c8abfdbcf2c01809fd737501763f" gracePeriod=30
Sep 17 15:38:28.978797 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:28.978775    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-759f779fcd-rdqb5"
Sep 17 15:38:29.127588 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:29.127558    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k26dj\" (UniqueName: \"kubernetes.io/projected/cc0d83f7-5d84-417b-a513-6649a7857803-kube-api-access-k26dj\") pod \"cc0d83f7-5d84-417b-a513-6649a7857803\" (UID: \"cc0d83f7-5d84-417b-a513-6649a7857803\") "
Sep 17 15:38:29.127740 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:29.127631    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc0d83f7-5d84-417b-a513-6649a7857803-metrics-certs\") pod \"cc0d83f7-5d84-417b-a513-6649a7857803\" (UID: \"cc0d83f7-5d84-417b-a513-6649a7857803\") "
Sep 17 15:38:29.127740 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:29.127673    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cc0d83f7-5d84-417b-a513-6649a7857803-webhook-cert\") pod \"cc0d83f7-5d84-417b-a513-6649a7857803\" (UID: \"cc0d83f7-5d84-417b-a513-6649a7857803\") "
Sep 17 15:38:29.129750 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:29.129721    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc0d83f7-5d84-417b-a513-6649a7857803-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "cc0d83f7-5d84-417b-a513-6649a7857803" (UID: "cc0d83f7-5d84-417b-a513-6649a7857803"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:38:29.129750 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:29.129738    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc0d83f7-5d84-417b-a513-6649a7857803-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cc0d83f7-5d84-417b-a513-6649a7857803" (UID: "cc0d83f7-5d84-417b-a513-6649a7857803"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:38:29.129889 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:29.129786    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc0d83f7-5d84-417b-a513-6649a7857803-kube-api-access-k26dj" (OuterVolumeSpecName: "kube-api-access-k26dj") pod "cc0d83f7-5d84-417b-a513-6649a7857803" (UID: "cc0d83f7-5d84-417b-a513-6649a7857803"). InnerVolumeSpecName "kube-api-access-k26dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:38:29.228600 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:29.228530    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc0d83f7-5d84-417b-a513-6649a7857803-metrics-certs\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:38:29.228600 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:29.228553    2556 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cc0d83f7-5d84-417b-a513-6649a7857803-webhook-cert\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:38:29.228600 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:29.228563    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k26dj\" (UniqueName: \"kubernetes.io/projected/cc0d83f7-5d84-417b-a513-6649a7857803-kube-api-access-k26dj\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:38:29.696657 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:29.696624    2556 generic.go:356] "Generic (PLEG): container finished" podID="cc0d83f7-5d84-417b-a513-6649a7857803" containerID="07bf293a8e9ba6e4429cea0fe4136768ddc8c8abfdbcf2c01809fd737501763f" exitCode=0
Sep 17 15:38:29.697080 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:29.696660    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-759f779fcd-rdqb5"
Sep 17 15:38:29.697080 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:29.696694    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-759f779fcd-rdqb5" event={"ID":"cc0d83f7-5d84-417b-a513-6649a7857803","Type":"ContainerDied","Data":"07bf293a8e9ba6e4429cea0fe4136768ddc8c8abfdbcf2c01809fd737501763f"}
Sep 17 15:38:29.697080 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:29.696753    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-759f779fcd-rdqb5" event={"ID":"cc0d83f7-5d84-417b-a513-6649a7857803","Type":"ContainerDied","Data":"9933b47502a1b01d59a3c2e5d067958fe2cd3e90843549f87be8db739bfdf39d"}
Sep 17 15:38:29.697080 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:29.696780    2556 scope.go:117] "RemoveContainer" containerID="07bf293a8e9ba6e4429cea0fe4136768ddc8c8abfdbcf2c01809fd737501763f"
Sep 17 15:38:29.706197 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:29.706180    2556 scope.go:117] "RemoveContainer" containerID="07bf293a8e9ba6e4429cea0fe4136768ddc8c8abfdbcf2c01809fd737501763f"
Sep 17 15:38:29.706477 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:38:29.706462    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07bf293a8e9ba6e4429cea0fe4136768ddc8c8abfdbcf2c01809fd737501763f\": container with ID starting with 07bf293a8e9ba6e4429cea0fe4136768ddc8c8abfdbcf2c01809fd737501763f not found: ID does not exist" containerID="07bf293a8e9ba6e4429cea0fe4136768ddc8c8abfdbcf2c01809fd737501763f"
Sep 17 15:38:29.706517 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:29.706484    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07bf293a8e9ba6e4429cea0fe4136768ddc8c8abfdbcf2c01809fd737501763f"} err="failed to get container status \"07bf293a8e9ba6e4429cea0fe4136768ddc8c8abfdbcf2c01809fd737501763f\": rpc error: code = NotFound desc = could not find container \"07bf293a8e9ba6e4429cea0fe4136768ddc8c8abfdbcf2c01809fd737501763f\": container with ID starting with 07bf293a8e9ba6e4429cea0fe4136768ddc8c8abfdbcf2c01809fd737501763f not found: ID does not exist"
Sep 17 15:38:29.734545 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:29.734524    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-759f779fcd-rdqb5"]
Sep 17 15:38:29.743764 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:29.743745    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-759f779fcd-rdqb5"]
Sep 17 15:38:30.135369 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:30.135340    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc0d83f7-5d84-417b-a513-6649a7857803" path="/var/lib/kubelet/pods/cc0d83f7-5d84-417b-a513-6649a7857803/volumes"
Sep 17 15:38:51.032240 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.032203    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh"]
Sep 17 15:38:51.032693 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.032568    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc0d83f7-5d84-417b-a513-6649a7857803" containerName="manager"
Sep 17 15:38:51.032693 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.032580    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc0d83f7-5d84-417b-a513-6649a7857803" containerName="manager"
Sep 17 15:38:51.032693 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.032646    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc0d83f7-5d84-417b-a513-6649a7857803" containerName="manager"
Sep 17 15:38:51.037003 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.036983    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh"
Sep 17 15:38:51.039384 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.039360    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 17 15:38:51.039615 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.039603    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 17 15:38:51.040255 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.040237    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mdtj9\""
Sep 17 15:38:51.048148 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.048128    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh"]
Sep 17 15:38:51.092111 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.092082    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca972d44-9d36-420c-a530-aa9d6add83ac-maas-api-tls\") pod \"maas-api-d5fdc6457-2swjh\" (UID: \"ca972d44-9d36-420c-a530-aa9d6add83ac\") " pod="odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh"
Sep 17 15:38:51.092240 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.092124    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca972d44-9d36-420c-a530-aa9d6add83ac-metrics-certs\") pod \"maas-api-d5fdc6457-2swjh\" (UID: \"ca972d44-9d36-420c-a530-aa9d6add83ac\") " pod="odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh"
Sep 17 15:38:51.092240 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.092150    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca972d44-9d36-420c-a530-aa9d6add83ac-openshift-service-ca\") pod \"maas-api-d5fdc6457-2swjh\" (UID: \"ca972d44-9d36-420c-a530-aa9d6add83ac\") " pod="odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh"
Sep 17 15:38:51.092329 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.092244    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf4nt\" (UniqueName: \"kubernetes.io/projected/ca972d44-9d36-420c-a530-aa9d6add83ac-kube-api-access-pf4nt\") pod \"maas-api-d5fdc6457-2swjh\" (UID: \"ca972d44-9d36-420c-a530-aa9d6add83ac\") " pod="odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh"
Sep 17 15:38:51.192771 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.192737    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca972d44-9d36-420c-a530-aa9d6add83ac-openshift-service-ca\") pod \"maas-api-d5fdc6457-2swjh\" (UID: \"ca972d44-9d36-420c-a530-aa9d6add83ac\") " pod="odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh"
Sep 17 15:38:51.192934 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.192790    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf4nt\" (UniqueName: \"kubernetes.io/projected/ca972d44-9d36-420c-a530-aa9d6add83ac-kube-api-access-pf4nt\") pod \"maas-api-d5fdc6457-2swjh\" (UID: \"ca972d44-9d36-420c-a530-aa9d6add83ac\") " pod="odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh"
Sep 17 15:38:51.192934 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.192844    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca972d44-9d36-420c-a530-aa9d6add83ac-maas-api-tls\") pod \"maas-api-d5fdc6457-2swjh\" (UID: \"ca972d44-9d36-420c-a530-aa9d6add83ac\") " pod="odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh"
Sep 17 15:38:51.192934 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.192882    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca972d44-9d36-420c-a530-aa9d6add83ac-metrics-certs\") pod \"maas-api-d5fdc6457-2swjh\" (UID: \"ca972d44-9d36-420c-a530-aa9d6add83ac\") " pod="odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh"
Sep 17 15:38:51.193110 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:38:51.192980    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 17 15:38:51.193110 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:38:51.193083    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca972d44-9d36-420c-a530-aa9d6add83ac-metrics-certs podName:ca972d44-9d36-420c-a530-aa9d6add83ac nodeName:}" failed. No retries permitted until 2026-09-17 15:38:51.693037274 +0000 UTC m=+554.073055015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca972d44-9d36-420c-a530-aa9d6add83ac-metrics-certs") pod "maas-api-d5fdc6457-2swjh" (UID: "ca972d44-9d36-420c-a530-aa9d6add83ac") : secret "maas-api-metrics-tls" not found
Sep 17 15:38:51.193439 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.193418    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca972d44-9d36-420c-a530-aa9d6add83ac-openshift-service-ca\") pod \"maas-api-d5fdc6457-2swjh\" (UID: \"ca972d44-9d36-420c-a530-aa9d6add83ac\") " pod="odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh"
Sep 17 15:38:51.195429 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.195403    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca972d44-9d36-420c-a530-aa9d6add83ac-maas-api-tls\") pod \"maas-api-d5fdc6457-2swjh\" (UID: \"ca972d44-9d36-420c-a530-aa9d6add83ac\") " pod="odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh"
Sep 17 15:38:51.201773 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.201753    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf4nt\" (UniqueName: \"kubernetes.io/projected/ca972d44-9d36-420c-a530-aa9d6add83ac-kube-api-access-pf4nt\") pod \"maas-api-d5fdc6457-2swjh\" (UID: \"ca972d44-9d36-420c-a530-aa9d6add83ac\") " pod="odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh"
Sep 17 15:38:51.595071 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.595028    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-btzhg"]
Sep 17 15:38:51.597535 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.597512    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-btzhg"
Sep 17 15:38:51.602525 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.602503    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 17 15:38:51.616022 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.615999    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-btzhg"]
Sep 17 15:38:51.642170 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.642145    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-2wlv5"]
Sep 17 15:38:51.646605 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.646589    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-2wlv5"
Sep 17 15:38:51.655816 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.655795    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-6krbs\""
Sep 17 15:38:51.675099 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.675079    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-2wlv5"]
Sep 17 15:38:51.696327 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.696302    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nncg\" (UniqueName: \"kubernetes.io/projected/2ec252e5-1112-40e2-b579-966af538ac4f-kube-api-access-9nncg\") pod \"payload-pre-processing-59fbc47bf8-btzhg\" (UID: \"2ec252e5-1112-40e2-b579-966af538ac4f\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-btzhg"
Sep 17 15:38:51.696419 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.696346    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec252e5-1112-40e2-b579-966af538ac4f-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-btzhg\" (UID: \"2ec252e5-1112-40e2-b579-966af538ac4f\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-btzhg"
Sep 17 15:38:51.696419 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.696369    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8ll2\" (UniqueName: \"kubernetes.io/projected/5bd2083c-fee3-4da3-95dd-1be423dab576-kube-api-access-m8ll2\") pod \"payload-processing-5b54b7fbb7-2wlv5\" (UID: \"5bd2083c-fee3-4da3-95dd-1be423dab576\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-2wlv5"
Sep 17 15:38:51.696494 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.696472    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca972d44-9d36-420c-a530-aa9d6add83ac-metrics-certs\") pod \"maas-api-d5fdc6457-2swjh\" (UID: \"ca972d44-9d36-420c-a530-aa9d6add83ac\") " pod="odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh"
Sep 17 15:38:51.696555 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.696537    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bd2083c-fee3-4da3-95dd-1be423dab576-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-2wlv5\" (UID: \"5bd2083c-fee3-4da3-95dd-1be423dab576\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-2wlv5"
Sep 17 15:38:51.698761 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.698738    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca972d44-9d36-420c-a530-aa9d6add83ac-metrics-certs\") pod \"maas-api-d5fdc6457-2swjh\" (UID: \"ca972d44-9d36-420c-a530-aa9d6add83ac\") " pod="odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh"
Sep 17 15:38:51.797461 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.797435    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nncg\" (UniqueName: \"kubernetes.io/projected/2ec252e5-1112-40e2-b579-966af538ac4f-kube-api-access-9nncg\") pod \"payload-pre-processing-59fbc47bf8-btzhg\" (UID: \"2ec252e5-1112-40e2-b579-966af538ac4f\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-btzhg"
Sep 17 15:38:51.797588 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.797475    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec252e5-1112-40e2-b579-966af538ac4f-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-btzhg\" (UID: \"2ec252e5-1112-40e2-b579-966af538ac4f\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-btzhg"
Sep 17 15:38:51.797588 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.797490    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ll2\" (UniqueName: \"kubernetes.io/projected/5bd2083c-fee3-4da3-95dd-1be423dab576-kube-api-access-m8ll2\") pod \"payload-processing-5b54b7fbb7-2wlv5\" (UID: \"5bd2083c-fee3-4da3-95dd-1be423dab576\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-2wlv5"
Sep 17 15:38:51.797588 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.797538    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bd2083c-fee3-4da3-95dd-1be423dab576-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-2wlv5\" (UID: \"5bd2083c-fee3-4da3-95dd-1be423dab576\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-2wlv5"
Sep 17 15:38:51.798146 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.798127    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec252e5-1112-40e2-b579-966af538ac4f-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-btzhg\" (UID: \"2ec252e5-1112-40e2-b579-966af538ac4f\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-btzhg"
Sep 17 15:38:51.798226 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.798208    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bd2083c-fee3-4da3-95dd-1be423dab576-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-2wlv5\" (UID: \"5bd2083c-fee3-4da3-95dd-1be423dab576\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-2wlv5"
Sep 17 15:38:51.808610 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.808583    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nncg\" (UniqueName: \"kubernetes.io/projected/2ec252e5-1112-40e2-b579-966af538ac4f-kube-api-access-9nncg\") pod \"payload-pre-processing-59fbc47bf8-btzhg\" (UID: \"2ec252e5-1112-40e2-b579-966af538ac4f\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-btzhg"
Sep 17 15:38:51.812109 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.812090    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8ll2\" (UniqueName: \"kubernetes.io/projected/5bd2083c-fee3-4da3-95dd-1be423dab576-kube-api-access-m8ll2\") pod \"payload-processing-5b54b7fbb7-2wlv5\" (UID: \"5bd2083c-fee3-4da3-95dd-1be423dab576\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-2wlv5"
Sep 17 15:38:51.907826 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.907800    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-btzhg"
Sep 17 15:38:51.947043 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.947016    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh"
Sep 17 15:38:51.956248 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:51.956040    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-2wlv5"
Sep 17 15:38:52.205361 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:52.205275    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-btzhg"]
Sep 17 15:38:52.210046 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:38:52.210018    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec252e5_1112_40e2_b579_966af538ac4f.slice/crio-90ab005fade7841846a7e8bb1e46b3bcef6b7784b42e7f767f6ddbc60098a657 WatchSource:0}: Error finding container 90ab005fade7841846a7e8bb1e46b3bcef6b7784b42e7f767f6ddbc60098a657: Status 404 returned error can't find the container with id 90ab005fade7841846a7e8bb1e46b3bcef6b7784b42e7f767f6ddbc60098a657
Sep 17 15:38:52.263611 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:52.263591    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh"]
Sep 17 15:38:52.266151 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:38:52.266127    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca972d44_9d36_420c_a530_aa9d6add83ac.slice/crio-73e8cbcb5e775921be153009f05daccbb6b9e27ab7f4bc8badaaab4438fb4a38 WatchSource:0}: Error finding container 73e8cbcb5e775921be153009f05daccbb6b9e27ab7f4bc8badaaab4438fb4a38: Status 404 returned error can't find the container with id 73e8cbcb5e775921be153009f05daccbb6b9e27ab7f4bc8badaaab4438fb4a38
Sep 17 15:38:52.272657 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:52.272621    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-2wlv5"]
Sep 17 15:38:52.274110 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:38:52.274085    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd2083c_fee3_4da3_95dd_1be423dab576.slice/crio-02c6d5162fd52ccf745afcc7f07956a8ead7979fe8ee7eafbeb265dcd74ba3d2 WatchSource:0}: Error finding container 02c6d5162fd52ccf745afcc7f07956a8ead7979fe8ee7eafbeb265dcd74ba3d2: Status 404 returned error can't find the container with id 02c6d5162fd52ccf745afcc7f07956a8ead7979fe8ee7eafbeb265dcd74ba3d2
Sep 17 15:38:52.803870 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:52.803840    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-btzhg" event={"ID":"2ec252e5-1112-40e2-b579-966af538ac4f","Type":"ContainerStarted","Data":"90ab005fade7841846a7e8bb1e46b3bcef6b7784b42e7f767f6ddbc60098a657"}
Sep 17 15:38:52.805221 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:52.805185    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-2wlv5" event={"ID":"5bd2083c-fee3-4da3-95dd-1be423dab576","Type":"ContainerStarted","Data":"02c6d5162fd52ccf745afcc7f07956a8ead7979fe8ee7eafbeb265dcd74ba3d2"}
Sep 17 15:38:52.806360 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:52.806337    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh" event={"ID":"ca972d44-9d36-420c-a530-aa9d6add83ac","Type":"ContainerStarted","Data":"73e8cbcb5e775921be153009f05daccbb6b9e27ab7f4bc8badaaab4438fb4a38"}
Sep 17 15:38:55.826745 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:55.826703    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-2wlv5" event={"ID":"5bd2083c-fee3-4da3-95dd-1be423dab576","Type":"ContainerStarted","Data":"4a7643bc51c37590e8595c5e36e06d59c766158c6067102e8bd7df9c5d07c40d"}
Sep 17 15:38:55.827232 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:55.826907    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-2wlv5"
Sep 17 15:38:55.828401 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:55.828377    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh" event={"ID":"ca972d44-9d36-420c-a530-aa9d6add83ac","Type":"ContainerStarted","Data":"27b9efe78d0ec174bea518226d8b6416fe4012b8e48a1da4fbf056a710e21474"}
Sep 17 15:38:55.828504 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:55.828456    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh"
Sep 17 15:38:55.829941 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:55.829922    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-btzhg" event={"ID":"2ec252e5-1112-40e2-b579-966af538ac4f","Type":"ContainerStarted","Data":"7e82ee2a6d888e48afaf5902c2c2e6fee77aa97853a2135c4f7733fb4d114617"}
Sep 17 15:38:55.852001 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:55.851953    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-2wlv5" podStartSLOduration=1.895894067 podStartE2EDuration="4.851938697s" totalImagesPullingTime="2.95604463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:38:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:38:55.851165936 +0000 UTC m=+558.231183693" watchObservedRunningTime="2026-09-17 15:38:55.851938697 +0000 UTC m=+558.231956451"
Sep 17 15:38:55.872793 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:55.872748    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-btzhg" podStartSLOduration=1.8473032310000002 podStartE2EDuration="4.872731583s" totalImagesPullingTime="3.025428352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:38:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:38:55.870822113 +0000 UTC m=+558.250839870" watchObservedRunningTime="2026-09-17 15:38:55.872731583 +0000 UTC m=+558.252749343"
Sep 17 15:38:56.834532 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:38:56.834499    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-btzhg"
Sep 17 15:39:01.839599 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:01.839572    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh"
Sep 17 15:39:01.864037 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:01.863992    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh" podStartSLOduration=7.94196732 podStartE2EDuration="10.863975558s" totalImagesPullingTime="2.922008238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:38:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:38:55.894654169 +0000 UTC m=+558.274671923" watchObservedRunningTime="2026-09-17 15:39:01.863975558 +0000 UTC m=+564.243993312"
Sep 17 15:39:06.836265 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:06.836228    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-2wlv5"
Sep 17 15:39:07.841622 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:07.841592    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-btzhg"
Sep 17 15:39:13.872207 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:13.872174    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4"]
Sep 17 15:39:13.875687 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:13.875668    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4"
Sep 17 15:39:13.885473 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:13.885455    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 17 15:39:13.885794 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:13.885776    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 17 15:39:13.885893 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:13.885872    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ndjls\""
Sep 17 15:39:13.894522 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:13.894497    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 17 15:39:13.896331 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:13.896309    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4"]
Sep 17 15:39:13.974111 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:13.974084    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebc729c3-e744-442a-afef-a32261d00baf-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4\" (UID: \"ebc729c3-e744-442a-afef-a32261d00baf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4"
Sep 17 15:39:13.974232 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:13.974126    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebc729c3-e744-442a-afef-a32261d00baf-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4\" (UID: \"ebc729c3-e744-442a-afef-a32261d00baf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4"
Sep 17 15:39:13.974232 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:13.974141    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebc729c3-e744-442a-afef-a32261d00baf-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4\" (UID: \"ebc729c3-e744-442a-afef-a32261d00baf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4"
Sep 17 15:39:13.974232 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:13.974158    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qkjp\" (UniqueName: \"kubernetes.io/projected/ebc729c3-e744-442a-afef-a32261d00baf-kube-api-access-9qkjp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4\" (UID: \"ebc729c3-e744-442a-afef-a32261d00baf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4"
Sep 17 15:39:13.974232 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:13.974207    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebc729c3-e744-442a-afef-a32261d00baf-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4\" (UID: \"ebc729c3-e744-442a-afef-a32261d00baf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4"
Sep 17 15:39:13.974355 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:13.974256    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebc729c3-e744-442a-afef-a32261d00baf-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4\" (UID: \"ebc729c3-e744-442a-afef-a32261d00baf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4"
Sep 17 15:39:14.074671 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:14.074640    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebc729c3-e744-442a-afef-a32261d00baf-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4\" (UID: \"ebc729c3-e744-442a-afef-a32261d00baf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4"
Sep 17 15:39:14.074671 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:14.074671    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebc729c3-e744-442a-afef-a32261d00baf-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4\" (UID: \"ebc729c3-e744-442a-afef-a32261d00baf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4"
Sep 17 15:39:14.074899 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:14.074690    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qkjp\" (UniqueName: \"kubernetes.io/projected/ebc729c3-e744-442a-afef-a32261d00baf-kube-api-access-9qkjp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4\" (UID: \"ebc729c3-e744-442a-afef-a32261d00baf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4"
Sep 17 15:39:14.074899 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:14.074713    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebc729c3-e744-442a-afef-a32261d00baf-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4\" (UID: \"ebc729c3-e744-442a-afef-a32261d00baf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4"
Sep 17 15:39:14.074899 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:14.074759    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebc729c3-e744-442a-afef-a32261d00baf-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4\" (UID: \"ebc729c3-e744-442a-afef-a32261d00baf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4"
Sep 17 15:39:14.074899 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:14.074785    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebc729c3-e744-442a-afef-a32261d00baf-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4\" (UID: \"ebc729c3-e744-442a-afef-a32261d00baf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4"
Sep 17 15:39:14.075141 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:14.075116    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebc729c3-e744-442a-afef-a32261d00baf-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4\" (UID: \"ebc729c3-e744-442a-afef-a32261d00baf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4"
Sep 17 15:39:14.075189 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:14.075128    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebc729c3-e744-442a-afef-a32261d00baf-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4\" (UID: \"ebc729c3-e744-442a-afef-a32261d00baf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4"
Sep 17 15:39:14.075189 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:14.075180    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebc729c3-e744-442a-afef-a32261d00baf-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4\" (UID: \"ebc729c3-e744-442a-afef-a32261d00baf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4"
Sep 17 15:39:14.077085 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:14.077047    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebc729c3-e744-442a-afef-a32261d00baf-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4\" (UID: \"ebc729c3-e744-442a-afef-a32261d00baf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4"
Sep 17 15:39:14.077393 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:14.077372    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebc729c3-e744-442a-afef-a32261d00baf-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4\" (UID: \"ebc729c3-e744-442a-afef-a32261d00baf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4"
Sep 17 15:39:14.085353 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:14.085332    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qkjp\" (UniqueName: \"kubernetes.io/projected/ebc729c3-e744-442a-afef-a32261d00baf-kube-api-access-9qkjp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4\" (UID: \"ebc729c3-e744-442a-afef-a32261d00baf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4"
Sep 17 15:39:14.185150 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:14.185080    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4"
Sep 17 15:39:14.457319 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:14.457290    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4"]
Sep 17 15:39:14.459103 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:39:14.459047    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc729c3_e744_442a_afef_a32261d00baf.slice/crio-5c4a42209b9c7fbc1727e56d8230ddb778c39044bdc4c4bf7aca988b10e25be9 WatchSource:0}: Error finding container 5c4a42209b9c7fbc1727e56d8230ddb778c39044bdc4c4bf7aca988b10e25be9: Status 404 returned error can't find the container with id 5c4a42209b9c7fbc1727e56d8230ddb778c39044bdc4c4bf7aca988b10e25be9
Sep 17 15:39:14.921132 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:14.921074    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4" event={"ID":"ebc729c3-e744-442a-afef-a32261d00baf","Type":"ContainerStarted","Data":"5c4a42209b9c7fbc1727e56d8230ddb778c39044bdc4c4bf7aca988b10e25be9"}
Sep 17 15:39:15.927851 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:15.927815    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4" event={"ID":"ebc729c3-e744-442a-afef-a32261d00baf","Type":"ContainerStarted","Data":"e767d85b409d42bf84e58c63f1b35b9434d9c04e70916b0c7a9f88c09c95602d"}
Sep 17 15:39:15.967269 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:15.967226    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4" podStartSLOduration=1.850014398 podStartE2EDuration="2.967212198s" totalImagesPullingTime="1.1171978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:15.962850641 +0000 UTC m=+578.342868396" watchObservedRunningTime="2026-09-17 15:39:15.967212198 +0000 UTC m=+578.347229952"
Sep 17 15:39:24.185602 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:24.185564    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4"
Sep 17 15:39:24.186220 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:24.185624    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4"
Sep 17 15:39:24.197284 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:24.197262    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4"
Sep 17 15:39:24.978571 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:24.978541    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4"
Sep 17 15:39:38.074541 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:38.074508    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-45zt7_2c069c9e-0856-46b0-ba10-67829c05ac94/console-operator/1.log"
Sep 17 15:39:38.078760 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:38.078734    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-45zt7_2c069c9e-0856-46b0-ba10-67829c05ac94/console-operator/1.log"
Sep 17 15:39:50.516700 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:50.516667    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t"]
Sep 17 15:39:50.522448 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:50.522429    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t"
Sep 17 15:39:50.524773 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:50.524746    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 17 15:39:50.555661 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:50.555627    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t"]
Sep 17 15:39:50.660801 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:50.660767    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdz2t\" (UID: \"7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t"
Sep 17 15:39:50.660801 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:50.660801    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdq95\" (UniqueName: \"kubernetes.io/projected/7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f-kube-api-access-xdq95\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdz2t\" (UID: \"7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t"
Sep 17 15:39:50.661036 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:50.660830    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdz2t\" (UID: \"7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t"
Sep 17 15:39:50.661036 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:50.660957    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdz2t\" (UID: \"7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t"
Sep 17 15:39:50.661176 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:50.661037    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdz2t\" (UID: \"7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t"
Sep 17 15:39:50.661176 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:50.661090    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdz2t\" (UID: \"7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t"
Sep 17 15:39:50.762579 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:50.762542    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdz2t\" (UID: \"7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t"
Sep 17 15:39:50.762741 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:50.762585    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdz2t\" (UID: \"7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t"
Sep 17 15:39:50.762741 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:50.762701    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdz2t\" (UID: \"7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t"
Sep 17 15:39:50.762850 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:50.762742    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdz2t\" (UID: \"7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t"
Sep 17 15:39:50.762850 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:50.762834    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdz2t\" (UID: \"7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t"
Sep 17 15:39:50.762957 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:50.762862    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdq95\" (UniqueName: \"kubernetes.io/projected/7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f-kube-api-access-xdq95\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdz2t\" (UID: \"7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t"
Sep 17 15:39:50.762957 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:50.762897    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdz2t\" (UID: \"7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t"
Sep 17 15:39:50.763037 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:50.762963    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdz2t\" (UID: \"7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t"
Sep 17 15:39:50.763172 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:50.763152    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdz2t\" (UID: \"7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t"
Sep 17 15:39:50.764838 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:50.764812    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdz2t\" (UID: \"7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t"
Sep 17 15:39:50.765187 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:50.765168    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdz2t\" (UID: \"7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t"
Sep 17 15:39:50.794823 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:50.794757    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdq95\" (UniqueName: \"kubernetes.io/projected/7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f-kube-api-access-xdq95\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sdz2t\" (UID: \"7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t"
Sep 17 15:39:50.832276 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:50.832253    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t"
Sep 17 15:39:51.123724 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:51.123693    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t"]
Sep 17 15:39:51.126678 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:39:51.126651    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e21c1f2_ff9e_4eaf_85ef_fe9fbd40860f.slice/crio-60c73668bec95a5fe82f1581916828d55c75f963c75bd3035d466b64d57ffd96 WatchSource:0}: Error finding container 60c73668bec95a5fe82f1581916828d55c75f963c75bd3035d466b64d57ffd96: Status 404 returned error can't find the container with id 60c73668bec95a5fe82f1581916828d55c75f963c75bd3035d466b64d57ffd96
Sep 17 15:39:52.091879 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:52.091831    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t" event={"ID":"7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f","Type":"ContainerStarted","Data":"e27449b3400eba15608e872bf6dbc438bd2cf7ce4ca7ba012fb3e928cf61441a"}
Sep 17 15:39:52.092363 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:52.091893    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t" event={"ID":"7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f","Type":"ContainerStarted","Data":"60c73668bec95a5fe82f1581916828d55c75f963c75bd3035d466b64d57ffd96"}
Sep 17 15:39:52.123980 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:39:52.123935    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t" podStartSLOduration=1.916026471 podStartE2EDuration="2.123920533s" totalImagesPullingTime="207.894062ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:52.123591065 +0000 UTC m=+614.503608847" watchObservedRunningTime="2026-09-17 15:39:52.123920533 +0000 UTC m=+614.503938287"
Sep 17 15:40:00.832828 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:00.832741    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t"
Sep 17 15:40:00.833432 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:00.832913    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t"
Sep 17 15:40:00.845171 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:00.845154    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t"
Sep 17 15:40:01.144147 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:01.144121    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sdz2t"
Sep 17 15:40:01.548921 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:01.548845    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd"]
Sep 17 15:40:01.553588 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:01.553563    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd"
Sep 17 15:40:01.556120 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:01.556034    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 17 15:40:01.567690 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:01.567666    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd"]
Sep 17 15:40:01.743420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:01.743384    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd6f7039-09e8-4a44-9fe9-212ee9d8b772-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd\" (UID: \"cd6f7039-09e8-4a44-9fe9-212ee9d8b772\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd"
Sep 17 15:40:01.743420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:01.743418    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drfjp\" (UniqueName: \"kubernetes.io/projected/cd6f7039-09e8-4a44-9fe9-212ee9d8b772-kube-api-access-drfjp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd\" (UID: \"cd6f7039-09e8-4a44-9fe9-212ee9d8b772\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd"
Sep 17 15:40:01.743611 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:01.743450    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6f7039-09e8-4a44-9fe9-212ee9d8b772-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd\" (UID: \"cd6f7039-09e8-4a44-9fe9-212ee9d8b772\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd"
Sep 17 15:40:01.743611 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:01.743498    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd6f7039-09e8-4a44-9fe9-212ee9d8b772-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd\" (UID: \"cd6f7039-09e8-4a44-9fe9-212ee9d8b772\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd"
Sep 17 15:40:01.743611 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:01.743521    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd6f7039-09e8-4a44-9fe9-212ee9d8b772-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd\" (UID: \"cd6f7039-09e8-4a44-9fe9-212ee9d8b772\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd"
Sep 17 15:40:01.743611 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:01.743564    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd6f7039-09e8-4a44-9fe9-212ee9d8b772-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd\" (UID: \"cd6f7039-09e8-4a44-9fe9-212ee9d8b772\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd"
Sep 17 15:40:01.844205 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:01.844128    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6f7039-09e8-4a44-9fe9-212ee9d8b772-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd\" (UID: \"cd6f7039-09e8-4a44-9fe9-212ee9d8b772\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd"
Sep 17 15:40:01.844205 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:01.844163    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd6f7039-09e8-4a44-9fe9-212ee9d8b772-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd\" (UID: \"cd6f7039-09e8-4a44-9fe9-212ee9d8b772\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd"
Sep 17 15:40:01.844205 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:01.844188    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd6f7039-09e8-4a44-9fe9-212ee9d8b772-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd\" (UID: \"cd6f7039-09e8-4a44-9fe9-212ee9d8b772\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd"
Sep 17 15:40:01.844753 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:01.844225    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd6f7039-09e8-4a44-9fe9-212ee9d8b772-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd\" (UID: \"cd6f7039-09e8-4a44-9fe9-212ee9d8b772\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd"
Sep 17 15:40:01.844753 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:01.844290    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd6f7039-09e8-4a44-9fe9-212ee9d8b772-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd\" (UID: \"cd6f7039-09e8-4a44-9fe9-212ee9d8b772\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd"
Sep 17 15:40:01.844753 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:01.844315    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drfjp\" (UniqueName: \"kubernetes.io/projected/cd6f7039-09e8-4a44-9fe9-212ee9d8b772-kube-api-access-drfjp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd\" (UID: \"cd6f7039-09e8-4a44-9fe9-212ee9d8b772\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd"
Sep 17 15:40:01.844753 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:01.844612    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6f7039-09e8-4a44-9fe9-212ee9d8b772-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd\" (UID: \"cd6f7039-09e8-4a44-9fe9-212ee9d8b772\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd"
Sep 17 15:40:01.844753 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:01.844662    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd6f7039-09e8-4a44-9fe9-212ee9d8b772-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd\" (UID: \"cd6f7039-09e8-4a44-9fe9-212ee9d8b772\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd"
Sep 17 15:40:01.844928 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:01.844753    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd6f7039-09e8-4a44-9fe9-212ee9d8b772-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd\" (UID: \"cd6f7039-09e8-4a44-9fe9-212ee9d8b772\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd"
Sep 17 15:40:01.846591 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:01.846570    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd6f7039-09e8-4a44-9fe9-212ee9d8b772-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd\" (UID: \"cd6f7039-09e8-4a44-9fe9-212ee9d8b772\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd"
Sep 17 15:40:01.846789 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:01.846771    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd6f7039-09e8-4a44-9fe9-212ee9d8b772-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd\" (UID: \"cd6f7039-09e8-4a44-9fe9-212ee9d8b772\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd"
Sep 17 15:40:01.856191 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:01.856169    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drfjp\" (UniqueName: \"kubernetes.io/projected/cd6f7039-09e8-4a44-9fe9-212ee9d8b772-kube-api-access-drfjp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd\" (UID: \"cd6f7039-09e8-4a44-9fe9-212ee9d8b772\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd"
Sep 17 15:40:01.864020 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:01.863997    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd"
Sep 17 15:40:02.137715 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:02.137693    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd"]
Sep 17 15:40:02.140272 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:40:02.140247    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd6f7039_09e8_4a44_9fe9_212ee9d8b772.slice/crio-c17e74a7fcafd3c509ebae49fc8c14d49ea7dfc93f7587bb69a2c83a43a44134 WatchSource:0}: Error finding container c17e74a7fcafd3c509ebae49fc8c14d49ea7dfc93f7587bb69a2c83a43a44134: Status 404 returned error can't find the container with id c17e74a7fcafd3c509ebae49fc8c14d49ea7dfc93f7587bb69a2c83a43a44134
Sep 17 15:40:02.141535 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:02.141519    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:40:03.145494 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:03.145458    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd" event={"ID":"cd6f7039-09e8-4a44-9fe9-212ee9d8b772","Type":"ContainerStarted","Data":"28a0065927e628e986c5e03b3b8441389fd65d089937a651d045ecc9c86fec43"}
Sep 17 15:40:03.145923 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:03.145513    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd" event={"ID":"cd6f7039-09e8-4a44-9fe9-212ee9d8b772","Type":"ContainerStarted","Data":"c17e74a7fcafd3c509ebae49fc8c14d49ea7dfc93f7587bb69a2c83a43a44134"}
Sep 17 15:40:03.166261 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:03.166217    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd" podStartSLOduration=1.956045252 podStartE2EDuration="2.166202418s" totalImagesPullingTime="210.157166ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:03.164388586 +0000 UTC m=+625.544406342" watchObservedRunningTime="2026-09-17 15:40:03.166202418 +0000 UTC m=+625.546220173"
Sep 17 15:40:10.836972 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:10.836940    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg"]
Sep 17 15:40:10.840616 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:10.840593    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg"
Sep 17 15:40:10.842846 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:10.842827    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 17 15:40:10.854727 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:10.854706    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg"]
Sep 17 15:40:10.917947 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:10.917918    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csnjg\" (UID: \"0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg"
Sep 17 15:40:10.918055 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:10.917952    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csnjg\" (UID: \"0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg"
Sep 17 15:40:10.918055 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:10.917994    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csnjg\" (UID: \"0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg"
Sep 17 15:40:10.918055 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:10.918010    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csnjg\" (UID: \"0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg"
Sep 17 15:40:10.918168 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:10.918053    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csnjg\" (UID: \"0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg"
Sep 17 15:40:10.918168 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:10.918112    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dksfj\" (UniqueName: \"kubernetes.io/projected/0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce-kube-api-access-dksfj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csnjg\" (UID: \"0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg"
Sep 17 15:40:11.018949 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:11.018925    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csnjg\" (UID: \"0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg"
Sep 17 15:40:11.019038 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:11.018952    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csnjg\" (UID: \"0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg"
Sep 17 15:40:11.019038 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:11.018980    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csnjg\" (UID: \"0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg"
Sep 17 15:40:11.019038 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:11.018995    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csnjg\" (UID: \"0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg"
Sep 17 15:40:11.019173 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:11.019044    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csnjg\" (UID: \"0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg"
Sep 17 15:40:11.019173 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:11.019103    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dksfj\" (UniqueName: \"kubernetes.io/projected/0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce-kube-api-access-dksfj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csnjg\" (UID: \"0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg"
Sep 17 15:40:11.019367 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:11.019348    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csnjg\" (UID: \"0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg"
Sep 17 15:40:11.019412 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:11.019383    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csnjg\" (UID: \"0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg"
Sep 17 15:40:11.019446 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:11.019419    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csnjg\" (UID: \"0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg"
Sep 17 15:40:11.021222 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:11.021200    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csnjg\" (UID: \"0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg"
Sep 17 15:40:11.021373 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:11.021358    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csnjg\" (UID: \"0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg"
Sep 17 15:40:11.027864 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:11.027834    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dksfj\" (UniqueName: \"kubernetes.io/projected/0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce-kube-api-access-dksfj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-csnjg\" (UID: \"0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg"
Sep 17 15:40:11.151585 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:11.151567    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg"
Sep 17 15:40:11.424366 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:11.424260    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg"]
Sep 17 15:40:11.427242 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:40:11.427216    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f96f508_5ac7_42f1_ab3a_d6d2ab6e24ce.slice/crio-7cd65e95b459534b4bb1a457f6a43807609e11a1a53d5b45c14c77816e43e41d WatchSource:0}: Error finding container 7cd65e95b459534b4bb1a457f6a43807609e11a1a53d5b45c14c77816e43e41d: Status 404 returned error can't find the container with id 7cd65e95b459534b4bb1a457f6a43807609e11a1a53d5b45c14c77816e43e41d
Sep 17 15:40:11.864806 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:11.864780    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd"
Sep 17 15:40:11.865158 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:11.864845    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd"
Sep 17 15:40:11.876529 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:11.876506    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd"
Sep 17 15:40:12.186836 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:12.186772    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg" event={"ID":"0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce","Type":"ContainerStarted","Data":"c5d6357421ce37627cb82450f2ec5b07b7a91b0da91d943e1055efb842000ba3"}
Sep 17 15:40:12.186836 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:12.186815    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg" event={"ID":"0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce","Type":"ContainerStarted","Data":"7cd65e95b459534b4bb1a457f6a43807609e11a1a53d5b45c14c77816e43e41d"}
Sep 17 15:40:12.198013 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:12.197986    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd"
Sep 17 15:40:12.206986 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:12.206936    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg" podStartSLOduration=2.01715995 podStartE2EDuration="2.206924339s" totalImagesPullingTime="189.764389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:12.205461133 +0000 UTC m=+634.585478886" watchObservedRunningTime="2026-09-17 15:40:12.206924339 +0000 UTC m=+634.586942093"
Sep 17 15:40:21.152012 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:21.151980    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg"
Sep 17 15:40:21.152487 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:21.152077    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg"
Sep 17 15:40:21.164129 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:21.164104    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg"
Sep 17 15:40:21.237376 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:40:21.237342    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-csnjg"
Sep 17 15:42:27.199112 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.199079    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z"]
Sep 17 15:42:27.202670 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.202652    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z"
Sep 17 15:42:27.214548 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:42:27.214523    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"payload-processing-plugins-e2e-worker-w4-c86e9a\" is forbidden: User \"system:node:ip-10-0-134-104.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-134-104.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-c86e9a\"" type="*v1.ConfigMap"
Sep 17 15:42:27.214658 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:42:27.214583    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"payload-processing-e2e-worker-w4-c86e9a-dockercfg-l9qhq\" is forbidden: User \"system:node:ip-10-0-134-104.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-134-104.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-c86e9a-dockercfg-l9qhq\"" type="*v1.Secret"
Sep 17 15:42:27.214658 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:42:27.214641    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z\" is forbidden: User \"system:node:ip-10-0-134-104.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-134-104.ec2.internal' and this object" podUID="c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c" pod="openshift-ingress/payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z"
Sep 17 15:42:27.233211 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.233190    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z\" (UID: \"c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z"
Sep 17 15:42:27.233266 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.233232    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gfb6\" (UniqueName: \"kubernetes.io/projected/c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c-kube-api-access-8gfb6\") pod \"payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z\" (UID: \"c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z"
Sep 17 15:42:27.277286 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.277256    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z"]
Sep 17 15:42:27.294253 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.294208    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc"]
Sep 17 15:42:27.297577 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.297556    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc"
Sep 17 15:42:27.317397 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.317374    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc"]
Sep 17 15:42:27.334445 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.334423    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z\" (UID: \"c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z"
Sep 17 15:42:27.334564 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.334482    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gfb6\" (UniqueName: \"kubernetes.io/projected/c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c-kube-api-access-8gfb6\") pod \"payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z\" (UID: \"c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z"
Sep 17 15:42:27.334564 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.334517    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fdkw\" (UniqueName: \"kubernetes.io/projected/f98d8a52-f711-4229-ac7a-59ddc0ad68a4-kube-api-access-9fdkw\") pod \"payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc\" (UID: \"f98d8a52-f711-4229-ac7a-59ddc0ad68a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc"
Sep 17 15:42:27.334681 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.334566    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98d8a52-f711-4229-ac7a-59ddc0ad68a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc\" (UID: \"f98d8a52-f711-4229-ac7a-59ddc0ad68a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc"
Sep 17 15:42:27.371526 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.371500    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gfb6\" (UniqueName: \"kubernetes.io/projected/c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c-kube-api-access-8gfb6\") pod \"payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z\" (UID: \"c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z"
Sep 17 15:42:27.435756 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.435729    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98d8a52-f711-4229-ac7a-59ddc0ad68a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc\" (UID: \"f98d8a52-f711-4229-ac7a-59ddc0ad68a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc"
Sep 17 15:42:27.435887 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.435805    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fdkw\" (UniqueName: \"kubernetes.io/projected/f98d8a52-f711-4229-ac7a-59ddc0ad68a4-kube-api-access-9fdkw\") pod \"payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc\" (UID: \"f98d8a52-f711-4229-ac7a-59ddc0ad68a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc"
Sep 17 15:42:27.450209 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.450155    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fdkw\" (UniqueName: \"kubernetes.io/projected/f98d8a52-f711-4229-ac7a-59ddc0ad68a4-kube-api-access-9fdkw\") pod \"payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc\" (UID: \"f98d8a52-f711-4229-ac7a-59ddc0ad68a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc"
Sep 17 15:42:27.752130 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.752043    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm"]
Sep 17 15:42:27.755601 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.755584    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm"
Sep 17 15:42:27.758819 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.758797    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-449c47-serving-cert\""
Sep 17 15:42:27.772667 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.772643    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm"]
Sep 17 15:42:27.839041 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.839011    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ee62332-4b3b-4bd6-9af4-80fbff417114-metrics-certs\") pod \"maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm\" (UID: \"9ee62332-4b3b-4bd6-9af4-80fbff417114\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm"
Sep 17 15:42:27.839212 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.839048    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qxc2\" (UniqueName: \"kubernetes.io/projected/9ee62332-4b3b-4bd6-9af4-80fbff417114-kube-api-access-2qxc2\") pod \"maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm\" (UID: \"9ee62332-4b3b-4bd6-9af4-80fbff417114\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm"
Sep 17 15:42:27.839212 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.839112    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ee62332-4b3b-4bd6-9af4-80fbff417114-openshift-service-ca\") pod \"maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm\" (UID: \"9ee62332-4b3b-4bd6-9af4-80fbff417114\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm"
Sep 17 15:42:27.839212 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.839147    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ee62332-4b3b-4bd6-9af4-80fbff417114-maas-api-tls\") pod \"maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm\" (UID: \"9ee62332-4b3b-4bd6-9af4-80fbff417114\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm"
Sep 17 15:42:27.939958 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.939930    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ee62332-4b3b-4bd6-9af4-80fbff417114-metrics-certs\") pod \"maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm\" (UID: \"9ee62332-4b3b-4bd6-9af4-80fbff417114\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm"
Sep 17 15:42:27.939958 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.939958    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qxc2\" (UniqueName: \"kubernetes.io/projected/9ee62332-4b3b-4bd6-9af4-80fbff417114-kube-api-access-2qxc2\") pod \"maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm\" (UID: \"9ee62332-4b3b-4bd6-9af4-80fbff417114\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm"
Sep 17 15:42:27.940205 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.939985    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ee62332-4b3b-4bd6-9af4-80fbff417114-openshift-service-ca\") pod \"maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm\" (UID: \"9ee62332-4b3b-4bd6-9af4-80fbff417114\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm"
Sep 17 15:42:27.940205 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.940010    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ee62332-4b3b-4bd6-9af4-80fbff417114-maas-api-tls\") pod \"maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm\" (UID: \"9ee62332-4b3b-4bd6-9af4-80fbff417114\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm"
Sep 17 15:42:27.940205 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:42:27.940143    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-serving-cert: secret "maas-api-e2e-inf-a-449c47-serving-cert" not found
Sep 17 15:42:27.940205 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:42:27.940193    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ee62332-4b3b-4bd6-9af4-80fbff417114-maas-api-tls podName:9ee62332-4b3b-4bd6-9af4-80fbff417114 nodeName:}" failed. No retries permitted until 2026-09-17 15:42:28.440179399 +0000 UTC m=+770.820197132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9ee62332-4b3b-4bd6-9af4-80fbff417114-maas-api-tls") pod "maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm" (UID: "9ee62332-4b3b-4bd6-9af4-80fbff417114") : secret "maas-api-e2e-inf-a-449c47-serving-cert" not found
Sep 17 15:42:27.940608 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.940589    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ee62332-4b3b-4bd6-9af4-80fbff417114-openshift-service-ca\") pod \"maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm\" (UID: \"9ee62332-4b3b-4bd6-9af4-80fbff417114\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm"
Sep 17 15:42:27.942474 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.942454    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ee62332-4b3b-4bd6-9af4-80fbff417114-metrics-certs\") pod \"maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm\" (UID: \"9ee62332-4b3b-4bd6-9af4-80fbff417114\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm"
Sep 17 15:42:27.952712 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:27.952686    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qxc2\" (UniqueName: \"kubernetes.io/projected/9ee62332-4b3b-4bd6-9af4-80fbff417114-kube-api-access-2qxc2\") pod \"maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm\" (UID: \"9ee62332-4b3b-4bd6-9af4-80fbff417114\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm"
Sep 17 15:42:28.251731 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:28.251703    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-c86e9a-dockercfg-l9qhq\""
Sep 17 15:42:28.335400 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:42:28.335358    2556 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-worker-w4-c86e9a: failed to sync configmap cache: timed out waiting for the condition
Sep 17 15:42:28.335618 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:42:28.335443    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c-plugins-config-volume podName:c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c nodeName:}" failed. No retries permitted until 2026-09-17 15:42:28.835423672 +0000 UTC m=+771.215441409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c-plugins-config-volume") pod "payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z" (UID: "c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c") : failed to sync configmap cache: timed out waiting for the condition
Sep 17 15:42:28.435919 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:42:28.435883    2556 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-worker-w4-c86e9a: failed to sync configmap cache: timed out waiting for the condition
Sep 17 15:42:28.436101 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:42:28.436032    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f98d8a52-f711-4229-ac7a-59ddc0ad68a4-plugins-config-volume podName:f98d8a52-f711-4229-ac7a-59ddc0ad68a4 nodeName:}" failed. No retries permitted until 2026-09-17 15:42:28.936007814 +0000 UTC m=+771.316025566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f98d8a52-f711-4229-ac7a-59ddc0ad68a4-plugins-config-volume") pod "payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc" (UID: "f98d8a52-f711-4229-ac7a-59ddc0ad68a4") : failed to sync configmap cache: timed out waiting for the condition
Sep 17 15:42:28.443479 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:28.443448    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ee62332-4b3b-4bd6-9af4-80fbff417114-maas-api-tls\") pod \"maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm\" (UID: \"9ee62332-4b3b-4bd6-9af4-80fbff417114\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm"
Sep 17 15:42:28.446025 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:28.446001    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ee62332-4b3b-4bd6-9af4-80fbff417114-maas-api-tls\") pod \"maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm\" (UID: \"9ee62332-4b3b-4bd6-9af4-80fbff417114\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm"
Sep 17 15:42:28.667977 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:28.667946    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm"
Sep 17 15:42:28.777784 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:28.777756    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-c86e9a\""
Sep 17 15:42:28.848191 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:28.848158    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z\" (UID: \"c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z"
Sep 17 15:42:28.848886 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:28.848862    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z\" (UID: \"c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z"
Sep 17 15:42:28.946455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:28.946429    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm"]
Sep 17 15:42:28.950156 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:28.950126    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98d8a52-f711-4229-ac7a-59ddc0ad68a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc\" (UID: \"f98d8a52-f711-4229-ac7a-59ddc0ad68a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc"
Sep 17 15:42:28.950746 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:42:28.950722    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee62332_4b3b_4bd6_9af4_80fbff417114.slice/crio-8f74784129b4f3385040179111336c80cec7e0d6a4fc936ee2217169eac358ea WatchSource:0}: Error finding container 8f74784129b4f3385040179111336c80cec7e0d6a4fc936ee2217169eac358ea: Status 404 returned error can't find the container with id 8f74784129b4f3385040179111336c80cec7e0d6a4fc936ee2217169eac358ea
Sep 17 15:42:28.950921 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:28.950901    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98d8a52-f711-4229-ac7a-59ddc0ad68a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc\" (UID: \"f98d8a52-f711-4229-ac7a-59ddc0ad68a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc"
Sep 17 15:42:29.012820 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:29.012787    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z"
Sep 17 15:42:29.107425 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:29.107395    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc"
Sep 17 15:42:29.300373 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:29.300348    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z"]
Sep 17 15:42:29.303366 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:42:29.303341    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3924af3_fbb7_4781_88c7_e6a2d5cb3a9c.slice/crio-c24812414b1e094ffe6012056316f291bf8b3d7ecf9528ed5dd1aaf735c94609 WatchSource:0}: Error finding container c24812414b1e094ffe6012056316f291bf8b3d7ecf9528ed5dd1aaf735c94609: Status 404 returned error can't find the container with id c24812414b1e094ffe6012056316f291bf8b3d7ecf9528ed5dd1aaf735c94609
Sep 17 15:42:29.385236 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:29.385199    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc"]
Sep 17 15:42:29.388625 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:42:29.388599    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98d8a52_f711_4229_ac7a_59ddc0ad68a4.slice/crio-e2279b6f6fd894539ad70262bb7d605fc3507c7127ce52ec7dfc785882d39bde WatchSource:0}: Error finding container e2279b6f6fd894539ad70262bb7d605fc3507c7127ce52ec7dfc785882d39bde: Status 404 returned error can't find the container with id e2279b6f6fd894539ad70262bb7d605fc3507c7127ce52ec7dfc785882d39bde
Sep 17 15:42:29.836126 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:29.836087    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc" event={"ID":"f98d8a52-f711-4229-ac7a-59ddc0ad68a4","Type":"ContainerStarted","Data":"4c4e2c3d2e8af6ed156be54a58e34c24ac3577378fa3a98bd7331af27604c260"}
Sep 17 15:42:29.836340 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:29.836155    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc" event={"ID":"f98d8a52-f711-4229-ac7a-59ddc0ad68a4","Type":"ContainerStarted","Data":"e2279b6f6fd894539ad70262bb7d605fc3507c7127ce52ec7dfc785882d39bde"}
Sep 17 15:42:29.836523 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:29.836465    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc"
Sep 17 15:42:29.837880 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:29.837853    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm" event={"ID":"9ee62332-4b3b-4bd6-9af4-80fbff417114","Type":"ContainerStarted","Data":"9308697d06ed5af78cc1e79e9c5fba0ecbd32b6e6861c17b6651c131a9b12824"}
Sep 17 15:42:29.837994 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:29.837906    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm"
Sep 17 15:42:29.837994 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:29.837930    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm" event={"ID":"9ee62332-4b3b-4bd6-9af4-80fbff417114","Type":"ContainerStarted","Data":"8f74784129b4f3385040179111336c80cec7e0d6a4fc936ee2217169eac358ea"}
Sep 17 15:42:29.839847 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:29.839823    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z" event={"ID":"c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c","Type":"ContainerStarted","Data":"4073959c7243eb8eab6b826b280e760c3039126b50683ae055a5fa3ee10b771c"}
Sep 17 15:42:29.840008 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:29.839874    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z" event={"ID":"c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c","Type":"ContainerStarted","Data":"c24812414b1e094ffe6012056316f291bf8b3d7ecf9528ed5dd1aaf735c94609"}
Sep 17 15:42:29.840008 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:29.839905    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z"
Sep 17 15:42:29.864422 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:29.864370    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc" podStartSLOduration=2.864350278 podStartE2EDuration="2.864350278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:29.863279174 +0000 UTC m=+772.243296931" watchObservedRunningTime="2026-09-17 15:42:29.864350278 +0000 UTC m=+772.244368028"
Sep 17 15:42:29.892597 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:29.891601    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z" podStartSLOduration=2.8915833319999997 podStartE2EDuration="2.891583332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:29.890009204 +0000 UTC m=+772.270026961" watchObservedRunningTime="2026-09-17 15:42:29.891583332 +0000 UTC m=+772.271601088"
Sep 17 15:42:29.923901 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:29.923844    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm" podStartSLOduration=2.636610204 podStartE2EDuration="2.923821602s" totalImagesPullingTime="287.211398ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:29.920753826 +0000 UTC m=+772.300771597" watchObservedRunningTime="2026-09-17 15:42:29.923821602 +0000 UTC m=+772.303839360"
Sep 17 15:42:35.850123 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:35.850090    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm"
Sep 17 15:42:36.488988 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:36.488948    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9"]
Sep 17 15:42:36.492547 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:36.492524    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9"
Sep 17 15:42:36.495469 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:36.495448    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-849b71-serving-cert\""
Sep 17 15:42:36.509195 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:36.509165    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9"]
Sep 17 15:42:36.618439 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:36.618403    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a573b2a2-1835-471b-a875-195912e1d592-maas-api-tls\") pod \"maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9\" (UID: \"a573b2a2-1835-471b-a875-195912e1d592\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9"
Sep 17 15:42:36.618439 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:36.618442    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a573b2a2-1835-471b-a875-195912e1d592-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9\" (UID: \"a573b2a2-1835-471b-a875-195912e1d592\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9"
Sep 17 15:42:36.618644 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:36.618467    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a573b2a2-1835-471b-a875-195912e1d592-metrics-certs\") pod \"maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9\" (UID: \"a573b2a2-1835-471b-a875-195912e1d592\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9"
Sep 17 15:42:36.618644 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:36.618586    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vxrl\" (UniqueName: \"kubernetes.io/projected/a573b2a2-1835-471b-a875-195912e1d592-kube-api-access-2vxrl\") pod \"maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9\" (UID: \"a573b2a2-1835-471b-a875-195912e1d592\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9"
Sep 17 15:42:36.719823 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:36.719786    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vxrl\" (UniqueName: \"kubernetes.io/projected/a573b2a2-1835-471b-a875-195912e1d592-kube-api-access-2vxrl\") pod \"maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9\" (UID: \"a573b2a2-1835-471b-a875-195912e1d592\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9"
Sep 17 15:42:36.719987 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:36.719842    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a573b2a2-1835-471b-a875-195912e1d592-maas-api-tls\") pod \"maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9\" (UID: \"a573b2a2-1835-471b-a875-195912e1d592\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9"
Sep 17 15:42:36.719987 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:36.719943    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a573b2a2-1835-471b-a875-195912e1d592-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9\" (UID: \"a573b2a2-1835-471b-a875-195912e1d592\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9"
Sep 17 15:42:36.719987 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:36.719976    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a573b2a2-1835-471b-a875-195912e1d592-metrics-certs\") pod \"maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9\" (UID: \"a573b2a2-1835-471b-a875-195912e1d592\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9"
Sep 17 15:42:36.720720 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:36.720699    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a573b2a2-1835-471b-a875-195912e1d592-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9\" (UID: \"a573b2a2-1835-471b-a875-195912e1d592\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9"
Sep 17 15:42:36.722370 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:36.722341    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a573b2a2-1835-471b-a875-195912e1d592-metrics-certs\") pod \"maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9\" (UID: \"a573b2a2-1835-471b-a875-195912e1d592\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9"
Sep 17 15:42:36.722477 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:36.722369    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a573b2a2-1835-471b-a875-195912e1d592-maas-api-tls\") pod \"maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9\" (UID: \"a573b2a2-1835-471b-a875-195912e1d592\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9"
Sep 17 15:42:36.733267 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:36.733244    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vxrl\" (UniqueName: \"kubernetes.io/projected/a573b2a2-1835-471b-a875-195912e1d592-kube-api-access-2vxrl\") pod \"maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9\" (UID: \"a573b2a2-1835-471b-a875-195912e1d592\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9"
Sep 17 15:42:36.806079 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:36.805975    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9"
Sep 17 15:42:37.095753 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:37.095725    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9"]
Sep 17 15:42:37.097899 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:42:37.097876    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda573b2a2_1835_471b_a875_195912e1d592.slice/crio-093d7cc484560a3716e1f0068d7fc6fd490772c2387b6f6f98805d67da207fa8 WatchSource:0}: Error finding container 093d7cc484560a3716e1f0068d7fc6fd490772c2387b6f6f98805d67da207fa8: Status 404 returned error can't find the container with id 093d7cc484560a3716e1f0068d7fc6fd490772c2387b6f6f98805d67da207fa8
Sep 17 15:42:37.597613 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:37.597580    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6"]
Sep 17 15:42:37.601214 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:37.601197    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6"
Sep 17 15:42:37.605531 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:37.605509    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-849b71-dockercfg-j6gpn\""
Sep 17 15:42:37.605706 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:37.605689    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-849b71\""
Sep 17 15:42:37.618182 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:37.618161    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6"]
Sep 17 15:42:37.730524 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:37.730461    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwzcg\" (UniqueName: \"kubernetes.io/projected/55be1385-28d7-4c8f-9d17-7b87c852a726-kube-api-access-fwzcg\") pod \"payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6\" (UID: \"55be1385-28d7-4c8f-9d17-7b87c852a726\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6"
Sep 17 15:42:37.730656 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:37.730615    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55be1385-28d7-4c8f-9d17-7b87c852a726-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6\" (UID: \"55be1385-28d7-4c8f-9d17-7b87c852a726\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6"
Sep 17 15:42:37.831469 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:37.831429    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55be1385-28d7-4c8f-9d17-7b87c852a726-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6\" (UID: \"55be1385-28d7-4c8f-9d17-7b87c852a726\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6"
Sep 17 15:42:37.831620 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:37.831480    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwzcg\" (UniqueName: \"kubernetes.io/projected/55be1385-28d7-4c8f-9d17-7b87c852a726-kube-api-access-fwzcg\") pod \"payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6\" (UID: \"55be1385-28d7-4c8f-9d17-7b87c852a726\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6"
Sep 17 15:42:37.832248 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:37.832229    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55be1385-28d7-4c8f-9d17-7b87c852a726-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6\" (UID: \"55be1385-28d7-4c8f-9d17-7b87c852a726\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6"
Sep 17 15:42:37.849131 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:37.849102    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwzcg\" (UniqueName: \"kubernetes.io/projected/55be1385-28d7-4c8f-9d17-7b87c852a726-kube-api-access-fwzcg\") pod \"payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6\" (UID: \"55be1385-28d7-4c8f-9d17-7b87c852a726\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6"
Sep 17 15:42:37.886592 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:37.886557    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9" event={"ID":"a573b2a2-1835-471b-a875-195912e1d592","Type":"ContainerStarted","Data":"735259c2bc3dd4c8f8908c0a655d687e6b10df9f347082b96833d846328890cf"}
Sep 17 15:42:37.886757 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:37.886616    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9" event={"ID":"a573b2a2-1835-471b-a875-195912e1d592","Type":"ContainerStarted","Data":"093d7cc484560a3716e1f0068d7fc6fd490772c2387b6f6f98805d67da207fa8"}
Sep 17 15:42:37.886757 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:37.886696    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9"
Sep 17 15:42:37.910952 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:37.910928    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6"
Sep 17 15:42:37.920100 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:37.920041    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9" podStartSLOduration=1.59905864 podStartE2EDuration="1.920028636s" totalImagesPullingTime="320.969996ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:37.916498366 +0000 UTC m=+780.296516131" watchObservedRunningTime="2026-09-17 15:42:37.920028636 +0000 UTC m=+780.300046389"
Sep 17 15:42:38.204845 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:38.204823    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6"]
Sep 17 15:42:38.206457 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:42:38.206434    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55be1385_28d7_4c8f_9d17_7b87c852a726.slice/crio-9a17349f2c9b829435d7e72cc6197ce5ebb4dfee6297fb89303e918a058381fa WatchSource:0}: Error finding container 9a17349f2c9b829435d7e72cc6197ce5ebb4dfee6297fb89303e918a058381fa: Status 404 returned error can't find the container with id 9a17349f2c9b829435d7e72cc6197ce5ebb4dfee6297fb89303e918a058381fa
Sep 17 15:42:38.735375 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:38.735344    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd"]
Sep 17 15:42:38.738888 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:38.738868    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd"
Sep 17 15:42:38.742298 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:38.742276    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-28eba6-dockercfg-x7lmb\""
Sep 17 15:42:38.742298 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:38.742292    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-28eba6\""
Sep 17 15:42:38.768644 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:38.768624    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd"]
Sep 17 15:42:38.841311 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:38.841283    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt8zf\" (UniqueName: \"kubernetes.io/projected/b007a856-3172-4b26-87b7-9e23fb2525ee-kube-api-access-gt8zf\") pod \"payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd\" (UID: \"b007a856-3172-4b26-87b7-9e23fb2525ee\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd"
Sep 17 15:42:38.841456 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:38.841344    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b007a856-3172-4b26-87b7-9e23fb2525ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd\" (UID: \"b007a856-3172-4b26-87b7-9e23fb2525ee\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd"
Sep 17 15:42:38.894536 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:38.894498    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6" event={"ID":"55be1385-28d7-4c8f-9d17-7b87c852a726","Type":"ContainerStarted","Data":"7ab5ecd293918e3ce1dd45b59579f3a509c3fc8ea484500e5b14bf617fc7be52"}
Sep 17 15:42:38.894695 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:38.894562    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6" event={"ID":"55be1385-28d7-4c8f-9d17-7b87c852a726","Type":"ContainerStarted","Data":"9a17349f2c9b829435d7e72cc6197ce5ebb4dfee6297fb89303e918a058381fa"}
Sep 17 15:42:38.894858 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:38.894839    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6"
Sep 17 15:42:38.930719 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:38.930662    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6" podStartSLOduration=1.9306430190000001 podStartE2EDuration="1.930643019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:38.930267327 +0000 UTC m=+781.310285082" watchObservedRunningTime="2026-09-17 15:42:38.930643019 +0000 UTC m=+781.310660774"
Sep 17 15:42:38.942680 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:38.942651    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b007a856-3172-4b26-87b7-9e23fb2525ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd\" (UID: \"b007a856-3172-4b26-87b7-9e23fb2525ee\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd"
Sep 17 15:42:38.942809 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:38.942741    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt8zf\" (UniqueName: \"kubernetes.io/projected/b007a856-3172-4b26-87b7-9e23fb2525ee-kube-api-access-gt8zf\") pod \"payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd\" (UID: \"b007a856-3172-4b26-87b7-9e23fb2525ee\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd"
Sep 17 15:42:38.943387 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:38.943368    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b007a856-3172-4b26-87b7-9e23fb2525ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd\" (UID: \"b007a856-3172-4b26-87b7-9e23fb2525ee\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd"
Sep 17 15:42:38.956739 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:38.956716    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt8zf\" (UniqueName: \"kubernetes.io/projected/b007a856-3172-4b26-87b7-9e23fb2525ee-kube-api-access-gt8zf\") pod \"payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd\" (UID: \"b007a856-3172-4b26-87b7-9e23fb2525ee\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd"
Sep 17 15:42:39.049244 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:39.049171    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd"
Sep 17 15:42:39.359896 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:39.359873    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd"]
Sep 17 15:42:39.361663 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:42:39.361636    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb007a856_3172_4b26_87b7_9e23fb2525ee.slice/crio-68c8df98fa27336701b4f13e1ec61406340818b9d783a79c87c8b302628bb3c4 WatchSource:0}: Error finding container 68c8df98fa27336701b4f13e1ec61406340818b9d783a79c87c8b302628bb3c4: Status 404 returned error can't find the container with id 68c8df98fa27336701b4f13e1ec61406340818b9d783a79c87c8b302628bb3c4
Sep 17 15:42:39.901249 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:39.901209    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd" event={"ID":"b007a856-3172-4b26-87b7-9e23fb2525ee","Type":"ContainerStarted","Data":"dec47b190e30ef90a7f7f76fedb47f755983a44b3ada21c327260d46e870cf6f"}
Sep 17 15:42:39.901428 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:39.901272    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd" event={"ID":"b007a856-3172-4b26-87b7-9e23fb2525ee","Type":"ContainerStarted","Data":"68c8df98fa27336701b4f13e1ec61406340818b9d783a79c87c8b302628bb3c4"}
Sep 17 15:42:39.901428 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:39.901314    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd"
Sep 17 15:42:39.929992 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:39.929940    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd" podStartSLOduration=1.9299229900000001 podStartE2EDuration="1.92992299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:39.92974835 +0000 UTC m=+782.309766116" watchObservedRunningTime="2026-09-17 15:42:39.92992299 +0000 UTC m=+782.309940735"
Sep 17 15:42:40.332878 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:40.332804    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7"]
Sep 17 15:42:40.337485 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:40.337457    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7"
Sep 17 15:42:40.340563 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:40.340540    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-76a88377-serving-cert\""
Sep 17 15:42:40.354421 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:40.354165    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7"]
Sep 17 15:42:40.455748 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:40.455716    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5c545e-89e7-4236-be63-39db23205b33-maas-api-tls\") pod \"maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7\" (UID: \"ab5c545e-89e7-4236-be63-39db23205b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7"
Sep 17 15:42:40.456184 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:40.455770    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzvbk\" (UniqueName: \"kubernetes.io/projected/ab5c545e-89e7-4236-be63-39db23205b33-kube-api-access-fzvbk\") pod \"maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7\" (UID: \"ab5c545e-89e7-4236-be63-39db23205b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7"
Sep 17 15:42:40.456184 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:40.455794    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab5c545e-89e7-4236-be63-39db23205b33-metrics-certs\") pod \"maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7\" (UID: \"ab5c545e-89e7-4236-be63-39db23205b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7"
Sep 17 15:42:40.456184 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:40.455823    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5c545e-89e7-4236-be63-39db23205b33-openshift-service-ca\") pod \"maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7\" (UID: \"ab5c545e-89e7-4236-be63-39db23205b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7"
Sep 17 15:42:40.556423 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:40.556388    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzvbk\" (UniqueName: \"kubernetes.io/projected/ab5c545e-89e7-4236-be63-39db23205b33-kube-api-access-fzvbk\") pod \"maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7\" (UID: \"ab5c545e-89e7-4236-be63-39db23205b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7"
Sep 17 15:42:40.556604 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:40.556437    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab5c545e-89e7-4236-be63-39db23205b33-metrics-certs\") pod \"maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7\" (UID: \"ab5c545e-89e7-4236-be63-39db23205b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7"
Sep 17 15:42:40.556604 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:40.556480    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5c545e-89e7-4236-be63-39db23205b33-openshift-service-ca\") pod \"maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7\" (UID: \"ab5c545e-89e7-4236-be63-39db23205b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7"
Sep 17 15:42:40.556687 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:40.556624    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5c545e-89e7-4236-be63-39db23205b33-maas-api-tls\") pod \"maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7\" (UID: \"ab5c545e-89e7-4236-be63-39db23205b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7"
Sep 17 15:42:40.556770 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:42:40.556754    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-serving-cert: secret "maas-api-e2e-ait-76a88377-serving-cert" not found
Sep 17 15:42:40.556833 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:42:40.556823    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab5c545e-89e7-4236-be63-39db23205b33-maas-api-tls podName:ab5c545e-89e7-4236-be63-39db23205b33 nodeName:}" failed. No retries permitted until 2026-09-17 15:42:41.056800419 +0000 UTC m=+783.436818167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ab5c545e-89e7-4236-be63-39db23205b33-maas-api-tls") pod "maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7" (UID: "ab5c545e-89e7-4236-be63-39db23205b33") : secret "maas-api-e2e-ait-76a88377-serving-cert" not found
Sep 17 15:42:40.557439 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:40.557408    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5c545e-89e7-4236-be63-39db23205b33-openshift-service-ca\") pod \"maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7\" (UID: \"ab5c545e-89e7-4236-be63-39db23205b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7"
Sep 17 15:42:40.559369 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:40.559344    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab5c545e-89e7-4236-be63-39db23205b33-metrics-certs\") pod \"maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7\" (UID: \"ab5c545e-89e7-4236-be63-39db23205b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7"
Sep 17 15:42:40.594098 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:40.594026    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzvbk\" (UniqueName: \"kubernetes.io/projected/ab5c545e-89e7-4236-be63-39db23205b33-kube-api-access-fzvbk\") pod \"maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7\" (UID: \"ab5c545e-89e7-4236-be63-39db23205b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7"
Sep 17 15:42:40.846890 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:40.846798    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z"
Sep 17 15:42:40.846890 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:40.846867    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc"
Sep 17 15:42:41.061862 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:41.061816    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5c545e-89e7-4236-be63-39db23205b33-maas-api-tls\") pod \"maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7\" (UID: \"ab5c545e-89e7-4236-be63-39db23205b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7"
Sep 17 15:42:41.062033 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:42:41.062012    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-serving-cert: secret "maas-api-e2e-ait-76a88377-serving-cert" not found
Sep 17 15:42:41.062131 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:42:41.062118    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab5c545e-89e7-4236-be63-39db23205b33-maas-api-tls podName:ab5c545e-89e7-4236-be63-39db23205b33 nodeName:}" failed. No retries permitted until 2026-09-17 15:42:42.062094711 +0000 UTC m=+784.442112455 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ab5c545e-89e7-4236-be63-39db23205b33-maas-api-tls") pod "maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7" (UID: "ab5c545e-89e7-4236-be63-39db23205b33") : secret "maas-api-e2e-ait-76a88377-serving-cert" not found
Sep 17 15:42:41.656605 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:41.656567    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm"]
Sep 17 15:42:41.660413 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:41.660390    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm"
Sep 17 15:42:41.663287 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:41.663270    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-76a88377-dockercfg-mg7gr\""
Sep 17 15:42:41.663372 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:41.663317    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-76a88377\""
Sep 17 15:42:41.669656 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:41.669637    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm"]
Sep 17 15:42:41.768806 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:41.768776    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hntb9\" (UniqueName: \"kubernetes.io/projected/b93ce7cb-0676-4ae4-a5bb-8e59d28466d3-kube-api-access-hntb9\") pod \"payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm\" (UID: \"b93ce7cb-0676-4ae4-a5bb-8e59d28466d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm"
Sep 17 15:42:41.768962 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:41.768868    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b93ce7cb-0676-4ae4-a5bb-8e59d28466d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm\" (UID: \"b93ce7cb-0676-4ae4-a5bb-8e59d28466d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm"
Sep 17 15:42:41.869484 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:41.869454    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hntb9\" (UniqueName: \"kubernetes.io/projected/b93ce7cb-0676-4ae4-a5bb-8e59d28466d3-kube-api-access-hntb9\") pod \"payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm\" (UID: \"b93ce7cb-0676-4ae4-a5bb-8e59d28466d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm"
Sep 17 15:42:41.869624 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:41.869537    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b93ce7cb-0676-4ae4-a5bb-8e59d28466d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm\" (UID: \"b93ce7cb-0676-4ae4-a5bb-8e59d28466d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm"
Sep 17 15:42:41.870159 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:41.870143    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b93ce7cb-0676-4ae4-a5bb-8e59d28466d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm\" (UID: \"b93ce7cb-0676-4ae4-a5bb-8e59d28466d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm"
Sep 17 15:42:41.878366 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:41.878347    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hntb9\" (UniqueName: \"kubernetes.io/projected/b93ce7cb-0676-4ae4-a5bb-8e59d28466d3-kube-api-access-hntb9\") pod \"payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm\" (UID: \"b93ce7cb-0676-4ae4-a5bb-8e59d28466d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm"
Sep 17 15:42:41.973594 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:41.973518    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm"
Sep 17 15:42:42.071725 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:42.071694    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5c545e-89e7-4236-be63-39db23205b33-maas-api-tls\") pod \"maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7\" (UID: \"ab5c545e-89e7-4236-be63-39db23205b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7"
Sep 17 15:42:42.074184 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:42.074160    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5c545e-89e7-4236-be63-39db23205b33-maas-api-tls\") pod \"maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7\" (UID: \"ab5c545e-89e7-4236-be63-39db23205b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7"
Sep 17 15:42:42.151871 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:42.151840    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7"
Sep 17 15:42:42.262925 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:42.262900    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm"]
Sep 17 15:42:42.264038 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:42:42.264007    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb93ce7cb_0676_4ae4_a5bb_8e59d28466d3.slice/crio-f9e919a7b5473c46fa7a6966b2d4cc10a07a7479a1b175f91a7a7c483ab03269 WatchSource:0}: Error finding container f9e919a7b5473c46fa7a6966b2d4cc10a07a7479a1b175f91a7a7c483ab03269: Status 404 returned error can't find the container with id f9e919a7b5473c46fa7a6966b2d4cc10a07a7479a1b175f91a7a7c483ab03269
Sep 17 15:42:42.434366 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:42.434342    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7"]
Sep 17 15:42:42.436496 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:42:42.436466    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5c545e_89e7_4236_be63_39db23205b33.slice/crio-a17eb94fd01e01234b387599a92d47c1916d131bffa70bc8d84995b28c492a41 WatchSource:0}: Error finding container a17eb94fd01e01234b387599a92d47c1916d131bffa70bc8d84995b28c492a41: Status 404 returned error can't find the container with id a17eb94fd01e01234b387599a92d47c1916d131bffa70bc8d84995b28c492a41
Sep 17 15:42:42.923652 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:42.923319    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm" event={"ID":"b93ce7cb-0676-4ae4-a5bb-8e59d28466d3","Type":"ContainerStarted","Data":"0b3a32ae6a5590eb7d60c162fd5ad59fcdb5791694c31203347048f16db6fd31"}
Sep 17 15:42:42.923652 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:42.923377    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm" event={"ID":"b93ce7cb-0676-4ae4-a5bb-8e59d28466d3","Type":"ContainerStarted","Data":"f9e919a7b5473c46fa7a6966b2d4cc10a07a7479a1b175f91a7a7c483ab03269"}
Sep 17 15:42:42.923652 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:42.923547    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm"
Sep 17 15:42:42.925131 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:42.925100    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7" event={"ID":"ab5c545e-89e7-4236-be63-39db23205b33","Type":"ContainerStarted","Data":"a17eb94fd01e01234b387599a92d47c1916d131bffa70bc8d84995b28c492a41"}
Sep 17 15:42:42.945406 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:42.945364    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm" podStartSLOduration=1.945351466 podStartE2EDuration="1.945351466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:42.943573979 +0000 UTC m=+785.323591734" watchObservedRunningTime="2026-09-17 15:42:42.945351466 +0000 UTC m=+785.325369220"
Sep 17 15:42:43.900144 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:43.900110    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9"
Sep 17 15:42:43.933440 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:43.933406    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7" event={"ID":"ab5c545e-89e7-4236-be63-39db23205b33","Type":"ContainerStarted","Data":"27299b83a58ceab00fb8d739ad17535e919c06bb2e8dd74782e04137c2fbba97"}
Sep 17 15:42:43.933819 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:43.933748    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7"
Sep 17 15:42:43.956298 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:43.956253    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7" podStartSLOduration=3.532644527 podStartE2EDuration="3.956239253s" totalImagesPullingTime="423.594726ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:43.954953514 +0000 UTC m=+786.334971282" watchObservedRunningTime="2026-09-17 15:42:43.956239253 +0000 UTC m=+786.336257007"
Sep 17 15:42:49.903902 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:49.903872    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6"
Sep 17 15:42:49.944415 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:49.944383    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7"
Sep 17 15:42:50.911042 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:50.911010    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd"
Sep 17 15:42:53.936233 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:42:53.936200    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm"
Sep 17 15:43:36.739783 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:36.739751    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7"]
Sep 17 15:43:36.740327 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:36.739963    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7" podUID="ab5c545e-89e7-4236-be63-39db23205b33" containerName="maas-api" containerID="cri-o://27299b83a58ceab00fb8d739ad17535e919c06bb2e8dd74782e04137c2fbba97" gracePeriod=30
Sep 17 15:43:36.814509 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:36.814472    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm"]
Sep 17 15:43:36.814833 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:36.814751    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm" podUID="b93ce7cb-0676-4ae4-a5bb-8e59d28466d3" containerName="payload-pre-processing" containerID="cri-o://0b3a32ae6a5590eb7d60c162fd5ad59fcdb5791694c31203347048f16db6fd31" gracePeriod=30
Sep 17 15:43:37.064694 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.064669    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7"
Sep 17 15:43:37.105438 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.105414    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm"
Sep 17 15:43:37.172997 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.172956    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5c545e-89e7-4236-be63-39db23205b33-openshift-service-ca\") pod \"ab5c545e-89e7-4236-be63-39db23205b33\" (UID: \"ab5c545e-89e7-4236-be63-39db23205b33\") "
Sep 17 15:43:37.173189 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.173045    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab5c545e-89e7-4236-be63-39db23205b33-metrics-certs\") pod \"ab5c545e-89e7-4236-be63-39db23205b33\" (UID: \"ab5c545e-89e7-4236-be63-39db23205b33\") "
Sep 17 15:43:37.173189 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.173137    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzvbk\" (UniqueName: \"kubernetes.io/projected/ab5c545e-89e7-4236-be63-39db23205b33-kube-api-access-fzvbk\") pod \"ab5c545e-89e7-4236-be63-39db23205b33\" (UID: \"ab5c545e-89e7-4236-be63-39db23205b33\") "
Sep 17 15:43:37.173305 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.173209    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5c545e-89e7-4236-be63-39db23205b33-maas-api-tls\") pod \"ab5c545e-89e7-4236-be63-39db23205b33\" (UID: \"ab5c545e-89e7-4236-be63-39db23205b33\") "
Sep 17 15:43:37.173445 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.173416    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab5c545e-89e7-4236-be63-39db23205b33-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ab5c545e-89e7-4236-be63-39db23205b33" (UID: "ab5c545e-89e7-4236-be63-39db23205b33"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:43:37.175941 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.175907    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab5c545e-89e7-4236-be63-39db23205b33-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ab5c545e-89e7-4236-be63-39db23205b33" (UID: "ab5c545e-89e7-4236-be63-39db23205b33"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:43:37.176056 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.175915    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5c545e-89e7-4236-be63-39db23205b33-kube-api-access-fzvbk" (OuterVolumeSpecName: "kube-api-access-fzvbk") pod "ab5c545e-89e7-4236-be63-39db23205b33" (UID: "ab5c545e-89e7-4236-be63-39db23205b33"). InnerVolumeSpecName "kube-api-access-fzvbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:43:37.176056 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.175990    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab5c545e-89e7-4236-be63-39db23205b33-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ab5c545e-89e7-4236-be63-39db23205b33" (UID: "ab5c545e-89e7-4236-be63-39db23205b33"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:43:37.200317 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.200288    2556 generic.go:356] "Generic (PLEG): container finished" podID="b93ce7cb-0676-4ae4-a5bb-8e59d28466d3" containerID="0b3a32ae6a5590eb7d60c162fd5ad59fcdb5791694c31203347048f16db6fd31" exitCode=0
Sep 17 15:43:37.200448 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.200333    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm"
Sep 17 15:43:37.200448 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.200377    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm" event={"ID":"b93ce7cb-0676-4ae4-a5bb-8e59d28466d3","Type":"ContainerDied","Data":"0b3a32ae6a5590eb7d60c162fd5ad59fcdb5791694c31203347048f16db6fd31"}
Sep 17 15:43:37.200448 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.200442    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm" event={"ID":"b93ce7cb-0676-4ae4-a5bb-8e59d28466d3","Type":"ContainerDied","Data":"f9e919a7b5473c46fa7a6966b2d4cc10a07a7479a1b175f91a7a7c483ab03269"}
Sep 17 15:43:37.200573 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.200471    2556 scope.go:117] "RemoveContainer" containerID="0b3a32ae6a5590eb7d60c162fd5ad59fcdb5791694c31203347048f16db6fd31"
Sep 17 15:43:37.202312 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.202278    2556 generic.go:356] "Generic (PLEG): container finished" podID="ab5c545e-89e7-4236-be63-39db23205b33" containerID="27299b83a58ceab00fb8d739ad17535e919c06bb2e8dd74782e04137c2fbba97" exitCode=0
Sep 17 15:43:37.202469 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.202326    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7"
Sep 17 15:43:37.202469 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.202339    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7" event={"ID":"ab5c545e-89e7-4236-be63-39db23205b33","Type":"ContainerDied","Data":"27299b83a58ceab00fb8d739ad17535e919c06bb2e8dd74782e04137c2fbba97"}
Sep 17 15:43:37.202469 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.202377    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7" event={"ID":"ab5c545e-89e7-4236-be63-39db23205b33","Type":"ContainerDied","Data":"a17eb94fd01e01234b387599a92d47c1916d131bffa70bc8d84995b28c492a41"}
Sep 17 15:43:37.211853 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.211835    2556 scope.go:117] "RemoveContainer" containerID="0b3a32ae6a5590eb7d60c162fd5ad59fcdb5791694c31203347048f16db6fd31"
Sep 17 15:43:37.212139 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:43:37.212115    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b3a32ae6a5590eb7d60c162fd5ad59fcdb5791694c31203347048f16db6fd31\": container with ID starting with 0b3a32ae6a5590eb7d60c162fd5ad59fcdb5791694c31203347048f16db6fd31 not found: ID does not exist" containerID="0b3a32ae6a5590eb7d60c162fd5ad59fcdb5791694c31203347048f16db6fd31"
Sep 17 15:43:37.212241 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.212144    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b3a32ae6a5590eb7d60c162fd5ad59fcdb5791694c31203347048f16db6fd31"} err="failed to get container status \"0b3a32ae6a5590eb7d60c162fd5ad59fcdb5791694c31203347048f16db6fd31\": rpc error: code = NotFound desc = could not find container \"0b3a32ae6a5590eb7d60c162fd5ad59fcdb5791694c31203347048f16db6fd31\": container with ID starting with 0b3a32ae6a5590eb7d60c162fd5ad59fcdb5791694c31203347048f16db6fd31 not found: ID does not exist"
Sep 17 15:43:37.212241 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.212162    2556 scope.go:117] "RemoveContainer" containerID="27299b83a58ceab00fb8d739ad17535e919c06bb2e8dd74782e04137c2fbba97"
Sep 17 15:43:37.220466 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.220452    2556 scope.go:117] "RemoveContainer" containerID="27299b83a58ceab00fb8d739ad17535e919c06bb2e8dd74782e04137c2fbba97"
Sep 17 15:43:37.220702 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:43:37.220688    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27299b83a58ceab00fb8d739ad17535e919c06bb2e8dd74782e04137c2fbba97\": container with ID starting with 27299b83a58ceab00fb8d739ad17535e919c06bb2e8dd74782e04137c2fbba97 not found: ID does not exist" containerID="27299b83a58ceab00fb8d739ad17535e919c06bb2e8dd74782e04137c2fbba97"
Sep 17 15:43:37.220751 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.220708    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27299b83a58ceab00fb8d739ad17535e919c06bb2e8dd74782e04137c2fbba97"} err="failed to get container status \"27299b83a58ceab00fb8d739ad17535e919c06bb2e8dd74782e04137c2fbba97\": rpc error: code = NotFound desc = could not find container \"27299b83a58ceab00fb8d739ad17535e919c06bb2e8dd74782e04137c2fbba97\": container with ID starting with 27299b83a58ceab00fb8d739ad17535e919c06bb2e8dd74782e04137c2fbba97 not found: ID does not exist"
Sep 17 15:43:37.226074 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.226038    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7"]
Sep 17 15:43:37.229395 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.229376    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-76a88377-f87b5b9b5-xndd7"]
Sep 17 15:43:37.274664 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.274633    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hntb9\" (UniqueName: \"kubernetes.io/projected/b93ce7cb-0676-4ae4-a5bb-8e59d28466d3-kube-api-access-hntb9\") pod \"b93ce7cb-0676-4ae4-a5bb-8e59d28466d3\" (UID: \"b93ce7cb-0676-4ae4-a5bb-8e59d28466d3\") "
Sep 17 15:43:37.274846 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.274710    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b93ce7cb-0676-4ae4-a5bb-8e59d28466d3-plugins-config-volume\") pod \"b93ce7cb-0676-4ae4-a5bb-8e59d28466d3\" (UID: \"b93ce7cb-0676-4ae4-a5bb-8e59d28466d3\") "
Sep 17 15:43:37.274925 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.274913    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzvbk\" (UniqueName: \"kubernetes.io/projected/ab5c545e-89e7-4236-be63-39db23205b33-kube-api-access-fzvbk\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:43:37.274982 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.274926    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5c545e-89e7-4236-be63-39db23205b33-maas-api-tls\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:43:37.274982 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.274936    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5c545e-89e7-4236-be63-39db23205b33-openshift-service-ca\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:43:37.274982 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.274947    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab5c545e-89e7-4236-be63-39db23205b33-metrics-certs\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:43:37.275170 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.275144    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b93ce7cb-0676-4ae4-a5bb-8e59d28466d3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b93ce7cb-0676-4ae4-a5bb-8e59d28466d3" (UID: "b93ce7cb-0676-4ae4-a5bb-8e59d28466d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:43:37.276782 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.276740    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93ce7cb-0676-4ae4-a5bb-8e59d28466d3-kube-api-access-hntb9" (OuterVolumeSpecName: "kube-api-access-hntb9") pod "b93ce7cb-0676-4ae4-a5bb-8e59d28466d3" (UID: "b93ce7cb-0676-4ae4-a5bb-8e59d28466d3"). InnerVolumeSpecName "kube-api-access-hntb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:43:37.376107 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.376073    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hntb9\" (UniqueName: \"kubernetes.io/projected/b93ce7cb-0676-4ae4-a5bb-8e59d28466d3-kube-api-access-hntb9\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:43:37.376107 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.376101    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b93ce7cb-0676-4ae4-a5bb-8e59d28466d3-plugins-config-volume\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:43:37.524107 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.524076    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm"]
Sep 17 15:43:37.532122 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:37.532070    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-76a88377-746f8dc854-8jvxm"]
Sep 17 15:43:38.137202 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:38.137174    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab5c545e-89e7-4236-be63-39db23205b33" path="/var/lib/kubelet/pods/ab5c545e-89e7-4236-be63-39db23205b33/volumes"
Sep 17 15:43:38.137683 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:38.137665    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b93ce7cb-0676-4ae4-a5bb-8e59d28466d3" path="/var/lib/kubelet/pods/b93ce7cb-0676-4ae4-a5bb-8e59d28466d3/volumes"
Sep 17 15:43:42.370763 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:42.370726    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb"]
Sep 17 15:43:42.371431 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:42.371411    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b93ce7cb-0676-4ae4-a5bb-8e59d28466d3" containerName="payload-pre-processing"
Sep 17 15:43:42.371479 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:42.371435    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b93ce7cb-0676-4ae4-a5bb-8e59d28466d3" containerName="payload-pre-processing"
Sep 17 15:43:42.371479 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:42.371465    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab5c545e-89e7-4236-be63-39db23205b33" containerName="maas-api"
Sep 17 15:43:42.371479 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:42.371474    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab5c545e-89e7-4236-be63-39db23205b33" containerName="maas-api"
Sep 17 15:43:42.371594 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:42.371582    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b93ce7cb-0676-4ae4-a5bb-8e59d28466d3" containerName="payload-pre-processing"
Sep 17 15:43:42.371627 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:42.371599    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab5c545e-89e7-4236-be63-39db23205b33" containerName="maas-api"
Sep 17 15:43:42.377523 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:42.377501    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb"
Sep 17 15:43:42.381159 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:42.381137    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-49b90ab7-serving-cert\""
Sep 17 15:43:42.387740 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:42.387708    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb"]
Sep 17 15:43:42.522550 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:42.522513    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqm4j\" (UniqueName: \"kubernetes.io/projected/f1102ec7-7c4c-475c-b311-4fe9c4b1de5b-kube-api-access-zqm4j\") pod \"maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb\" (UID: \"f1102ec7-7c4c-475c-b311-4fe9c4b1de5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb"
Sep 17 15:43:42.522714 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:42.522561    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1102ec7-7c4c-475c-b311-4fe9c4b1de5b-openshift-service-ca\") pod \"maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb\" (UID: \"f1102ec7-7c4c-475c-b311-4fe9c4b1de5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb"
Sep 17 15:43:42.522714 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:42.522621    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1102ec7-7c4c-475c-b311-4fe9c4b1de5b-metrics-certs\") pod \"maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb\" (UID: \"f1102ec7-7c4c-475c-b311-4fe9c4b1de5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb"
Sep 17 15:43:42.522714 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:42.522709    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1102ec7-7c4c-475c-b311-4fe9c4b1de5b-maas-api-tls\") pod \"maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb\" (UID: \"f1102ec7-7c4c-475c-b311-4fe9c4b1de5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb"
Sep 17 15:43:42.623969 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:42.623882    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1102ec7-7c4c-475c-b311-4fe9c4b1de5b-maas-api-tls\") pod \"maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb\" (UID: \"f1102ec7-7c4c-475c-b311-4fe9c4b1de5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb"
Sep 17 15:43:42.623969 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:42.623952    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqm4j\" (UniqueName: \"kubernetes.io/projected/f1102ec7-7c4c-475c-b311-4fe9c4b1de5b-kube-api-access-zqm4j\") pod \"maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb\" (UID: \"f1102ec7-7c4c-475c-b311-4fe9c4b1de5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb"
Sep 17 15:43:42.624194 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:42.623982    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1102ec7-7c4c-475c-b311-4fe9c4b1de5b-openshift-service-ca\") pod \"maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb\" (UID: \"f1102ec7-7c4c-475c-b311-4fe9c4b1de5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb"
Sep 17 15:43:42.624194 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:42.624000    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1102ec7-7c4c-475c-b311-4fe9c4b1de5b-metrics-certs\") pod \"maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb\" (UID: \"f1102ec7-7c4c-475c-b311-4fe9c4b1de5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb"
Sep 17 15:43:42.624687 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:42.624660    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1102ec7-7c4c-475c-b311-4fe9c4b1de5b-openshift-service-ca\") pod \"maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb\" (UID: \"f1102ec7-7c4c-475c-b311-4fe9c4b1de5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb"
Sep 17 15:43:42.626441 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:42.626412    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1102ec7-7c4c-475c-b311-4fe9c4b1de5b-metrics-certs\") pod \"maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb\" (UID: \"f1102ec7-7c4c-475c-b311-4fe9c4b1de5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb"
Sep 17 15:43:42.626441 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:42.626424    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1102ec7-7c4c-475c-b311-4fe9c4b1de5b-maas-api-tls\") pod \"maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb\" (UID: \"f1102ec7-7c4c-475c-b311-4fe9c4b1de5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb"
Sep 17 15:43:42.635240 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:42.635207    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqm4j\" (UniqueName: \"kubernetes.io/projected/f1102ec7-7c4c-475c-b311-4fe9c4b1de5b-kube-api-access-zqm4j\") pod \"maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb\" (UID: \"f1102ec7-7c4c-475c-b311-4fe9c4b1de5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb"
Sep 17 15:43:42.690675 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:42.690653    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb"
Sep 17 15:43:42.966952 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:42.966931    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb"]
Sep 17 15:43:42.968646 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:43:42.968617    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1102ec7_7c4c_475c_b311_4fe9c4b1de5b.slice/crio-a7065043989b98d4f4b430b4ea2e46a69a110c97f123457bd6ed4c58c9466cd0 WatchSource:0}: Error finding container a7065043989b98d4f4b430b4ea2e46a69a110c97f123457bd6ed4c58c9466cd0: Status 404 returned error can't find the container with id a7065043989b98d4f4b430b4ea2e46a69a110c97f123457bd6ed4c58c9466cd0
Sep 17 15:43:43.234808 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:43.234727    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb" event={"ID":"f1102ec7-7c4c-475c-b311-4fe9c4b1de5b","Type":"ContainerStarted","Data":"a7065043989b98d4f4b430b4ea2e46a69a110c97f123457bd6ed4c58c9466cd0"}
Sep 17 15:43:44.241584 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:44.241548    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb" event={"ID":"f1102ec7-7c4c-475c-b311-4fe9c4b1de5b","Type":"ContainerStarted","Data":"5ff9a4e808e6398ae8b2cf36badcff3e0e858699a971cf6064a7fb6bb46f78b9"}
Sep 17 15:43:44.241966 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:44.241699    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb"
Sep 17 15:43:44.262842 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:44.262793    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb" podStartSLOduration=1.837874302 podStartE2EDuration="2.262778053s" totalImagesPullingTime="424.903751ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:43:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:43:44.26097751 +0000 UTC m=+846.640995265" watchObservedRunningTime="2026-09-17 15:43:44.262778053 +0000 UTC m=+846.642795807"
Sep 17 15:43:50.251531 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:43:50.251504    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb"
Sep 17 15:44:12.231946 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:12.231910    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb"]
Sep 17 15:44:12.232426 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:12.232137    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb" podUID="f1102ec7-7c4c-475c-b311-4fe9c4b1de5b" containerName="maas-api" containerID="cri-o://5ff9a4e808e6398ae8b2cf36badcff3e0e858699a971cf6064a7fb6bb46f78b9" gracePeriod=30
Sep 17 15:44:12.380338 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:12.380101    2556 generic.go:356] "Generic (PLEG): container finished" podID="f1102ec7-7c4c-475c-b311-4fe9c4b1de5b" containerID="5ff9a4e808e6398ae8b2cf36badcff3e0e858699a971cf6064a7fb6bb46f78b9" exitCode=0
Sep 17 15:44:12.380338 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:12.380265    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb" event={"ID":"f1102ec7-7c4c-475c-b311-4fe9c4b1de5b","Type":"ContainerDied","Data":"5ff9a4e808e6398ae8b2cf36badcff3e0e858699a971cf6064a7fb6bb46f78b9"}
Sep 17 15:44:12.494449 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:12.494397    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb"
Sep 17 15:44:12.574015 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:12.573979    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqm4j\" (UniqueName: \"kubernetes.io/projected/f1102ec7-7c4c-475c-b311-4fe9c4b1de5b-kube-api-access-zqm4j\") pod \"f1102ec7-7c4c-475c-b311-4fe9c4b1de5b\" (UID: \"f1102ec7-7c4c-475c-b311-4fe9c4b1de5b\") "
Sep 17 15:44:12.574198 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:12.574026    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1102ec7-7c4c-475c-b311-4fe9c4b1de5b-maas-api-tls\") pod \"f1102ec7-7c4c-475c-b311-4fe9c4b1de5b\" (UID: \"f1102ec7-7c4c-475c-b311-4fe9c4b1de5b\") "
Sep 17 15:44:12.574198 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:12.574108    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1102ec7-7c4c-475c-b311-4fe9c4b1de5b-metrics-certs\") pod \"f1102ec7-7c4c-475c-b311-4fe9c4b1de5b\" (UID: \"f1102ec7-7c4c-475c-b311-4fe9c4b1de5b\") "
Sep 17 15:44:12.574198 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:12.574152    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1102ec7-7c4c-475c-b311-4fe9c4b1de5b-openshift-service-ca\") pod \"f1102ec7-7c4c-475c-b311-4fe9c4b1de5b\" (UID: \"f1102ec7-7c4c-475c-b311-4fe9c4b1de5b\") "
Sep 17 15:44:12.574525 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:12.574496    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1102ec7-7c4c-475c-b311-4fe9c4b1de5b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f1102ec7-7c4c-475c-b311-4fe9c4b1de5b" (UID: "f1102ec7-7c4c-475c-b311-4fe9c4b1de5b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:44:12.576221 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:12.576176    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1102ec7-7c4c-475c-b311-4fe9c4b1de5b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f1102ec7-7c4c-475c-b311-4fe9c4b1de5b" (UID: "f1102ec7-7c4c-475c-b311-4fe9c4b1de5b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:44:12.576331 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:12.576190    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1102ec7-7c4c-475c-b311-4fe9c4b1de5b-kube-api-access-zqm4j" (OuterVolumeSpecName: "kube-api-access-zqm4j") pod "f1102ec7-7c4c-475c-b311-4fe9c4b1de5b" (UID: "f1102ec7-7c4c-475c-b311-4fe9c4b1de5b"). InnerVolumeSpecName "kube-api-access-zqm4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:44:12.576420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:12.576396    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1102ec7-7c4c-475c-b311-4fe9c4b1de5b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f1102ec7-7c4c-475c-b311-4fe9c4b1de5b" (UID: "f1102ec7-7c4c-475c-b311-4fe9c4b1de5b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:44:12.675032 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:12.675007    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqm4j\" (UniqueName: \"kubernetes.io/projected/f1102ec7-7c4c-475c-b311-4fe9c4b1de5b-kube-api-access-zqm4j\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:44:12.675032 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:12.675029    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1102ec7-7c4c-475c-b311-4fe9c4b1de5b-maas-api-tls\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:44:12.675032 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:12.675040    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1102ec7-7c4c-475c-b311-4fe9c4b1de5b-metrics-certs\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:44:12.675258 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:12.675050    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1102ec7-7c4c-475c-b311-4fe9c4b1de5b-openshift-service-ca\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:44:13.386188 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:13.386155    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb"
Sep 17 15:44:13.386188 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:13.386177    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb" event={"ID":"f1102ec7-7c4c-475c-b311-4fe9c4b1de5b","Type":"ContainerDied","Data":"a7065043989b98d4f4b430b4ea2e46a69a110c97f123457bd6ed4c58c9466cd0"}
Sep 17 15:44:13.386618 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:13.386238    2556 scope.go:117] "RemoveContainer" containerID="5ff9a4e808e6398ae8b2cf36badcff3e0e858699a971cf6064a7fb6bb46f78b9"
Sep 17 15:44:13.411046 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:13.411024    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb"]
Sep 17 15:44:13.416743 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:13.416706    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-49b90ab7-844b9b65c9-xznsb"]
Sep 17 15:44:14.135893 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:14.135863    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1102ec7-7c4c-475c-b311-4fe9c4b1de5b" path="/var/lib/kubelet/pods/f1102ec7-7c4c-475c-b311-4fe9c4b1de5b/volumes"
Sep 17 15:44:17.996908 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:17.996874    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"]
Sep 17 15:44:17.997314 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:17.997302    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1102ec7-7c4c-475c-b311-4fe9c4b1de5b" containerName="maas-api"
Sep 17 15:44:17.997314 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:17.997315    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1102ec7-7c4c-475c-b311-4fe9c4b1de5b" containerName="maas-api"
Sep 17 15:44:17.997410 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:17.997401    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1102ec7-7c4c-475c-b311-4fe9c4b1de5b" containerName="maas-api"
Sep 17 15:44:18.001493 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:18.001472    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"
Sep 17 15:44:18.004972 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:18.004949    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-449c47\"/\"default-dockercfg-gqln6\""
Sep 17 15:44:18.005129 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:18.004987    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-449c47\"/\"openshift-service-ca.crt\""
Sep 17 15:44:18.005630 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:18.005611    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-449c47\"/\"test-model-449c47-kserve-self-signed-certs\""
Sep 17 15:44:18.005713 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:18.005631    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-449c47\"/\"kube-root-ca.crt\""
Sep 17 15:44:18.013921 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:18.013897    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"]
Sep 17 15:44:18.119117 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:18.119084    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0301bd68-cb45-4a20-89fe-7399a16436ba-model-cache\") pod \"test-model-449c47-kserve-57546bd5c9-cq4dv\" (UID: \"0301bd68-cb45-4a20-89fe-7399a16436ba\") " pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"
Sep 17 15:44:18.119283 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:18.119157    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtlgk\" (UniqueName: \"kubernetes.io/projected/0301bd68-cb45-4a20-89fe-7399a16436ba-kube-api-access-jtlgk\") pod \"test-model-449c47-kserve-57546bd5c9-cq4dv\" (UID: \"0301bd68-cb45-4a20-89fe-7399a16436ba\") " pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"
Sep 17 15:44:18.119327 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:18.119283    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0301bd68-cb45-4a20-89fe-7399a16436ba-dshm\") pod \"test-model-449c47-kserve-57546bd5c9-cq4dv\" (UID: \"0301bd68-cb45-4a20-89fe-7399a16436ba\") " pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"
Sep 17 15:44:18.119327 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:18.119308    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0301bd68-cb45-4a20-89fe-7399a16436ba-tmp-dir\") pod \"test-model-449c47-kserve-57546bd5c9-cq4dv\" (UID: \"0301bd68-cb45-4a20-89fe-7399a16436ba\") " pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"
Sep 17 15:44:18.119391 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:18.119329    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0301bd68-cb45-4a20-89fe-7399a16436ba-tls-certs\") pod \"test-model-449c47-kserve-57546bd5c9-cq4dv\" (UID: \"0301bd68-cb45-4a20-89fe-7399a16436ba\") " pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"
Sep 17 15:44:18.119391 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:18.119354    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0301bd68-cb45-4a20-89fe-7399a16436ba-home\") pod \"test-model-449c47-kserve-57546bd5c9-cq4dv\" (UID: \"0301bd68-cb45-4a20-89fe-7399a16436ba\") " pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"
Sep 17 15:44:18.220164 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:18.220134    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0301bd68-cb45-4a20-89fe-7399a16436ba-dshm\") pod \"test-model-449c47-kserve-57546bd5c9-cq4dv\" (UID: \"0301bd68-cb45-4a20-89fe-7399a16436ba\") " pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"
Sep 17 15:44:18.220164 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:18.220168    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0301bd68-cb45-4a20-89fe-7399a16436ba-tmp-dir\") pod \"test-model-449c47-kserve-57546bd5c9-cq4dv\" (UID: \"0301bd68-cb45-4a20-89fe-7399a16436ba\") " pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"
Sep 17 15:44:18.220406 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:18.220186    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0301bd68-cb45-4a20-89fe-7399a16436ba-tls-certs\") pod \"test-model-449c47-kserve-57546bd5c9-cq4dv\" (UID: \"0301bd68-cb45-4a20-89fe-7399a16436ba\") " pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"
Sep 17 15:44:18.220406 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:18.220204    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0301bd68-cb45-4a20-89fe-7399a16436ba-home\") pod \"test-model-449c47-kserve-57546bd5c9-cq4dv\" (UID: \"0301bd68-cb45-4a20-89fe-7399a16436ba\") " pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"
Sep 17 15:44:18.220406 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:18.220243    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0301bd68-cb45-4a20-89fe-7399a16436ba-model-cache\") pod \"test-model-449c47-kserve-57546bd5c9-cq4dv\" (UID: \"0301bd68-cb45-4a20-89fe-7399a16436ba\") " pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"
Sep 17 15:44:18.220406 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:18.220279    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtlgk\" (UniqueName: \"kubernetes.io/projected/0301bd68-cb45-4a20-89fe-7399a16436ba-kube-api-access-jtlgk\") pod \"test-model-449c47-kserve-57546bd5c9-cq4dv\" (UID: \"0301bd68-cb45-4a20-89fe-7399a16436ba\") " pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"
Sep 17 15:44:18.220711 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:18.220688    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0301bd68-cb45-4a20-89fe-7399a16436ba-model-cache\") pod \"test-model-449c47-kserve-57546bd5c9-cq4dv\" (UID: \"0301bd68-cb45-4a20-89fe-7399a16436ba\") " pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"
Sep 17 15:44:18.220778 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:18.220718    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0301bd68-cb45-4a20-89fe-7399a16436ba-home\") pod \"test-model-449c47-kserve-57546bd5c9-cq4dv\" (UID: \"0301bd68-cb45-4a20-89fe-7399a16436ba\") " pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"
Sep 17 15:44:18.220778 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:18.220732    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0301bd68-cb45-4a20-89fe-7399a16436ba-tmp-dir\") pod \"test-model-449c47-kserve-57546bd5c9-cq4dv\" (UID: \"0301bd68-cb45-4a20-89fe-7399a16436ba\") " pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"
Sep 17 15:44:18.222403 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:18.222379    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0301bd68-cb45-4a20-89fe-7399a16436ba-dshm\") pod \"test-model-449c47-kserve-57546bd5c9-cq4dv\" (UID: \"0301bd68-cb45-4a20-89fe-7399a16436ba\") " pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"
Sep 17 15:44:18.222786 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:18.222764    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0301bd68-cb45-4a20-89fe-7399a16436ba-tls-certs\") pod \"test-model-449c47-kserve-57546bd5c9-cq4dv\" (UID: \"0301bd68-cb45-4a20-89fe-7399a16436ba\") " pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"
Sep 17 15:44:18.228961 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:18.228936    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtlgk\" (UniqueName: \"kubernetes.io/projected/0301bd68-cb45-4a20-89fe-7399a16436ba-kube-api-access-jtlgk\") pod \"test-model-449c47-kserve-57546bd5c9-cq4dv\" (UID: \"0301bd68-cb45-4a20-89fe-7399a16436ba\") " pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"
Sep 17 15:44:18.310861 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:18.310786    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"
Sep 17 15:44:18.592249 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:18.592223    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"]
Sep 17 15:44:18.594044 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:44:18.594016    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0301bd68_cb45_4a20_89fe_7399a16436ba.slice/crio-1ae5c20f90a52576a6208c6908606a48cb700d283eb7e921d393c20100cb1738 WatchSource:0}: Error finding container 1ae5c20f90a52576a6208c6908606a48cb700d283eb7e921d393c20100cb1738: Status 404 returned error can't find the container with id 1ae5c20f90a52576a6208c6908606a48cb700d283eb7e921d393c20100cb1738
Sep 17 15:44:19.166587 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:19.166556    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj"]
Sep 17 15:44:19.170314 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:19.170291    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj"
Sep 17 15:44:19.174685 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:19.174667    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6b2abe85-serving-cert\""
Sep 17 15:44:19.189437 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:19.189362    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj"]
Sep 17 15:44:19.239140 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:19.239103    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85m59\" (UniqueName: \"kubernetes.io/projected/b09ba505-7dee-4fc9-9a2b-3cdad8de4578-kube-api-access-85m59\") pod \"maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj\" (UID: \"b09ba505-7dee-4fc9-9a2b-3cdad8de4578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj"
Sep 17 15:44:19.239427 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:19.239408    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b09ba505-7dee-4fc9-9a2b-3cdad8de4578-metrics-certs\") pod \"maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj\" (UID: \"b09ba505-7dee-4fc9-9a2b-3cdad8de4578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj"
Sep 17 15:44:19.239845 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:19.239815    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b09ba505-7dee-4fc9-9a2b-3cdad8de4578-maas-api-tls\") pod \"maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj\" (UID: \"b09ba505-7dee-4fc9-9a2b-3cdad8de4578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj"
Sep 17 15:44:19.239943 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:19.239899    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b09ba505-7dee-4fc9-9a2b-3cdad8de4578-openshift-service-ca\") pod \"maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj\" (UID: \"b09ba505-7dee-4fc9-9a2b-3cdad8de4578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj"
Sep 17 15:44:19.341096 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:19.341053    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b09ba505-7dee-4fc9-9a2b-3cdad8de4578-maas-api-tls\") pod \"maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj\" (UID: \"b09ba505-7dee-4fc9-9a2b-3cdad8de4578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj"
Sep 17 15:44:19.341242 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:19.341106    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b09ba505-7dee-4fc9-9a2b-3cdad8de4578-openshift-service-ca\") pod \"maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj\" (UID: \"b09ba505-7dee-4fc9-9a2b-3cdad8de4578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj"
Sep 17 15:44:19.341242 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:44:19.341198    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-serving-cert: secret "maas-api-e2e-ait-6b2abe85-serving-cert" not found
Sep 17 15:44:19.341361 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:19.341241    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85m59\" (UniqueName: \"kubernetes.io/projected/b09ba505-7dee-4fc9-9a2b-3cdad8de4578-kube-api-access-85m59\") pod \"maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj\" (UID: \"b09ba505-7dee-4fc9-9a2b-3cdad8de4578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj"
Sep 17 15:44:19.341361 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:44:19.341266    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b09ba505-7dee-4fc9-9a2b-3cdad8de4578-maas-api-tls podName:b09ba505-7dee-4fc9-9a2b-3cdad8de4578 nodeName:}" failed. No retries permitted until 2026-09-17 15:44:19.841246286 +0000 UTC m=+882.221264036 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b09ba505-7dee-4fc9-9a2b-3cdad8de4578-maas-api-tls") pod "maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj" (UID: "b09ba505-7dee-4fc9-9a2b-3cdad8de4578") : secret "maas-api-e2e-ait-6b2abe85-serving-cert" not found
Sep 17 15:44:19.341466 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:19.341405    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b09ba505-7dee-4fc9-9a2b-3cdad8de4578-metrics-certs\") pod \"maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj\" (UID: \"b09ba505-7dee-4fc9-9a2b-3cdad8de4578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj"
Sep 17 15:44:19.341655 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:19.341636    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b09ba505-7dee-4fc9-9a2b-3cdad8de4578-openshift-service-ca\") pod \"maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj\" (UID: \"b09ba505-7dee-4fc9-9a2b-3cdad8de4578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj"
Sep 17 15:44:19.343653 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:19.343633    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b09ba505-7dee-4fc9-9a2b-3cdad8de4578-metrics-certs\") pod \"maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj\" (UID: \"b09ba505-7dee-4fc9-9a2b-3cdad8de4578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj"
Sep 17 15:44:19.352442 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:19.352421    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85m59\" (UniqueName: \"kubernetes.io/projected/b09ba505-7dee-4fc9-9a2b-3cdad8de4578-kube-api-access-85m59\") pod \"maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj\" (UID: \"b09ba505-7dee-4fc9-9a2b-3cdad8de4578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj"
Sep 17 15:44:19.419987 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:19.419924    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv" event={"ID":"0301bd68-cb45-4a20-89fe-7399a16436ba","Type":"ContainerStarted","Data":"2e62f0ad8285077a0c51a36c5738c2b1d7ce7af6f38d551002ddc99759133ba6"}
Sep 17 15:44:19.419987 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:19.419970    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv" event={"ID":"0301bd68-cb45-4a20-89fe-7399a16436ba","Type":"ContainerStarted","Data":"1ae5c20f90a52576a6208c6908606a48cb700d283eb7e921d393c20100cb1738"}
Sep 17 15:44:19.440933 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:19.440890    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv" podStartSLOduration=2.440876843 podStartE2EDuration="2.440876843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:44:19.438531395 +0000 UTC m=+881.818549148" watchObservedRunningTime="2026-09-17 15:44:19.440876843 +0000 UTC m=+881.820894597"
Sep 17 15:44:19.846265 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:19.846187    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b09ba505-7dee-4fc9-9a2b-3cdad8de4578-maas-api-tls\") pod \"maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj\" (UID: \"b09ba505-7dee-4fc9-9a2b-3cdad8de4578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj"
Sep 17 15:44:19.848433 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:19.848414    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b09ba505-7dee-4fc9-9a2b-3cdad8de4578-maas-api-tls\") pod \"maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj\" (UID: \"b09ba505-7dee-4fc9-9a2b-3cdad8de4578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj"
Sep 17 15:44:20.080867 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:20.080837    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj"
Sep 17 15:44:20.357470 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:20.357447    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj"]
Sep 17 15:44:20.359432 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:44:20.359406    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb09ba505_7dee_4fc9_9a2b_3cdad8de4578.slice/crio-a6a85269510279e884aa2415ad1facc177bae047076f3227cfce8a791344c642 WatchSource:0}: Error finding container a6a85269510279e884aa2415ad1facc177bae047076f3227cfce8a791344c642: Status 404 returned error can't find the container with id a6a85269510279e884aa2415ad1facc177bae047076f3227cfce8a791344c642
Sep 17 15:44:20.425657 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:20.425590    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj" event={"ID":"b09ba505-7dee-4fc9-9a2b-3cdad8de4578","Type":"ContainerStarted","Data":"a6a85269510279e884aa2415ad1facc177bae047076f3227cfce8a791344c642"}
Sep 17 15:44:21.432334 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:21.432300    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj" event={"ID":"b09ba505-7dee-4fc9-9a2b-3cdad8de4578","Type":"ContainerStarted","Data":"fc1a5d74623e04d34f529a57408367179ac45ac0ff28587b1f064c8dde54cd3e"}
Sep 17 15:44:21.432725 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:21.432367    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj"
Sep 17 15:44:27.442695 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:27.442661    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj"
Sep 17 15:44:27.462604 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:27.462561    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj" podStartSLOduration=7.924816567 podStartE2EDuration="8.462548475s" totalImagesPullingTime="537.731908ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:44:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:44:21.455989236 +0000 UTC m=+883.836006992" watchObservedRunningTime="2026-09-17 15:44:27.462548475 +0000 UTC m=+889.842566212"
Sep 17 15:44:28.310897 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:28.310864    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"
Sep 17 15:44:28.311079 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:28.310910    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"
Sep 17 15:44:28.325434 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:28.325406    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"
Sep 17 15:44:28.478178 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:28.478151    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"
Sep 17 15:44:38.107928 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:38.107856    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-45zt7_2c069c9e-0856-46b0-ba10-67829c05ac94/console-operator/1.log"
Sep 17 15:44:38.110611 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:38.110584    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-45zt7_2c069c9e-0856-46b0-ba10-67829c05ac94/console-operator/1.log"
Sep 17 15:44:49.131160 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:49.131119    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd"]
Sep 17 15:44:49.131711 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:49.131337    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd" podUID="b007a856-3172-4b26-87b7-9e23fb2525ee" containerName="payload-pre-processing" containerID="cri-o://dec47b190e30ef90a7f7f76fedb47f755983a44b3ada21c327260d46e870cf6f" gracePeriod=30
Sep 17 15:44:49.398602 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:49.398546    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd"
Sep 17 15:44:49.497322 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:49.497293    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gt8zf\" (UniqueName: \"kubernetes.io/projected/b007a856-3172-4b26-87b7-9e23fb2525ee-kube-api-access-gt8zf\") pod \"b007a856-3172-4b26-87b7-9e23fb2525ee\" (UID: \"b007a856-3172-4b26-87b7-9e23fb2525ee\") "
Sep 17 15:44:49.497477 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:49.497403    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b007a856-3172-4b26-87b7-9e23fb2525ee-plugins-config-volume\") pod \"b007a856-3172-4b26-87b7-9e23fb2525ee\" (UID: \"b007a856-3172-4b26-87b7-9e23fb2525ee\") "
Sep 17 15:44:49.497781 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:49.497752    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b007a856-3172-4b26-87b7-9e23fb2525ee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b007a856-3172-4b26-87b7-9e23fb2525ee" (UID: "b007a856-3172-4b26-87b7-9e23fb2525ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:44:49.499456 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:49.499433    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b007a856-3172-4b26-87b7-9e23fb2525ee-kube-api-access-gt8zf" (OuterVolumeSpecName: "kube-api-access-gt8zf") pod "b007a856-3172-4b26-87b7-9e23fb2525ee" (UID: "b007a856-3172-4b26-87b7-9e23fb2525ee"). InnerVolumeSpecName "kube-api-access-gt8zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:44:49.573901 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:49.573873    2556 generic.go:356] "Generic (PLEG): container finished" podID="b007a856-3172-4b26-87b7-9e23fb2525ee" containerID="dec47b190e30ef90a7f7f76fedb47f755983a44b3ada21c327260d46e870cf6f" exitCode=0
Sep 17 15:44:49.574047 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:49.573920    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd" event={"ID":"b007a856-3172-4b26-87b7-9e23fb2525ee","Type":"ContainerDied","Data":"dec47b190e30ef90a7f7f76fedb47f755983a44b3ada21c327260d46e870cf6f"}
Sep 17 15:44:49.574047 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:49.573927    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd"
Sep 17 15:44:49.574047 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:49.573966    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd" event={"ID":"b007a856-3172-4b26-87b7-9e23fb2525ee","Type":"ContainerDied","Data":"68c8df98fa27336701b4f13e1ec61406340818b9d783a79c87c8b302628bb3c4"}
Sep 17 15:44:49.574047 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:49.573989    2556 scope.go:117] "RemoveContainer" containerID="dec47b190e30ef90a7f7f76fedb47f755983a44b3ada21c327260d46e870cf6f"
Sep 17 15:44:49.583106 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:49.583048    2556 scope.go:117] "RemoveContainer" containerID="dec47b190e30ef90a7f7f76fedb47f755983a44b3ada21c327260d46e870cf6f"
Sep 17 15:44:49.583369 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:44:49.583348    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dec47b190e30ef90a7f7f76fedb47f755983a44b3ada21c327260d46e870cf6f\": container with ID starting with dec47b190e30ef90a7f7f76fedb47f755983a44b3ada21c327260d46e870cf6f not found: ID does not exist" containerID="dec47b190e30ef90a7f7f76fedb47f755983a44b3ada21c327260d46e870cf6f"
Sep 17 15:44:49.583503 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:49.583378    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec47b190e30ef90a7f7f76fedb47f755983a44b3ada21c327260d46e870cf6f"} err="failed to get container status \"dec47b190e30ef90a7f7f76fedb47f755983a44b3ada21c327260d46e870cf6f\": rpc error: code = NotFound desc = could not find container \"dec47b190e30ef90a7f7f76fedb47f755983a44b3ada21c327260d46e870cf6f\": container with ID starting with dec47b190e30ef90a7f7f76fedb47f755983a44b3ada21c327260d46e870cf6f not found: ID does not exist"
Sep 17 15:44:49.598482 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:49.598457    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt8zf\" (UniqueName: \"kubernetes.io/projected/b007a856-3172-4b26-87b7-9e23fb2525ee-kube-api-access-gt8zf\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:44:49.598574 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:49.598488    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b007a856-3172-4b26-87b7-9e23fb2525ee-plugins-config-volume\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:44:49.599578 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:49.599562    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd"]
Sep 17 15:44:49.610376 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:49.610358    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-28eba6-5cbfdb995d-rncwd"]
Sep 17 15:44:50.136080 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:50.136031    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b007a856-3172-4b26-87b7-9e23fb2525ee" path="/var/lib/kubelet/pods/b007a856-3172-4b26-87b7-9e23fb2525ee/volumes"
Sep 17 15:44:58.113924 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:58.113896    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj"]
Sep 17 15:44:58.116377 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:58.114106    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj" podUID="b09ba505-7dee-4fc9-9a2b-3cdad8de4578" containerName="maas-api" containerID="cri-o://fc1a5d74623e04d34f529a57408367179ac45ac0ff28587b1f064c8dde54cd3e" gracePeriod=30
Sep 17 15:44:58.363074 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:58.363032    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj"
Sep 17 15:44:58.482163 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:58.482083    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85m59\" (UniqueName: \"kubernetes.io/projected/b09ba505-7dee-4fc9-9a2b-3cdad8de4578-kube-api-access-85m59\") pod \"b09ba505-7dee-4fc9-9a2b-3cdad8de4578\" (UID: \"b09ba505-7dee-4fc9-9a2b-3cdad8de4578\") "
Sep 17 15:44:58.482329 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:58.482171    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b09ba505-7dee-4fc9-9a2b-3cdad8de4578-metrics-certs\") pod \"b09ba505-7dee-4fc9-9a2b-3cdad8de4578\" (UID: \"b09ba505-7dee-4fc9-9a2b-3cdad8de4578\") "
Sep 17 15:44:58.482329 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:58.482207    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b09ba505-7dee-4fc9-9a2b-3cdad8de4578-maas-api-tls\") pod \"b09ba505-7dee-4fc9-9a2b-3cdad8de4578\" (UID: \"b09ba505-7dee-4fc9-9a2b-3cdad8de4578\") "
Sep 17 15:44:58.482329 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:58.482263    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b09ba505-7dee-4fc9-9a2b-3cdad8de4578-openshift-service-ca\") pod \"b09ba505-7dee-4fc9-9a2b-3cdad8de4578\" (UID: \"b09ba505-7dee-4fc9-9a2b-3cdad8de4578\") "
Sep 17 15:44:58.482757 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:58.482706    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b09ba505-7dee-4fc9-9a2b-3cdad8de4578-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b09ba505-7dee-4fc9-9a2b-3cdad8de4578" (UID: "b09ba505-7dee-4fc9-9a2b-3cdad8de4578"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:44:58.484256 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:58.484227    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b09ba505-7dee-4fc9-9a2b-3cdad8de4578-kube-api-access-85m59" (OuterVolumeSpecName: "kube-api-access-85m59") pod "b09ba505-7dee-4fc9-9a2b-3cdad8de4578" (UID: "b09ba505-7dee-4fc9-9a2b-3cdad8de4578"). InnerVolumeSpecName "kube-api-access-85m59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:44:58.484679 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:58.484658    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b09ba505-7dee-4fc9-9a2b-3cdad8de4578-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b09ba505-7dee-4fc9-9a2b-3cdad8de4578" (UID: "b09ba505-7dee-4fc9-9a2b-3cdad8de4578"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:44:58.484750 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:58.484679    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b09ba505-7dee-4fc9-9a2b-3cdad8de4578-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b09ba505-7dee-4fc9-9a2b-3cdad8de4578" (UID: "b09ba505-7dee-4fc9-9a2b-3cdad8de4578"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:44:58.583822 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:58.583793    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b09ba505-7dee-4fc9-9a2b-3cdad8de4578-metrics-certs\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:44:58.583822 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:58.583816    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b09ba505-7dee-4fc9-9a2b-3cdad8de4578-maas-api-tls\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:44:58.583822 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:58.583825    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b09ba505-7dee-4fc9-9a2b-3cdad8de4578-openshift-service-ca\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:44:58.584021 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:58.583833    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85m59\" (UniqueName: \"kubernetes.io/projected/b09ba505-7dee-4fc9-9a2b-3cdad8de4578-kube-api-access-85m59\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:44:58.627750 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:58.627720    2556 generic.go:356] "Generic (PLEG): container finished" podID="b09ba505-7dee-4fc9-9a2b-3cdad8de4578" containerID="fc1a5d74623e04d34f529a57408367179ac45ac0ff28587b1f064c8dde54cd3e" exitCode=0
Sep 17 15:44:58.627888 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:58.627764    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj"
Sep 17 15:44:58.627888 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:58.627803    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj" event={"ID":"b09ba505-7dee-4fc9-9a2b-3cdad8de4578","Type":"ContainerDied","Data":"fc1a5d74623e04d34f529a57408367179ac45ac0ff28587b1f064c8dde54cd3e"}
Sep 17 15:44:58.627888 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:58.627856    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj" event={"ID":"b09ba505-7dee-4fc9-9a2b-3cdad8de4578","Type":"ContainerDied","Data":"a6a85269510279e884aa2415ad1facc177bae047076f3227cfce8a791344c642"}
Sep 17 15:44:58.627888 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:58.627877    2556 scope.go:117] "RemoveContainer" containerID="fc1a5d74623e04d34f529a57408367179ac45ac0ff28587b1f064c8dde54cd3e"
Sep 17 15:44:58.637570 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:58.637552    2556 scope.go:117] "RemoveContainer" containerID="fc1a5d74623e04d34f529a57408367179ac45ac0ff28587b1f064c8dde54cd3e"
Sep 17 15:44:58.637820 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:44:58.637794    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc1a5d74623e04d34f529a57408367179ac45ac0ff28587b1f064c8dde54cd3e\": container with ID starting with fc1a5d74623e04d34f529a57408367179ac45ac0ff28587b1f064c8dde54cd3e not found: ID does not exist" containerID="fc1a5d74623e04d34f529a57408367179ac45ac0ff28587b1f064c8dde54cd3e"
Sep 17 15:44:58.637926 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:58.637817    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc1a5d74623e04d34f529a57408367179ac45ac0ff28587b1f064c8dde54cd3e"} err="failed to get container status \"fc1a5d74623e04d34f529a57408367179ac45ac0ff28587b1f064c8dde54cd3e\": rpc error: code = NotFound desc = could not find container \"fc1a5d74623e04d34f529a57408367179ac45ac0ff28587b1f064c8dde54cd3e\": container with ID starting with fc1a5d74623e04d34f529a57408367179ac45ac0ff28587b1f064c8dde54cd3e not found: ID does not exist"
Sep 17 15:44:58.652845 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:58.652822    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj"]
Sep 17 15:44:58.657583 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:44:58.657563    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6b2abe85-66d4554bf-8j5qj"]
Sep 17 15:45:00.136328 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:00.136293    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b09ba505-7dee-4fc9-9a2b-3cdad8de4578" path="/var/lib/kubelet/pods/b09ba505-7dee-4fc9-9a2b-3cdad8de4578/volumes"
Sep 17 15:45:20.057248 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.057210    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"]
Sep 17 15:45:20.057854 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.057655    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b09ba505-7dee-4fc9-9a2b-3cdad8de4578" containerName="maas-api"
Sep 17 15:45:20.057854 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.057667    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b09ba505-7dee-4fc9-9a2b-3cdad8de4578" containerName="maas-api"
Sep 17 15:45:20.057854 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.057689    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b007a856-3172-4b26-87b7-9e23fb2525ee" containerName="payload-pre-processing"
Sep 17 15:45:20.057854 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.057694    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b007a856-3172-4b26-87b7-9e23fb2525ee" containerName="payload-pre-processing"
Sep 17 15:45:20.057854 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.057761    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b09ba505-7dee-4fc9-9a2b-3cdad8de4578" containerName="maas-api"
Sep 17 15:45:20.057854 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.057772    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b007a856-3172-4b26-87b7-9e23fb2525ee" containerName="payload-pre-processing"
Sep 17 15:45:20.060942 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.060924    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"
Sep 17 15:45:20.064458 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.064428    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c86e9a\"/\"kube-root-ca.crt\""
Sep 17 15:45:20.064577 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.064428    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c86e9a\"/\"default-dockercfg-287z2\""
Sep 17 15:45:20.064577 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.064510    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c86e9a\"/\"premium-w4-c86e9a-kserve-self-signed-certs\""
Sep 17 15:45:20.064577 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.064525    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c86e9a\"/\"openshift-service-ca.crt\""
Sep 17 15:45:20.071394 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.071370    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"]
Sep 17 15:45:20.186156 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.186123    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1341d441-3541-4f1a-8e1a-9a8fe9f94398-tls-certs\") pod \"premium-w4-c86e9a-kserve-856c688968-4777n\" (UID: \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\") " pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"
Sep 17 15:45:20.186338 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.186175    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1341d441-3541-4f1a-8e1a-9a8fe9f94398-home\") pod \"premium-w4-c86e9a-kserve-856c688968-4777n\" (UID: \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\") " pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"
Sep 17 15:45:20.186338 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.186269    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnd6c\" (UniqueName: \"kubernetes.io/projected/1341d441-3541-4f1a-8e1a-9a8fe9f94398-kube-api-access-mnd6c\") pod \"premium-w4-c86e9a-kserve-856c688968-4777n\" (UID: \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\") " pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"
Sep 17 15:45:20.186338 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.186301    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1341d441-3541-4f1a-8e1a-9a8fe9f94398-tmp-dir\") pod \"premium-w4-c86e9a-kserve-856c688968-4777n\" (UID: \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\") " pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"
Sep 17 15:45:20.186338 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.186325    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1341d441-3541-4f1a-8e1a-9a8fe9f94398-model-cache\") pod \"premium-w4-c86e9a-kserve-856c688968-4777n\" (UID: \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\") " pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"
Sep 17 15:45:20.186534 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.186383    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1341d441-3541-4f1a-8e1a-9a8fe9f94398-dshm\") pod \"premium-w4-c86e9a-kserve-856c688968-4777n\" (UID: \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\") " pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"
Sep 17 15:45:20.287901 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.287859    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnd6c\" (UniqueName: \"kubernetes.io/projected/1341d441-3541-4f1a-8e1a-9a8fe9f94398-kube-api-access-mnd6c\") pod \"premium-w4-c86e9a-kserve-856c688968-4777n\" (UID: \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\") " pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"
Sep 17 15:45:20.288096 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.287925    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1341d441-3541-4f1a-8e1a-9a8fe9f94398-tmp-dir\") pod \"premium-w4-c86e9a-kserve-856c688968-4777n\" (UID: \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\") " pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"
Sep 17 15:45:20.288096 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.287944    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1341d441-3541-4f1a-8e1a-9a8fe9f94398-model-cache\") pod \"premium-w4-c86e9a-kserve-856c688968-4777n\" (UID: \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\") " pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"
Sep 17 15:45:20.288096 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.287971    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1341d441-3541-4f1a-8e1a-9a8fe9f94398-dshm\") pod \"premium-w4-c86e9a-kserve-856c688968-4777n\" (UID: \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\") " pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"
Sep 17 15:45:20.288096 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.288019    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1341d441-3541-4f1a-8e1a-9a8fe9f94398-tls-certs\") pod \"premium-w4-c86e9a-kserve-856c688968-4777n\" (UID: \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\") " pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"
Sep 17 15:45:20.288096 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.288093    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1341d441-3541-4f1a-8e1a-9a8fe9f94398-home\") pod \"premium-w4-c86e9a-kserve-856c688968-4777n\" (UID: \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\") " pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"
Sep 17 15:45:20.288394 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.288378    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1341d441-3541-4f1a-8e1a-9a8fe9f94398-model-cache\") pod \"premium-w4-c86e9a-kserve-856c688968-4777n\" (UID: \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\") " pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"
Sep 17 15:45:20.288496 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.288473    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1341d441-3541-4f1a-8e1a-9a8fe9f94398-home\") pod \"premium-w4-c86e9a-kserve-856c688968-4777n\" (UID: \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\") " pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"
Sep 17 15:45:20.288910 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.288890    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1341d441-3541-4f1a-8e1a-9a8fe9f94398-tmp-dir\") pod \"premium-w4-c86e9a-kserve-856c688968-4777n\" (UID: \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\") " pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"
Sep 17 15:45:20.290745 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.290720    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1341d441-3541-4f1a-8e1a-9a8fe9f94398-dshm\") pod \"premium-w4-c86e9a-kserve-856c688968-4777n\" (UID: \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\") " pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"
Sep 17 15:45:20.291131 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.291113    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1341d441-3541-4f1a-8e1a-9a8fe9f94398-tls-certs\") pod \"premium-w4-c86e9a-kserve-856c688968-4777n\" (UID: \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\") " pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"
Sep 17 15:45:20.299706 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.299677    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnd6c\" (UniqueName: \"kubernetes.io/projected/1341d441-3541-4f1a-8e1a-9a8fe9f94398-kube-api-access-mnd6c\") pod \"premium-w4-c86e9a-kserve-856c688968-4777n\" (UID: \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\") " pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"
Sep 17 15:45:20.372047 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.372015    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"
Sep 17 15:45:20.651664 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.651633    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"]
Sep 17 15:45:20.653264 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:45:20.653237    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1341d441_3541_4f1a_8e1a_9a8fe9f94398.slice/crio-8349346827bcfd65948ad83d314d94c884af99e89e8133ddafdf30583e43a261 WatchSource:0}: Error finding container 8349346827bcfd65948ad83d314d94c884af99e89e8133ddafdf30583e43a261: Status 404 returned error can't find the container with id 8349346827bcfd65948ad83d314d94c884af99e89e8133ddafdf30583e43a261
Sep 17 15:45:20.743322 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.743286    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n" event={"ID":"1341d441-3541-4f1a-8e1a-9a8fe9f94398","Type":"ContainerStarted","Data":"afcc73c029cfb69166aa73f25be0b7ea8d641e98568ddcbeca2c53d2b089506b"}
Sep 17 15:45:20.743472 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.743339    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n" event={"ID":"1341d441-3541-4f1a-8e1a-9a8fe9f94398","Type":"ContainerStarted","Data":"8349346827bcfd65948ad83d314d94c884af99e89e8133ddafdf30583e43a261"}
Sep 17 15:45:20.764033 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:20.763977    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n" podStartSLOduration=0.763960407 podStartE2EDuration="763.960407ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:45:20.762483196 +0000 UTC m=+943.142500948" watchObservedRunningTime="2026-09-17 15:45:20.763960407 +0000 UTC m=+943.143978163"
Sep 17 15:45:29.680643 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:29.680605    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"]
Sep 17 15:45:29.684035 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:29.684011    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"
Sep 17 15:45:29.686608 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:29.686588    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-849b71\"/\"openshift-service-ca.crt\""
Sep 17 15:45:29.686765 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:29.686741    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-849b71\"/\"default-dockercfg-k96cw\""
Sep 17 15:45:29.687714 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:29.687696    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-849b71\"/\"premium-w0-849b71-kserve-self-signed-certs\""
Sep 17 15:45:29.687821 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:29.687718    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-849b71\"/\"kube-root-ca.crt\""
Sep 17 15:45:29.702623 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:29.702594    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"]
Sep 17 15:45:29.773091 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:29.773042    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e2e1a5a-48d7-4164-adae-4d10011730fe-dshm\") pod \"premium-w0-849b71-kserve-6bb4fcbcbf-8fl68\" (UID: \"6e2e1a5a-48d7-4164-adae-4d10011730fe\") " pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"
Sep 17 15:45:29.773091 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:29.773093    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e2e1a5a-48d7-4164-adae-4d10011730fe-tmp-dir\") pod \"premium-w0-849b71-kserve-6bb4fcbcbf-8fl68\" (UID: \"6e2e1a5a-48d7-4164-adae-4d10011730fe\") " pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"
Sep 17 15:45:29.773285 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:29.773197    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e2e1a5a-48d7-4164-adae-4d10011730fe-home\") pod \"premium-w0-849b71-kserve-6bb4fcbcbf-8fl68\" (UID: \"6e2e1a5a-48d7-4164-adae-4d10011730fe\") " pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"
Sep 17 15:45:29.773285 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:29.773266    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e2e1a5a-48d7-4164-adae-4d10011730fe-model-cache\") pod \"premium-w0-849b71-kserve-6bb4fcbcbf-8fl68\" (UID: \"6e2e1a5a-48d7-4164-adae-4d10011730fe\") " pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"
Sep 17 15:45:29.773355 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:29.773335    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tt4r\" (UniqueName: \"kubernetes.io/projected/6e2e1a5a-48d7-4164-adae-4d10011730fe-kube-api-access-2tt4r\") pod \"premium-w0-849b71-kserve-6bb4fcbcbf-8fl68\" (UID: \"6e2e1a5a-48d7-4164-adae-4d10011730fe\") " pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"
Sep 17 15:45:29.773389 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:29.773367    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e2e1a5a-48d7-4164-adae-4d10011730fe-tls-certs\") pod \"premium-w0-849b71-kserve-6bb4fcbcbf-8fl68\" (UID: \"6e2e1a5a-48d7-4164-adae-4d10011730fe\") " pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"
Sep 17 15:45:29.874795 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:29.874761    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tt4r\" (UniqueName: \"kubernetes.io/projected/6e2e1a5a-48d7-4164-adae-4d10011730fe-kube-api-access-2tt4r\") pod \"premium-w0-849b71-kserve-6bb4fcbcbf-8fl68\" (UID: \"6e2e1a5a-48d7-4164-adae-4d10011730fe\") " pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"
Sep 17 15:45:29.874795 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:29.874802    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e2e1a5a-48d7-4164-adae-4d10011730fe-tls-certs\") pod \"premium-w0-849b71-kserve-6bb4fcbcbf-8fl68\" (UID: \"6e2e1a5a-48d7-4164-adae-4d10011730fe\") " pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"
Sep 17 15:45:29.875036 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:29.874829    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e2e1a5a-48d7-4164-adae-4d10011730fe-dshm\") pod \"premium-w0-849b71-kserve-6bb4fcbcbf-8fl68\" (UID: \"6e2e1a5a-48d7-4164-adae-4d10011730fe\") " pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"
Sep 17 15:45:29.875036 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:29.874848    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e2e1a5a-48d7-4164-adae-4d10011730fe-tmp-dir\") pod \"premium-w0-849b71-kserve-6bb4fcbcbf-8fl68\" (UID: \"6e2e1a5a-48d7-4164-adae-4d10011730fe\") " pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"
Sep 17 15:45:29.875036 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:29.874877    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e2e1a5a-48d7-4164-adae-4d10011730fe-home\") pod \"premium-w0-849b71-kserve-6bb4fcbcbf-8fl68\" (UID: \"6e2e1a5a-48d7-4164-adae-4d10011730fe\") " pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"
Sep 17 15:45:29.875036 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:29.874918    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e2e1a5a-48d7-4164-adae-4d10011730fe-model-cache\") pod \"premium-w0-849b71-kserve-6bb4fcbcbf-8fl68\" (UID: \"6e2e1a5a-48d7-4164-adae-4d10011730fe\") " pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"
Sep 17 15:45:29.875390 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:29.875365    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e2e1a5a-48d7-4164-adae-4d10011730fe-home\") pod \"premium-w0-849b71-kserve-6bb4fcbcbf-8fl68\" (UID: \"6e2e1a5a-48d7-4164-adae-4d10011730fe\") " pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"
Sep 17 15:45:29.875481 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:29.875389    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e2e1a5a-48d7-4164-adae-4d10011730fe-tmp-dir\") pod \"premium-w0-849b71-kserve-6bb4fcbcbf-8fl68\" (UID: \"6e2e1a5a-48d7-4164-adae-4d10011730fe\") " pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"
Sep 17 15:45:29.875481 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:29.875424    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e2e1a5a-48d7-4164-adae-4d10011730fe-model-cache\") pod \"premium-w0-849b71-kserve-6bb4fcbcbf-8fl68\" (UID: \"6e2e1a5a-48d7-4164-adae-4d10011730fe\") " pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"
Sep 17 15:45:29.877108 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:29.877083    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e2e1a5a-48d7-4164-adae-4d10011730fe-dshm\") pod \"premium-w0-849b71-kserve-6bb4fcbcbf-8fl68\" (UID: \"6e2e1a5a-48d7-4164-adae-4d10011730fe\") " pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"
Sep 17 15:45:29.877433 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:29.877415    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e2e1a5a-48d7-4164-adae-4d10011730fe-tls-certs\") pod \"premium-w0-849b71-kserve-6bb4fcbcbf-8fl68\" (UID: \"6e2e1a5a-48d7-4164-adae-4d10011730fe\") " pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"
Sep 17 15:45:29.887843 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:29.887819    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tt4r\" (UniqueName: \"kubernetes.io/projected/6e2e1a5a-48d7-4164-adae-4d10011730fe-kube-api-access-2tt4r\") pod \"premium-w0-849b71-kserve-6bb4fcbcbf-8fl68\" (UID: \"6e2e1a5a-48d7-4164-adae-4d10011730fe\") " pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"
Sep 17 15:45:29.995304 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:29.995221    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"
Sep 17 15:45:30.268618 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:30.268594    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"]
Sep 17 15:45:30.270648 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:45:30.270620    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e2e1a5a_48d7_4164_adae_4d10011730fe.slice/crio-6310d026a7779fa95c932b27fe7a3c7333ecec200cb88e04c4dcfd1d89ccbe1b WatchSource:0}: Error finding container 6310d026a7779fa95c932b27fe7a3c7333ecec200cb88e04c4dcfd1d89ccbe1b: Status 404 returned error can't find the container with id 6310d026a7779fa95c932b27fe7a3c7333ecec200cb88e04c4dcfd1d89ccbe1b
Sep 17 15:45:30.372543 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:30.372509    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"
Sep 17 15:45:30.372781 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:30.372762    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"
Sep 17 15:45:30.384827 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:30.384801    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"
Sep 17 15:45:30.793543 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:30.793509    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68" event={"ID":"6e2e1a5a-48d7-4164-adae-4d10011730fe","Type":"ContainerStarted","Data":"efec482158f292cb20d951c4b791fd398a6c83c159fd54263fc19bc4763ef1ea"}
Sep 17 15:45:30.794002 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:30.793557    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68" event={"ID":"6e2e1a5a-48d7-4164-adae-4d10011730fe","Type":"ContainerStarted","Data":"6310d026a7779fa95c932b27fe7a3c7333ecec200cb88e04c4dcfd1d89ccbe1b"}
Sep 17 15:45:30.804772 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:30.804741    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"
Sep 17 15:45:30.819179 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:30.819125    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68" podStartSLOduration=1.8191109060000001 podStartE2EDuration="1.819110906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:45:30.818048727 +0000 UTC m=+953.198066482" watchObservedRunningTime="2026-09-17 15:45:30.819110906 +0000 UTC m=+953.199128660"
Sep 17 15:45:39.607521 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:39.607485    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq"]
Sep 17 15:45:39.612420 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:39.612401    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq"
Sep 17 15:45:39.614832 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:39.614815    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-5b08fa-dockercfg-zwmzx\""
Sep 17 15:45:39.615113 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:39.615091    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-5b08fa\""
Sep 17 15:45:39.624186 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:39.624164    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq"]
Sep 17 15:45:39.670967 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:39.670943    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1b590ff-24e0-4afd-90bf-7646f81f6755-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq\" (UID: \"e1b590ff-24e0-4afd-90bf-7646f81f6755\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq"
Sep 17 15:45:39.671114 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:39.670976    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4qhv\" (UniqueName: \"kubernetes.io/projected/e1b590ff-24e0-4afd-90bf-7646f81f6755-kube-api-access-x4qhv\") pod \"payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq\" (UID: \"e1b590ff-24e0-4afd-90bf-7646f81f6755\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq"
Sep 17 15:45:39.772458 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:39.772420    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1b590ff-24e0-4afd-90bf-7646f81f6755-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq\" (UID: \"e1b590ff-24e0-4afd-90bf-7646f81f6755\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq"
Sep 17 15:45:39.772458 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:39.772462    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4qhv\" (UniqueName: \"kubernetes.io/projected/e1b590ff-24e0-4afd-90bf-7646f81f6755-kube-api-access-x4qhv\") pod \"payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq\" (UID: \"e1b590ff-24e0-4afd-90bf-7646f81f6755\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq"
Sep 17 15:45:39.773402 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:39.773378    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1b590ff-24e0-4afd-90bf-7646f81f6755-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq\" (UID: \"e1b590ff-24e0-4afd-90bf-7646f81f6755\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq"
Sep 17 15:45:39.783952 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:39.783931    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4qhv\" (UniqueName: \"kubernetes.io/projected/e1b590ff-24e0-4afd-90bf-7646f81f6755-kube-api-access-x4qhv\") pod \"payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq\" (UID: \"e1b590ff-24e0-4afd-90bf-7646f81f6755\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq"
Sep 17 15:45:39.923491 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:39.923411    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq"
Sep 17 15:45:39.996000 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:39.995970    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"
Sep 17 15:45:39.996167 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:39.996041    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"
Sep 17 15:45:40.010774 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:40.010747    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"
Sep 17 15:45:40.206886 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:40.206864    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq"]
Sep 17 15:45:40.208777 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:45:40.208752    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1b590ff_24e0_4afd_90bf_7646f81f6755.slice/crio-c87e9bb92301f95483207244f96d15cae5a35565f5cfe080d6281efa38491db8 WatchSource:0}: Error finding container c87e9bb92301f95483207244f96d15cae5a35565f5cfe080d6281efa38491db8: Status 404 returned error can't find the container with id c87e9bb92301f95483207244f96d15cae5a35565f5cfe080d6281efa38491db8
Sep 17 15:45:40.851766 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:40.851736    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq" event={"ID":"e1b590ff-24e0-4afd-90bf-7646f81f6755","Type":"ContainerStarted","Data":"30f8b6129324de0cf9832f511663116f1e04e78b41d9cabf5de439d26f63876c"}
Sep 17 15:45:40.852227 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:40.851787    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq" event={"ID":"e1b590ff-24e0-4afd-90bf-7646f81f6755","Type":"ContainerStarted","Data":"c87e9bb92301f95483207244f96d15cae5a35565f5cfe080d6281efa38491db8"}
Sep 17 15:45:40.852227 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:40.851936    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq"
Sep 17 15:45:40.862502 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:40.862482    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"
Sep 17 15:45:40.874463 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:40.874418    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq" podStartSLOduration=1.8744031319999999 podStartE2EDuration="1.874403132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:45:40.8719843 +0000 UTC m=+963.252002079" watchObservedRunningTime="2026-09-17 15:45:40.874403132 +0000 UTC m=+963.254420894"
Sep 17 15:45:51.860914 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:45:51.860883    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq"
Sep 17 15:47:20.498594 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:20.498557    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq"]
Sep 17 15:47:20.499005 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:20.498759    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq" podUID="e1b590ff-24e0-4afd-90bf-7646f81f6755" containerName="payload-pre-processing" containerID="cri-o://30f8b6129324de0cf9832f511663116f1e04e78b41d9cabf5de439d26f63876c" gracePeriod=30
Sep 17 15:47:20.763685 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:20.763633    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq"
Sep 17 15:47:20.935656 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:20.935623    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4qhv\" (UniqueName: \"kubernetes.io/projected/e1b590ff-24e0-4afd-90bf-7646f81f6755-kube-api-access-x4qhv\") pod \"e1b590ff-24e0-4afd-90bf-7646f81f6755\" (UID: \"e1b590ff-24e0-4afd-90bf-7646f81f6755\") "
Sep 17 15:47:20.935848 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:20.935752    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1b590ff-24e0-4afd-90bf-7646f81f6755-plugins-config-volume\") pod \"e1b590ff-24e0-4afd-90bf-7646f81f6755\" (UID: \"e1b590ff-24e0-4afd-90bf-7646f81f6755\") "
Sep 17 15:47:20.936082 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:20.936041    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1b590ff-24e0-4afd-90bf-7646f81f6755-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e1b590ff-24e0-4afd-90bf-7646f81f6755" (UID: "e1b590ff-24e0-4afd-90bf-7646f81f6755"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:47:20.936170 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:20.936157    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1b590ff-24e0-4afd-90bf-7646f81f6755-plugins-config-volume\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:47:20.937700 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:20.937676    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1b590ff-24e0-4afd-90bf-7646f81f6755-kube-api-access-x4qhv" (OuterVolumeSpecName: "kube-api-access-x4qhv") pod "e1b590ff-24e0-4afd-90bf-7646f81f6755" (UID: "e1b590ff-24e0-4afd-90bf-7646f81f6755"). InnerVolumeSpecName "kube-api-access-x4qhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:47:21.037749 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:21.037671    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4qhv\" (UniqueName: \"kubernetes.io/projected/e1b590ff-24e0-4afd-90bf-7646f81f6755-kube-api-access-x4qhv\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:47:21.391255 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:21.391225    2556 generic.go:356] "Generic (PLEG): container finished" podID="e1b590ff-24e0-4afd-90bf-7646f81f6755" containerID="30f8b6129324de0cf9832f511663116f1e04e78b41d9cabf5de439d26f63876c" exitCode=0
Sep 17 15:47:21.391422 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:21.391280    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq"
Sep 17 15:47:21.391422 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:21.391311    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq" event={"ID":"e1b590ff-24e0-4afd-90bf-7646f81f6755","Type":"ContainerDied","Data":"30f8b6129324de0cf9832f511663116f1e04e78b41d9cabf5de439d26f63876c"}
Sep 17 15:47:21.391422 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:21.391366    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq" event={"ID":"e1b590ff-24e0-4afd-90bf-7646f81f6755","Type":"ContainerDied","Data":"c87e9bb92301f95483207244f96d15cae5a35565f5cfe080d6281efa38491db8"}
Sep 17 15:47:21.391422 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:21.391394    2556 scope.go:117] "RemoveContainer" containerID="30f8b6129324de0cf9832f511663116f1e04e78b41d9cabf5de439d26f63876c"
Sep 17 15:47:21.401169 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:21.401147    2556 scope.go:117] "RemoveContainer" containerID="30f8b6129324de0cf9832f511663116f1e04e78b41d9cabf5de439d26f63876c"
Sep 17 15:47:21.401468 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:47:21.401451    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30f8b6129324de0cf9832f511663116f1e04e78b41d9cabf5de439d26f63876c\": container with ID starting with 30f8b6129324de0cf9832f511663116f1e04e78b41d9cabf5de439d26f63876c not found: ID does not exist" containerID="30f8b6129324de0cf9832f511663116f1e04e78b41d9cabf5de439d26f63876c"
Sep 17 15:47:21.401534 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:21.401477    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30f8b6129324de0cf9832f511663116f1e04e78b41d9cabf5de439d26f63876c"} err="failed to get container status \"30f8b6129324de0cf9832f511663116f1e04e78b41d9cabf5de439d26f63876c\": rpc error: code = NotFound desc = could not find container \"30f8b6129324de0cf9832f511663116f1e04e78b41d9cabf5de439d26f63876c\": container with ID starting with 30f8b6129324de0cf9832f511663116f1e04e78b41d9cabf5de439d26f63876c not found: ID does not exist"
Sep 17 15:47:21.466916 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:21.466885    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq"]
Sep 17 15:47:21.475677 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:21.475653    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5b08fa-787896c79d-8pvtq"]
Sep 17 15:47:22.136330 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:22.136298    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1b590ff-24e0-4afd-90bf-7646f81f6755" path="/var/lib/kubelet/pods/e1b590ff-24e0-4afd-90bf-7646f81f6755/volumes"
Sep 17 15:47:42.680707 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:42.680673    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp"]
Sep 17 15:47:42.681308 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:42.681289    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1b590ff-24e0-4afd-90bf-7646f81f6755" containerName="payload-pre-processing"
Sep 17 15:47:42.681394 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:42.681312    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1b590ff-24e0-4afd-90bf-7646f81f6755" containerName="payload-pre-processing"
Sep 17 15:47:42.681475 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:42.681463    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1b590ff-24e0-4afd-90bf-7646f81f6755" containerName="payload-pre-processing"
Sep 17 15:47:42.684711 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:42.684691    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp"
Sep 17 15:47:42.688991 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:42.688965    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-70207142-serving-cert\""
Sep 17 15:47:42.733036 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:42.733013    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp"]
Sep 17 15:47:42.736589 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:42.736561    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsplq\" (UniqueName: \"kubernetes.io/projected/102c667c-4db9-4925-a6e8-3444f531f90a-kube-api-access-wsplq\") pod \"maas-api-e2e-mt-70207142-867cfd8b48-8b5qp\" (UID: \"102c667c-4db9-4925-a6e8-3444f531f90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp"
Sep 17 15:47:42.736703 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:42.736606    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/102c667c-4db9-4925-a6e8-3444f531f90a-maas-api-tls\") pod \"maas-api-e2e-mt-70207142-867cfd8b48-8b5qp\" (UID: \"102c667c-4db9-4925-a6e8-3444f531f90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp"
Sep 17 15:47:42.736703 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:42.736674    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/102c667c-4db9-4925-a6e8-3444f531f90a-metrics-certs\") pod \"maas-api-e2e-mt-70207142-867cfd8b48-8b5qp\" (UID: \"102c667c-4db9-4925-a6e8-3444f531f90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp"
Sep 17 15:47:42.736836 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:42.736770    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/102c667c-4db9-4925-a6e8-3444f531f90a-openshift-service-ca\") pod \"maas-api-e2e-mt-70207142-867cfd8b48-8b5qp\" (UID: \"102c667c-4db9-4925-a6e8-3444f531f90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp"
Sep 17 15:47:42.837622 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:42.837591    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsplq\" (UniqueName: \"kubernetes.io/projected/102c667c-4db9-4925-a6e8-3444f531f90a-kube-api-access-wsplq\") pod \"maas-api-e2e-mt-70207142-867cfd8b48-8b5qp\" (UID: \"102c667c-4db9-4925-a6e8-3444f531f90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp"
Sep 17 15:47:42.837622 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:42.837625    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/102c667c-4db9-4925-a6e8-3444f531f90a-maas-api-tls\") pod \"maas-api-e2e-mt-70207142-867cfd8b48-8b5qp\" (UID: \"102c667c-4db9-4925-a6e8-3444f531f90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp"
Sep 17 15:47:42.837791 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:42.837650    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/102c667c-4db9-4925-a6e8-3444f531f90a-metrics-certs\") pod \"maas-api-e2e-mt-70207142-867cfd8b48-8b5qp\" (UID: \"102c667c-4db9-4925-a6e8-3444f531f90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp"
Sep 17 15:47:42.837791 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:42.837684    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/102c667c-4db9-4925-a6e8-3444f531f90a-openshift-service-ca\") pod \"maas-api-e2e-mt-70207142-867cfd8b48-8b5qp\" (UID: \"102c667c-4db9-4925-a6e8-3444f531f90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp"
Sep 17 15:47:42.838445 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:42.838410    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/102c667c-4db9-4925-a6e8-3444f531f90a-openshift-service-ca\") pod \"maas-api-e2e-mt-70207142-867cfd8b48-8b5qp\" (UID: \"102c667c-4db9-4925-a6e8-3444f531f90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp"
Sep 17 15:47:42.840170 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:42.840148    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/102c667c-4db9-4925-a6e8-3444f531f90a-maas-api-tls\") pod \"maas-api-e2e-mt-70207142-867cfd8b48-8b5qp\" (UID: \"102c667c-4db9-4925-a6e8-3444f531f90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp"
Sep 17 15:47:42.840262 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:42.840168    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/102c667c-4db9-4925-a6e8-3444f531f90a-metrics-certs\") pod \"maas-api-e2e-mt-70207142-867cfd8b48-8b5qp\" (UID: \"102c667c-4db9-4925-a6e8-3444f531f90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp"
Sep 17 15:47:42.864893 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:42.864878    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsplq\" (UniqueName: \"kubernetes.io/projected/102c667c-4db9-4925-a6e8-3444f531f90a-kube-api-access-wsplq\") pod \"maas-api-e2e-mt-70207142-867cfd8b48-8b5qp\" (UID: \"102c667c-4db9-4925-a6e8-3444f531f90a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp"
Sep 17 15:47:42.994974 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:42.994901    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp"
Sep 17 15:47:43.271516 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:43.271492    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp"]
Sep 17 15:47:43.273985 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:47:43.273960    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod102c667c_4db9_4925_a6e8_3444f531f90a.slice/crio-2ef64412a82dbb9d85bc52b8a2102fa6d320ca5f410b7dcf26924b8f9ae80860 WatchSource:0}: Error finding container 2ef64412a82dbb9d85bc52b8a2102fa6d320ca5f410b7dcf26924b8f9ae80860: Status 404 returned error can't find the container with id 2ef64412a82dbb9d85bc52b8a2102fa6d320ca5f410b7dcf26924b8f9ae80860
Sep 17 15:47:43.275374 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:43.275357    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:47:43.508770 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:43.508735    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp" event={"ID":"102c667c-4db9-4925-a6e8-3444f531f90a","Type":"ContainerStarted","Data":"2ef64412a82dbb9d85bc52b8a2102fa6d320ca5f410b7dcf26924b8f9ae80860"}
Sep 17 15:47:44.515211 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:44.515174    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp" event={"ID":"102c667c-4db9-4925-a6e8-3444f531f90a","Type":"ContainerStarted","Data":"6c9123bb520604eee0977df2dbe686572381f8e7d877713451a7207f81e705b2"}
Sep 17 15:47:44.515684 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:44.515259    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp"
Sep 17 15:47:44.536863 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:44.536804    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp" podStartSLOduration=1.889942679 podStartE2EDuration="2.536787849s" totalImagesPullingTime="646.84517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:47:44.534342733 +0000 UTC m=+1086.914360487" watchObservedRunningTime="2026-09-17 15:47:44.536787849 +0000 UTC m=+1086.916805603"
Sep 17 15:47:50.526490 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:47:50.526461    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp"
Sep 17 15:48:27.650653 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:27.650620    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp"]
Sep 17 15:48:27.651234 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:27.650889    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp" podUID="102c667c-4db9-4925-a6e8-3444f531f90a" containerName="maas-api" containerID="cri-o://6c9123bb520604eee0977df2dbe686572381f8e7d877713451a7207f81e705b2" gracePeriod=30
Sep 17 15:48:27.912686 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:27.912629    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp"
Sep 17 15:48:27.975565 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:27.975537    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/102c667c-4db9-4925-a6e8-3444f531f90a-maas-api-tls\") pod \"102c667c-4db9-4925-a6e8-3444f531f90a\" (UID: \"102c667c-4db9-4925-a6e8-3444f531f90a\") "
Sep 17 15:48:27.975686 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:27.975624    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsplq\" (UniqueName: \"kubernetes.io/projected/102c667c-4db9-4925-a6e8-3444f531f90a-kube-api-access-wsplq\") pod \"102c667c-4db9-4925-a6e8-3444f531f90a\" (UID: \"102c667c-4db9-4925-a6e8-3444f531f90a\") "
Sep 17 15:48:27.975686 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:27.975676    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/102c667c-4db9-4925-a6e8-3444f531f90a-metrics-certs\") pod \"102c667c-4db9-4925-a6e8-3444f531f90a\" (UID: \"102c667c-4db9-4925-a6e8-3444f531f90a\") "
Sep 17 15:48:27.975768 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:27.975713    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/102c667c-4db9-4925-a6e8-3444f531f90a-openshift-service-ca\") pod \"102c667c-4db9-4925-a6e8-3444f531f90a\" (UID: \"102c667c-4db9-4925-a6e8-3444f531f90a\") "
Sep 17 15:48:27.976175 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:27.976145    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/102c667c-4db9-4925-a6e8-3444f531f90a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "102c667c-4db9-4925-a6e8-3444f531f90a" (UID: "102c667c-4db9-4925-a6e8-3444f531f90a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:48:27.977805 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:27.977702    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/102c667c-4db9-4925-a6e8-3444f531f90a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "102c667c-4db9-4925-a6e8-3444f531f90a" (UID: "102c667c-4db9-4925-a6e8-3444f531f90a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:48:27.977805 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:27.977759    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/102c667c-4db9-4925-a6e8-3444f531f90a-kube-api-access-wsplq" (OuterVolumeSpecName: "kube-api-access-wsplq") pod "102c667c-4db9-4925-a6e8-3444f531f90a" (UID: "102c667c-4db9-4925-a6e8-3444f531f90a"). InnerVolumeSpecName "kube-api-access-wsplq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:48:27.978087 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:27.977898    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/102c667c-4db9-4925-a6e8-3444f531f90a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "102c667c-4db9-4925-a6e8-3444f531f90a" (UID: "102c667c-4db9-4925-a6e8-3444f531f90a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:48:28.076672 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:28.076644    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsplq\" (UniqueName: \"kubernetes.io/projected/102c667c-4db9-4925-a6e8-3444f531f90a-kube-api-access-wsplq\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:48:28.076672 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:28.076667    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/102c667c-4db9-4925-a6e8-3444f531f90a-metrics-certs\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:48:28.076672 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:28.076676    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/102c667c-4db9-4925-a6e8-3444f531f90a-openshift-service-ca\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:48:28.076877 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:28.076686    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/102c667c-4db9-4925-a6e8-3444f531f90a-maas-api-tls\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:48:28.736144 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:28.736116    2556 generic.go:356] "Generic (PLEG): container finished" podID="102c667c-4db9-4925-a6e8-3444f531f90a" containerID="6c9123bb520604eee0977df2dbe686572381f8e7d877713451a7207f81e705b2" exitCode=0
Sep 17 15:48:28.736555 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:28.736160    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp"
Sep 17 15:48:28.736555 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:28.736192    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp" event={"ID":"102c667c-4db9-4925-a6e8-3444f531f90a","Type":"ContainerDied","Data":"6c9123bb520604eee0977df2dbe686572381f8e7d877713451a7207f81e705b2"}
Sep 17 15:48:28.736555 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:28.736242    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp" event={"ID":"102c667c-4db9-4925-a6e8-3444f531f90a","Type":"ContainerDied","Data":"2ef64412a82dbb9d85bc52b8a2102fa6d320ca5f410b7dcf26924b8f9ae80860"}
Sep 17 15:48:28.736555 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:28.736265    2556 scope.go:117] "RemoveContainer" containerID="6c9123bb520604eee0977df2dbe686572381f8e7d877713451a7207f81e705b2"
Sep 17 15:48:28.745334 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:28.745318    2556 scope.go:117] "RemoveContainer" containerID="6c9123bb520604eee0977df2dbe686572381f8e7d877713451a7207f81e705b2"
Sep 17 15:48:28.745623 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:48:28.745606    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c9123bb520604eee0977df2dbe686572381f8e7d877713451a7207f81e705b2\": container with ID starting with 6c9123bb520604eee0977df2dbe686572381f8e7d877713451a7207f81e705b2 not found: ID does not exist" containerID="6c9123bb520604eee0977df2dbe686572381f8e7d877713451a7207f81e705b2"
Sep 17 15:48:28.745674 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:28.745632    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c9123bb520604eee0977df2dbe686572381f8e7d877713451a7207f81e705b2"} err="failed to get container status \"6c9123bb520604eee0977df2dbe686572381f8e7d877713451a7207f81e705b2\": rpc error: code = NotFound desc = could not find container \"6c9123bb520604eee0977df2dbe686572381f8e7d877713451a7207f81e705b2\": container with ID starting with 6c9123bb520604eee0977df2dbe686572381f8e7d877713451a7207f81e705b2 not found: ID does not exist"
Sep 17 15:48:28.755445 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:28.755424    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp"]
Sep 17 15:48:28.760698 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:28.760678    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-70207142-867cfd8b48-8b5qp"]
Sep 17 15:48:30.143885 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:30.143854    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="102c667c-4db9-4925-a6e8-3444f531f90a" path="/var/lib/kubelet/pods/102c667c-4db9-4925-a6e8-3444f531f90a/volumes"
Sep 17 15:48:52.264512 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:52.264474    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"]
Sep 17 15:48:53.649767 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:53.649733    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm"]
Sep 17 15:48:53.866268 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:53.866205    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm" podUID="9ee62332-4b3b-4bd6-9af4-80fbff417114" containerName="maas-api" containerID="cri-o://9308697d06ed5af78cc1e79e9c5fba0ecbd32b6e6861c17b6651c131a9b12824" gracePeriod=30
Sep 17 15:48:54.120999 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:54.120975    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm"
Sep 17 15:48:54.214902 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:54.214870    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qxc2\" (UniqueName: \"kubernetes.io/projected/9ee62332-4b3b-4bd6-9af4-80fbff417114-kube-api-access-2qxc2\") pod \"9ee62332-4b3b-4bd6-9af4-80fbff417114\" (UID: \"9ee62332-4b3b-4bd6-9af4-80fbff417114\") "
Sep 17 15:48:54.214902 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:54.214907    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ee62332-4b3b-4bd6-9af4-80fbff417114-metrics-certs\") pod \"9ee62332-4b3b-4bd6-9af4-80fbff417114\" (UID: \"9ee62332-4b3b-4bd6-9af4-80fbff417114\") "
Sep 17 15:48:54.215172 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:54.214936    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ee62332-4b3b-4bd6-9af4-80fbff417114-maas-api-tls\") pod \"9ee62332-4b3b-4bd6-9af4-80fbff417114\" (UID: \"9ee62332-4b3b-4bd6-9af4-80fbff417114\") "
Sep 17 15:48:54.215172 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:54.214972    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ee62332-4b3b-4bd6-9af4-80fbff417114-openshift-service-ca\") pod \"9ee62332-4b3b-4bd6-9af4-80fbff417114\" (UID: \"9ee62332-4b3b-4bd6-9af4-80fbff417114\") "
Sep 17 15:48:54.215450 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:54.215421    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ee62332-4b3b-4bd6-9af4-80fbff417114-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9ee62332-4b3b-4bd6-9af4-80fbff417114" (UID: "9ee62332-4b3b-4bd6-9af4-80fbff417114"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:48:54.217100 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:54.217074    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ee62332-4b3b-4bd6-9af4-80fbff417114-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9ee62332-4b3b-4bd6-9af4-80fbff417114" (UID: "9ee62332-4b3b-4bd6-9af4-80fbff417114"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:48:54.217507 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:54.217482    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ee62332-4b3b-4bd6-9af4-80fbff417114-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9ee62332-4b3b-4bd6-9af4-80fbff417114" (UID: "9ee62332-4b3b-4bd6-9af4-80fbff417114"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:48:54.217507 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:54.217495    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee62332-4b3b-4bd6-9af4-80fbff417114-kube-api-access-2qxc2" (OuterVolumeSpecName: "kube-api-access-2qxc2") pod "9ee62332-4b3b-4bd6-9af4-80fbff417114" (UID: "9ee62332-4b3b-4bd6-9af4-80fbff417114"). InnerVolumeSpecName "kube-api-access-2qxc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:48:54.316554 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:54.316462    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qxc2\" (UniqueName: \"kubernetes.io/projected/9ee62332-4b3b-4bd6-9af4-80fbff417114-kube-api-access-2qxc2\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:48:54.316554 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:54.316503    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ee62332-4b3b-4bd6-9af4-80fbff417114-metrics-certs\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:48:54.316554 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:54.316518    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ee62332-4b3b-4bd6-9af4-80fbff417114-maas-api-tls\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:48:54.316554 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:54.316532    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ee62332-4b3b-4bd6-9af4-80fbff417114-openshift-service-ca\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:48:54.873049 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:54.873010    2556 generic.go:356] "Generic (PLEG): container finished" podID="9ee62332-4b3b-4bd6-9af4-80fbff417114" containerID="9308697d06ed5af78cc1e79e9c5fba0ecbd32b6e6861c17b6651c131a9b12824" exitCode=0
Sep 17 15:48:54.873491 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:54.873052    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm" event={"ID":"9ee62332-4b3b-4bd6-9af4-80fbff417114","Type":"ContainerDied","Data":"9308697d06ed5af78cc1e79e9c5fba0ecbd32b6e6861c17b6651c131a9b12824"}
Sep 17 15:48:54.873491 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:54.873120    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm" event={"ID":"9ee62332-4b3b-4bd6-9af4-80fbff417114","Type":"ContainerDied","Data":"8f74784129b4f3385040179111336c80cec7e0d6a4fc936ee2217169eac358ea"}
Sep 17 15:48:54.873491 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:54.873149    2556 scope.go:117] "RemoveContainer" containerID="9308697d06ed5af78cc1e79e9c5fba0ecbd32b6e6861c17b6651c131a9b12824"
Sep 17 15:48:54.873491 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:54.873070    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm"
Sep 17 15:48:54.882932 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:54.882912    2556 scope.go:117] "RemoveContainer" containerID="9308697d06ed5af78cc1e79e9c5fba0ecbd32b6e6861c17b6651c131a9b12824"
Sep 17 15:48:54.883284 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:48:54.883264    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9308697d06ed5af78cc1e79e9c5fba0ecbd32b6e6861c17b6651c131a9b12824\": container with ID starting with 9308697d06ed5af78cc1e79e9c5fba0ecbd32b6e6861c17b6651c131a9b12824 not found: ID does not exist" containerID="9308697d06ed5af78cc1e79e9c5fba0ecbd32b6e6861c17b6651c131a9b12824"
Sep 17 15:48:54.883378 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:54.883289    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9308697d06ed5af78cc1e79e9c5fba0ecbd32b6e6861c17b6651c131a9b12824"} err="failed to get container status \"9308697d06ed5af78cc1e79e9c5fba0ecbd32b6e6861c17b6651c131a9b12824\": rpc error: code = NotFound desc = could not find container \"9308697d06ed5af78cc1e79e9c5fba0ecbd32b6e6861c17b6651c131a9b12824\": container with ID starting with 9308697d06ed5af78cc1e79e9c5fba0ecbd32b6e6861c17b6651c131a9b12824 not found: ID does not exist"
Sep 17 15:48:54.901186 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:54.901151    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm"]
Sep 17 15:48:54.906129 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:54.906086    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-449c47-7c54c7bd84-l49rm"]
Sep 17 15:48:56.136611 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:56.136575    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ee62332-4b3b-4bd6-9af4-80fbff417114" path="/var/lib/kubelet/pods/9ee62332-4b3b-4bd6-9af4-80fbff417114/volumes"
Sep 17 15:48:58.218358 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:58.218327    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf"]
Sep 17 15:48:58.219304 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:58.219278    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="102c667c-4db9-4925-a6e8-3444f531f90a" containerName="maas-api"
Sep 17 15:48:58.219304 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:58.219304    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="102c667c-4db9-4925-a6e8-3444f531f90a" containerName="maas-api"
Sep 17 15:48:58.219478 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:58.219321    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ee62332-4b3b-4bd6-9af4-80fbff417114" containerName="maas-api"
Sep 17 15:48:58.219478 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:58.219332    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ee62332-4b3b-4bd6-9af4-80fbff417114" containerName="maas-api"
Sep 17 15:48:58.219576 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:58.219477    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ee62332-4b3b-4bd6-9af4-80fbff417114" containerName="maas-api"
Sep 17 15:48:58.219576 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:58.219490    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="102c667c-4db9-4925-a6e8-3444f531f90a" containerName="maas-api"
Sep 17 15:48:58.224016 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:58.223998    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf"
Sep 17 15:48:58.226342 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:58.226322    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-2b23c7-serving-cert\""
Sep 17 15:48:58.235611 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:58.235586    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf"]
Sep 17 15:48:58.256232 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:58.256201    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7227a335-5e51-4adf-a705-dafc223fd4fa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf\" (UID: \"7227a335-5e51-4adf-a705-dafc223fd4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf"
Sep 17 15:48:58.256356 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:58.256239    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6s74\" (UniqueName: \"kubernetes.io/projected/7227a335-5e51-4adf-a705-dafc223fd4fa-kube-api-access-z6s74\") pod \"maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf\" (UID: \"7227a335-5e51-4adf-a705-dafc223fd4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf"
Sep 17 15:48:58.256356 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:58.256289    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7227a335-5e51-4adf-a705-dafc223fd4fa-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf\" (UID: \"7227a335-5e51-4adf-a705-dafc223fd4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf"
Sep 17 15:48:58.256440 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:58.256347    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7227a335-5e51-4adf-a705-dafc223fd4fa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf\" (UID: \"7227a335-5e51-4adf-a705-dafc223fd4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf"
Sep 17 15:48:58.357175 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:58.357140    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6s74\" (UniqueName: \"kubernetes.io/projected/7227a335-5e51-4adf-a705-dafc223fd4fa-kube-api-access-z6s74\") pod \"maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf\" (UID: \"7227a335-5e51-4adf-a705-dafc223fd4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf"
Sep 17 15:48:58.357354 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:58.357206    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7227a335-5e51-4adf-a705-dafc223fd4fa-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf\" (UID: \"7227a335-5e51-4adf-a705-dafc223fd4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf"
Sep 17 15:48:58.357354 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:58.357237    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7227a335-5e51-4adf-a705-dafc223fd4fa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf\" (UID: \"7227a335-5e51-4adf-a705-dafc223fd4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf"
Sep 17 15:48:58.357354 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:58.357299    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7227a335-5e51-4adf-a705-dafc223fd4fa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf\" (UID: \"7227a335-5e51-4adf-a705-dafc223fd4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf"
Sep 17 15:48:58.357959 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:58.357930    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7227a335-5e51-4adf-a705-dafc223fd4fa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf\" (UID: \"7227a335-5e51-4adf-a705-dafc223fd4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf"
Sep 17 15:48:58.359694 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:58.359670    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7227a335-5e51-4adf-a705-dafc223fd4fa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf\" (UID: \"7227a335-5e51-4adf-a705-dafc223fd4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf"
Sep 17 15:48:58.359789 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:58.359678    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7227a335-5e51-4adf-a705-dafc223fd4fa-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf\" (UID: \"7227a335-5e51-4adf-a705-dafc223fd4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf"
Sep 17 15:48:58.365130 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:58.365112    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6s74\" (UniqueName: \"kubernetes.io/projected/7227a335-5e51-4adf-a705-dafc223fd4fa-kube-api-access-z6s74\") pod \"maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf\" (UID: \"7227a335-5e51-4adf-a705-dafc223fd4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf"
Sep 17 15:48:58.537804 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:58.537711    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf"
Sep 17 15:48:58.813728 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:58.813701    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf"]
Sep 17 15:48:58.815375 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:48:58.815349    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7227a335_5e51_4adf_a705_dafc223fd4fa.slice/crio-ee44edf8b4b18022844035a0e7041d0b5ccedbb2608dec0a4df1757b4f1a9254 WatchSource:0}: Error finding container ee44edf8b4b18022844035a0e7041d0b5ccedbb2608dec0a4df1757b4f1a9254: Status 404 returned error can't find the container with id ee44edf8b4b18022844035a0e7041d0b5ccedbb2608dec0a4df1757b4f1a9254
Sep 17 15:48:58.897445 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:58.897408    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf" event={"ID":"7227a335-5e51-4adf-a705-dafc223fd4fa","Type":"ContainerStarted","Data":"ee44edf8b4b18022844035a0e7041d0b5ccedbb2608dec0a4df1757b4f1a9254"}
Sep 17 15:48:59.903728 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:59.903689    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf" event={"ID":"7227a335-5e51-4adf-a705-dafc223fd4fa","Type":"ContainerStarted","Data":"01e4d3ce1137865cd08709d09069e9d0a0251773b99ce47180dccbcca8bee492"}
Sep 17 15:48:59.904126 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:59.903753    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf"
Sep 17 15:48:59.927836 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:48:59.927784    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf" podStartSLOduration=1.493065897 podStartE2EDuration="1.927768143s" totalImagesPullingTime="434.702246ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:48:59.926393242 +0000 UTC m=+1162.306410999" watchObservedRunningTime="2026-09-17 15:48:59.927768143 +0000 UTC m=+1162.307785939"
Sep 17 15:49:05.914508 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:05.914477    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf"
Sep 17 15:49:07.284167 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.284123    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv" podUID="0301bd68-cb45-4a20-89fe-7399a16436ba" containerName="main" containerID="cri-o://2e62f0ad8285077a0c51a36c5738c2b1d7ce7af6f38d551002ddc99759133ba6" gracePeriod=45
Sep 17 15:49:07.538102 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.538041    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"
Sep 17 15:49:07.651748 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.651712    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0301bd68-cb45-4a20-89fe-7399a16436ba-tmp-dir\") pod \"0301bd68-cb45-4a20-89fe-7399a16436ba\" (UID: \"0301bd68-cb45-4a20-89fe-7399a16436ba\") "
Sep 17 15:49:07.651748 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.651750    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0301bd68-cb45-4a20-89fe-7399a16436ba-model-cache\") pod \"0301bd68-cb45-4a20-89fe-7399a16436ba\" (UID: \"0301bd68-cb45-4a20-89fe-7399a16436ba\") "
Sep 17 15:49:07.651979 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.651792    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0301bd68-cb45-4a20-89fe-7399a16436ba-dshm\") pod \"0301bd68-cb45-4a20-89fe-7399a16436ba\" (UID: \"0301bd68-cb45-4a20-89fe-7399a16436ba\") "
Sep 17 15:49:07.651979 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.651823    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtlgk\" (UniqueName: \"kubernetes.io/projected/0301bd68-cb45-4a20-89fe-7399a16436ba-kube-api-access-jtlgk\") pod \"0301bd68-cb45-4a20-89fe-7399a16436ba\" (UID: \"0301bd68-cb45-4a20-89fe-7399a16436ba\") "
Sep 17 15:49:07.651979 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.651877    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0301bd68-cb45-4a20-89fe-7399a16436ba-tls-certs\") pod \"0301bd68-cb45-4a20-89fe-7399a16436ba\" (UID: \"0301bd68-cb45-4a20-89fe-7399a16436ba\") "
Sep 17 15:49:07.651979 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.651908    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0301bd68-cb45-4a20-89fe-7399a16436ba-home\") pod \"0301bd68-cb45-4a20-89fe-7399a16436ba\" (UID: \"0301bd68-cb45-4a20-89fe-7399a16436ba\") "
Sep 17 15:49:07.652233 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.652037    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0301bd68-cb45-4a20-89fe-7399a16436ba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0301bd68-cb45-4a20-89fe-7399a16436ba" (UID: "0301bd68-cb45-4a20-89fe-7399a16436ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:07.652233 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.652052    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0301bd68-cb45-4a20-89fe-7399a16436ba-model-cache" (OuterVolumeSpecName: "model-cache") pod "0301bd68-cb45-4a20-89fe-7399a16436ba" (UID: "0301bd68-cb45-4a20-89fe-7399a16436ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:07.652328 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.652231    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0301bd68-cb45-4a20-89fe-7399a16436ba-home" (OuterVolumeSpecName: "home") pod "0301bd68-cb45-4a20-89fe-7399a16436ba" (UID: "0301bd68-cb45-4a20-89fe-7399a16436ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:07.652372 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.652354    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0301bd68-cb45-4a20-89fe-7399a16436ba-home\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:49:07.652423 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.652375    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0301bd68-cb45-4a20-89fe-7399a16436ba-tmp-dir\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:49:07.652423 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.652389    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0301bd68-cb45-4a20-89fe-7399a16436ba-model-cache\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:49:07.654052 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.654028    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0301bd68-cb45-4a20-89fe-7399a16436ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0301bd68-cb45-4a20-89fe-7399a16436ba" (UID: "0301bd68-cb45-4a20-89fe-7399a16436ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:49:07.654336 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.654319    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0301bd68-cb45-4a20-89fe-7399a16436ba-dshm" (OuterVolumeSpecName: "dshm") pod "0301bd68-cb45-4a20-89fe-7399a16436ba" (UID: "0301bd68-cb45-4a20-89fe-7399a16436ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:07.654378 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.654346    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0301bd68-cb45-4a20-89fe-7399a16436ba-kube-api-access-jtlgk" (OuterVolumeSpecName: "kube-api-access-jtlgk") pod "0301bd68-cb45-4a20-89fe-7399a16436ba" (UID: "0301bd68-cb45-4a20-89fe-7399a16436ba"). InnerVolumeSpecName "kube-api-access-jtlgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:49:07.753569 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.753530    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0301bd68-cb45-4a20-89fe-7399a16436ba-dshm\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:49:07.753569 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.753561    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtlgk\" (UniqueName: \"kubernetes.io/projected/0301bd68-cb45-4a20-89fe-7399a16436ba-kube-api-access-jtlgk\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:49:07.753569 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.753572    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0301bd68-cb45-4a20-89fe-7399a16436ba-tls-certs\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:49:07.944293 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.944257    2556 generic.go:356] "Generic (PLEG): container finished" podID="0301bd68-cb45-4a20-89fe-7399a16436ba" containerID="2e62f0ad8285077a0c51a36c5738c2b1d7ce7af6f38d551002ddc99759133ba6" exitCode=0
Sep 17 15:49:07.944447 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.944302    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"
Sep 17 15:49:07.944447 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.944301    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv" event={"ID":"0301bd68-cb45-4a20-89fe-7399a16436ba","Type":"ContainerDied","Data":"2e62f0ad8285077a0c51a36c5738c2b1d7ce7af6f38d551002ddc99759133ba6"}
Sep 17 15:49:07.944447 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.944339    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv" event={"ID":"0301bd68-cb45-4a20-89fe-7399a16436ba","Type":"ContainerDied","Data":"1ae5c20f90a52576a6208c6908606a48cb700d283eb7e921d393c20100cb1738"}
Sep 17 15:49:07.944447 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.944360    2556 scope.go:117] "RemoveContainer" containerID="2e62f0ad8285077a0c51a36c5738c2b1d7ce7af6f38d551002ddc99759133ba6"
Sep 17 15:49:07.953296 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.953278    2556 scope.go:117] "RemoveContainer" containerID="2e62f0ad8285077a0c51a36c5738c2b1d7ce7af6f38d551002ddc99759133ba6"
Sep 17 15:49:07.953551 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:49:07.953534    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e62f0ad8285077a0c51a36c5738c2b1d7ce7af6f38d551002ddc99759133ba6\": container with ID starting with 2e62f0ad8285077a0c51a36c5738c2b1d7ce7af6f38d551002ddc99759133ba6 not found: ID does not exist" containerID="2e62f0ad8285077a0c51a36c5738c2b1d7ce7af6f38d551002ddc99759133ba6"
Sep 17 15:49:07.953589 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.953560    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e62f0ad8285077a0c51a36c5738c2b1d7ce7af6f38d551002ddc99759133ba6"} err="failed to get container status \"2e62f0ad8285077a0c51a36c5738c2b1d7ce7af6f38d551002ddc99759133ba6\": rpc error: code = NotFound desc = could not find container \"2e62f0ad8285077a0c51a36c5738c2b1d7ce7af6f38d551002ddc99759133ba6\": container with ID starting with 2e62f0ad8285077a0c51a36c5738c2b1d7ce7af6f38d551002ddc99759133ba6 not found: ID does not exist"
Sep 17 15:49:07.967202 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.967179    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"]
Sep 17 15:49:07.972728 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:07.972705    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-449c47/test-model-449c47-kserve-57546bd5c9-cq4dv"]
Sep 17 15:49:08.136539 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:08.136510    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0301bd68-cb45-4a20-89fe-7399a16436ba" path="/var/lib/kubelet/pods/0301bd68-cb45-4a20-89fe-7399a16436ba/volumes"
Sep 17 15:49:25.853573 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:25.853537    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp"]
Sep 17 15:49:25.854198 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:25.854179    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0301bd68-cb45-4a20-89fe-7399a16436ba" containerName="main"
Sep 17 15:49:25.854198 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:25.854199    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0301bd68-cb45-4a20-89fe-7399a16436ba" containerName="main"
Sep 17 15:49:25.854367 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:25.854316    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0301bd68-cb45-4a20-89fe-7399a16436ba" containerName="main"
Sep 17 15:49:25.859104 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:25.859083    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp"
Sep 17 15:49:25.861757 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:25.861739    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-608b97-serving-cert\""
Sep 17 15:49:25.872199 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:25.872179    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp"]
Sep 17 15:49:25.929644 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:25.929616    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42caa626-be8b-4d3b-9bdc-34eea6c345b9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp\" (UID: \"42caa626-be8b-4d3b-9bdc-34eea6c345b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp"
Sep 17 15:49:25.929780 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:25.929723    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42caa626-be8b-4d3b-9bdc-34eea6c345b9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp\" (UID: \"42caa626-be8b-4d3b-9bdc-34eea6c345b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp"
Sep 17 15:49:25.929780 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:25.929753    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjt2g\" (UniqueName: \"kubernetes.io/projected/42caa626-be8b-4d3b-9bdc-34eea6c345b9-kube-api-access-rjt2g\") pod \"maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp\" (UID: \"42caa626-be8b-4d3b-9bdc-34eea6c345b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp"
Sep 17 15:49:25.929880 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:25.929797    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42caa626-be8b-4d3b-9bdc-34eea6c345b9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp\" (UID: \"42caa626-be8b-4d3b-9bdc-34eea6c345b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp"
Sep 17 15:49:26.030404 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:26.030370    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42caa626-be8b-4d3b-9bdc-34eea6c345b9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp\" (UID: \"42caa626-be8b-4d3b-9bdc-34eea6c345b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp"
Sep 17 15:49:26.030404 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:26.030409    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjt2g\" (UniqueName: \"kubernetes.io/projected/42caa626-be8b-4d3b-9bdc-34eea6c345b9-kube-api-access-rjt2g\") pod \"maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp\" (UID: \"42caa626-be8b-4d3b-9bdc-34eea6c345b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp"
Sep 17 15:49:26.030647 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:26.030464    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42caa626-be8b-4d3b-9bdc-34eea6c345b9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp\" (UID: \"42caa626-be8b-4d3b-9bdc-34eea6c345b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp"
Sep 17 15:49:26.030647 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:26.030512    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42caa626-be8b-4d3b-9bdc-34eea6c345b9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp\" (UID: \"42caa626-be8b-4d3b-9bdc-34eea6c345b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp"
Sep 17 15:49:26.030647 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:49:26.030623    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-serving-cert: secret "maas-api-e2e-shared-b-w2-608b97-serving-cert" not found
Sep 17 15:49:26.030798 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:49:26.030682    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42caa626-be8b-4d3b-9bdc-34eea6c345b9-maas-api-tls podName:42caa626-be8b-4d3b-9bdc-34eea6c345b9 nodeName:}" failed. No retries permitted until 2026-09-17 15:49:26.530662939 +0000 UTC m=+1188.910680680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/42caa626-be8b-4d3b-9bdc-34eea6c345b9-maas-api-tls") pod "maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp" (UID: "42caa626-be8b-4d3b-9bdc-34eea6c345b9") : secret "maas-api-e2e-shared-b-w2-608b97-serving-cert" not found
Sep 17 15:49:26.031090 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:26.031055    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42caa626-be8b-4d3b-9bdc-34eea6c345b9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp\" (UID: \"42caa626-be8b-4d3b-9bdc-34eea6c345b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp"
Sep 17 15:49:26.033127 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:26.033104    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42caa626-be8b-4d3b-9bdc-34eea6c345b9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp\" (UID: \"42caa626-be8b-4d3b-9bdc-34eea6c345b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp"
Sep 17 15:49:26.044224 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:26.044203    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjt2g\" (UniqueName: \"kubernetes.io/projected/42caa626-be8b-4d3b-9bdc-34eea6c345b9-kube-api-access-rjt2g\") pod \"maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp\" (UID: \"42caa626-be8b-4d3b-9bdc-34eea6c345b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp"
Sep 17 15:49:26.534113 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:26.534052    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42caa626-be8b-4d3b-9bdc-34eea6c345b9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp\" (UID: \"42caa626-be8b-4d3b-9bdc-34eea6c345b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp"
Sep 17 15:49:26.536232 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:26.536209    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42caa626-be8b-4d3b-9bdc-34eea6c345b9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp\" (UID: \"42caa626-be8b-4d3b-9bdc-34eea6c345b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp"
Sep 17 15:49:26.771559 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:26.771527    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp"
Sep 17 15:49:27.046844 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:27.046820    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp"]
Sep 17 15:49:28.047985 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:28.047945    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp" event={"ID":"42caa626-be8b-4d3b-9bdc-34eea6c345b9","Type":"ContainerStarted","Data":"070259a8dd59c6d35b6a3ae285bb56f2ee0fcd1d3cee7af8ee4734dcf485acac"}
Sep 17 15:49:28.048399 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:28.048095    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp"
Sep 17 15:49:28.048399 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:28.048143    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp" event={"ID":"42caa626-be8b-4d3b-9bdc-34eea6c345b9","Type":"ContainerStarted","Data":"f7fa3ca49fe18ddc310e005fba082f0cbe3b1b5a4fd9187328466e51b35b4ef2"}
Sep 17 15:49:28.067441 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:28.067397    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp" podStartSLOduration=2.22524388 podStartE2EDuration="3.067382776s" totalImagesPullingTime="842.138896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:49:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:49:28.065428795 +0000 UTC m=+1190.445446561" watchObservedRunningTime="2026-09-17 15:49:28.067382776 +0000 UTC m=+1190.447400531"
Sep 17 15:49:34.058177 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:34.058146    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp"
Sep 17 15:49:38.142378 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:38.142346    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-45zt7_2c069c9e-0856-46b0-ba10-67829c05ac94/console-operator/1.log"
Sep 17 15:49:38.142841 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:49:38.142626    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-45zt7_2c069c9e-0856-46b0-ba10-67829c05ac94/console-operator/1.log"
Sep 17 15:50:08.254373 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:08.254333    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84"]
Sep 17 15:50:08.259142 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:08.259107    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84"
Sep 17 15:50:08.261488 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:08.261467    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-1d63c0-serving-cert\""
Sep 17 15:50:08.272179 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:08.272157    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84"]
Sep 17 15:50:08.334501 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:08.334475    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85ecfe7-6d52-4877-898b-22614e00eb8a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84\" (UID: \"b85ecfe7-6d52-4877-898b-22614e00eb8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84"
Sep 17 15:50:08.334620 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:08.334516    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b85ecfe7-6d52-4877-898b-22614e00eb8a-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84\" (UID: \"b85ecfe7-6d52-4877-898b-22614e00eb8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84"
Sep 17 15:50:08.334620 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:08.334553    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfdg5\" (UniqueName: \"kubernetes.io/projected/b85ecfe7-6d52-4877-898b-22614e00eb8a-kube-api-access-tfdg5\") pod \"maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84\" (UID: \"b85ecfe7-6d52-4877-898b-22614e00eb8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84"
Sep 17 15:50:08.334620 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:08.334576    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b85ecfe7-6d52-4877-898b-22614e00eb8a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84\" (UID: \"b85ecfe7-6d52-4877-898b-22614e00eb8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84"
Sep 17 15:50:08.435442 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:08.435404    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85ecfe7-6d52-4877-898b-22614e00eb8a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84\" (UID: \"b85ecfe7-6d52-4877-898b-22614e00eb8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84"
Sep 17 15:50:08.435581 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:08.435460    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b85ecfe7-6d52-4877-898b-22614e00eb8a-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84\" (UID: \"b85ecfe7-6d52-4877-898b-22614e00eb8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84"
Sep 17 15:50:08.435581 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:08.435501    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfdg5\" (UniqueName: \"kubernetes.io/projected/b85ecfe7-6d52-4877-898b-22614e00eb8a-kube-api-access-tfdg5\") pod \"maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84\" (UID: \"b85ecfe7-6d52-4877-898b-22614e00eb8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84"
Sep 17 15:50:08.435581 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:08.435520    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b85ecfe7-6d52-4877-898b-22614e00eb8a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84\" (UID: \"b85ecfe7-6d52-4877-898b-22614e00eb8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84"
Sep 17 15:50:08.435735 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:50:08.435711    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-serving-cert: secret "maas-api-e2e-shared-a-w3-1d63c0-serving-cert" not found
Sep 17 15:50:08.435811 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:50:08.435798    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b85ecfe7-6d52-4877-898b-22614e00eb8a-maas-api-tls podName:b85ecfe7-6d52-4877-898b-22614e00eb8a nodeName:}" failed. No retries permitted until 2026-09-17 15:50:08.935776683 +0000 UTC m=+1231.315794434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b85ecfe7-6d52-4877-898b-22614e00eb8a-maas-api-tls") pod "maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84" (UID: "b85ecfe7-6d52-4877-898b-22614e00eb8a") : secret "maas-api-e2e-shared-a-w3-1d63c0-serving-cert" not found
Sep 17 15:50:08.436208 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:08.436189    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85ecfe7-6d52-4877-898b-22614e00eb8a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84\" (UID: \"b85ecfe7-6d52-4877-898b-22614e00eb8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84"
Sep 17 15:50:08.437811 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:08.437790    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b85ecfe7-6d52-4877-898b-22614e00eb8a-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84\" (UID: \"b85ecfe7-6d52-4877-898b-22614e00eb8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84"
Sep 17 15:50:08.443956 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:08.443933    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfdg5\" (UniqueName: \"kubernetes.io/projected/b85ecfe7-6d52-4877-898b-22614e00eb8a-kube-api-access-tfdg5\") pod \"maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84\" (UID: \"b85ecfe7-6d52-4877-898b-22614e00eb8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84"
Sep 17 15:50:08.940288 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:08.940257    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b85ecfe7-6d52-4877-898b-22614e00eb8a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84\" (UID: \"b85ecfe7-6d52-4877-898b-22614e00eb8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84"
Sep 17 15:50:08.942725 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:08.942695    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b85ecfe7-6d52-4877-898b-22614e00eb8a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84\" (UID: \"b85ecfe7-6d52-4877-898b-22614e00eb8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84"
Sep 17 15:50:09.174347 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:09.174314    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84"
Sep 17 15:50:09.471777 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:09.471755    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84"]
Sep 17 15:50:09.473782 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:50:09.473754    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85ecfe7_6d52_4877_898b_22614e00eb8a.slice/crio-20ac4454146f7a965a4d616f5bab5013477ff07a3d5da2da2425d730ed86e361 WatchSource:0}: Error finding container 20ac4454146f7a965a4d616f5bab5013477ff07a3d5da2da2425d730ed86e361: Status 404 returned error can't find the container with id 20ac4454146f7a965a4d616f5bab5013477ff07a3d5da2da2425d730ed86e361
Sep 17 15:50:10.268943 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:10.268908    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84" event={"ID":"b85ecfe7-6d52-4877-898b-22614e00eb8a","Type":"ContainerStarted","Data":"e1cce7be627d879b0aca3111255d5bb131fcb1d7bfe4b2b7d4c9e6d00b3139e6"}
Sep 17 15:50:10.269151 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:10.268967    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84" event={"ID":"b85ecfe7-6d52-4877-898b-22614e00eb8a","Type":"ContainerStarted","Data":"20ac4454146f7a965a4d616f5bab5013477ff07a3d5da2da2425d730ed86e361"}
Sep 17 15:50:10.269151 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:10.268998    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84"
Sep 17 15:50:10.306013 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:10.305949    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84" podStartSLOduration=1.991506437 podStartE2EDuration="2.305928364s" totalImagesPullingTime="314.421927ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:50:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:50:10.302553148 +0000 UTC m=+1232.682570903" watchObservedRunningTime="2026-09-17 15:50:10.305928364 +0000 UTC m=+1232.685946119"
Sep 17 15:50:16.279700 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:16.279674    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84"
Sep 17 15:50:42.873856 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:42.873771    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72"]
Sep 17 15:50:42.877859 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:42.877841    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72"
Sep 17 15:50:42.880790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:42.880771    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-5818c9-serving-cert\""
Sep 17 15:50:42.892475 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:42.892453    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72"]
Sep 17 15:50:42.961625 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:42.961595    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6920f1-5a16-4b44-a613-d067ee36c8bd-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72\" (UID: \"6b6920f1-5a16-4b44-a613-d067ee36c8bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72"
Sep 17 15:50:42.961791 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:42.961654    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6920f1-5a16-4b44-a613-d067ee36c8bd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72\" (UID: \"6b6920f1-5a16-4b44-a613-d067ee36c8bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72"
Sep 17 15:50:42.961791 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:42.961726    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x59h\" (UniqueName: \"kubernetes.io/projected/6b6920f1-5a16-4b44-a613-d067ee36c8bd-kube-api-access-7x59h\") pod \"maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72\" (UID: \"6b6920f1-5a16-4b44-a613-d067ee36c8bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72"
Sep 17 15:50:42.961791 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:42.961778    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b6920f1-5a16-4b44-a613-d067ee36c8bd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72\" (UID: \"6b6920f1-5a16-4b44-a613-d067ee36c8bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72"
Sep 17 15:50:43.062436 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:43.062393    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b6920f1-5a16-4b44-a613-d067ee36c8bd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72\" (UID: \"6b6920f1-5a16-4b44-a613-d067ee36c8bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72"
Sep 17 15:50:43.062436 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:43.062445    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6920f1-5a16-4b44-a613-d067ee36c8bd-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72\" (UID: \"6b6920f1-5a16-4b44-a613-d067ee36c8bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72"
Sep 17 15:50:43.062686 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:43.062473    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6920f1-5a16-4b44-a613-d067ee36c8bd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72\" (UID: \"6b6920f1-5a16-4b44-a613-d067ee36c8bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72"
Sep 17 15:50:43.062686 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:43.062534    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x59h\" (UniqueName: \"kubernetes.io/projected/6b6920f1-5a16-4b44-a613-d067ee36c8bd-kube-api-access-7x59h\") pod \"maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72\" (UID: \"6b6920f1-5a16-4b44-a613-d067ee36c8bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72"
Sep 17 15:50:43.062686 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:50:43.062623    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-serving-cert: secret "maas-api-e2e-shared-b-w3-5818c9-serving-cert" not found
Sep 17 15:50:43.062851 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:50:43.062698    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b6920f1-5a16-4b44-a613-d067ee36c8bd-maas-api-tls podName:6b6920f1-5a16-4b44-a613-d067ee36c8bd nodeName:}" failed. No retries permitted until 2026-09-17 15:50:43.562681102 +0000 UTC m=+1265.942698837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6b6920f1-5a16-4b44-a613-d067ee36c8bd-maas-api-tls") pod "maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72" (UID: "6b6920f1-5a16-4b44-a613-d067ee36c8bd") : secret "maas-api-e2e-shared-b-w3-5818c9-serving-cert" not found
Sep 17 15:50:43.063159 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:43.063138    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b6920f1-5a16-4b44-a613-d067ee36c8bd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72\" (UID: \"6b6920f1-5a16-4b44-a613-d067ee36c8bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72"
Sep 17 15:50:43.064947 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:43.064922    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6920f1-5a16-4b44-a613-d067ee36c8bd-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72\" (UID: \"6b6920f1-5a16-4b44-a613-d067ee36c8bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72"
Sep 17 15:50:43.074348 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:43.074326    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x59h\" (UniqueName: \"kubernetes.io/projected/6b6920f1-5a16-4b44-a613-d067ee36c8bd-kube-api-access-7x59h\") pod \"maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72\" (UID: \"6b6920f1-5a16-4b44-a613-d067ee36c8bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72"
Sep 17 15:50:43.567229 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:43.567196    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6920f1-5a16-4b44-a613-d067ee36c8bd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72\" (UID: \"6b6920f1-5a16-4b44-a613-d067ee36c8bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72"
Sep 17 15:50:43.569436 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:43.569415    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6920f1-5a16-4b44-a613-d067ee36c8bd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72\" (UID: \"6b6920f1-5a16-4b44-a613-d067ee36c8bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72"
Sep 17 15:50:43.788894 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:43.788859    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72"
Sep 17 15:50:44.071568 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:44.071540    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72"]
Sep 17 15:50:44.072896 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:50:44.072873    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6920f1_5a16_4b44_a613_d067ee36c8bd.slice/crio-62c31e66ecf1ec81ccacbb6f6880ea6d2d937881d3eb5cd756a7efd48ea4a2aa WatchSource:0}: Error finding container 62c31e66ecf1ec81ccacbb6f6880ea6d2d937881d3eb5cd756a7efd48ea4a2aa: Status 404 returned error can't find the container with id 62c31e66ecf1ec81ccacbb6f6880ea6d2d937881d3eb5cd756a7efd48ea4a2aa
Sep 17 15:50:44.444593 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:44.444564    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72" event={"ID":"6b6920f1-5a16-4b44-a613-d067ee36c8bd","Type":"ContainerStarted","Data":"62c31e66ecf1ec81ccacbb6f6880ea6d2d937881d3eb5cd756a7efd48ea4a2aa"}
Sep 17 15:50:45.452787 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:45.452749    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72" event={"ID":"6b6920f1-5a16-4b44-a613-d067ee36c8bd","Type":"ContainerStarted","Data":"5e5303458ffcda4f67307b3824f08b070a9ed71a36ba06880279e61579f3516b"}
Sep 17 15:50:45.453200 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:45.452940    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72"
Sep 17 15:50:45.481159 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:45.481112    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72" podStartSLOduration=3.129541917 podStartE2EDuration="3.481097391s" totalImagesPullingTime="351.555474ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:50:45.478547939 +0000 UTC m=+1267.858565696" watchObservedRunningTime="2026-09-17 15:50:45.481097391 +0000 UTC m=+1267.861115142"
Sep 17 15:50:51.463690 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:50:51.463660    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72"
Sep 17 15:51:06.453151 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:06.453121    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"]
Sep 17 15:51:06.456647 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:06.456627    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"
Sep 17 15:51:06.459560 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:06.459533    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c86e9a\"/\"distinct-2-w4-c86e9a-kserve-self-signed-certs\""
Sep 17 15:51:06.474445 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:06.474421    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"]
Sep 17 15:51:06.578712 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:06.578686    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8d7e63a-20e0-4b78-aade-5009e35214fe-tls-certs\") pod \"distinct-2-w4-c86e9a-kserve-595f64764d-br8kc\" (UID: \"a8d7e63a-20e0-4b78-aade-5009e35214fe\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"
Sep 17 15:51:06.578863 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:06.578720    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8d7e63a-20e0-4b78-aade-5009e35214fe-tmp-dir\") pod \"distinct-2-w4-c86e9a-kserve-595f64764d-br8kc\" (UID: \"a8d7e63a-20e0-4b78-aade-5009e35214fe\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"
Sep 17 15:51:06.578863 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:06.578741    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8d7e63a-20e0-4b78-aade-5009e35214fe-dshm\") pod \"distinct-2-w4-c86e9a-kserve-595f64764d-br8kc\" (UID: \"a8d7e63a-20e0-4b78-aade-5009e35214fe\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"
Sep 17 15:51:06.578863 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:06.578846    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8d7e63a-20e0-4b78-aade-5009e35214fe-home\") pod \"distinct-2-w4-c86e9a-kserve-595f64764d-br8kc\" (UID: \"a8d7e63a-20e0-4b78-aade-5009e35214fe\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"
Sep 17 15:51:06.579028 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:06.578901    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8d7e63a-20e0-4b78-aade-5009e35214fe-model-cache\") pod \"distinct-2-w4-c86e9a-kserve-595f64764d-br8kc\" (UID: \"a8d7e63a-20e0-4b78-aade-5009e35214fe\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"
Sep 17 15:51:06.579028 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:06.578962    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsjlm\" (UniqueName: \"kubernetes.io/projected/a8d7e63a-20e0-4b78-aade-5009e35214fe-kube-api-access-wsjlm\") pod \"distinct-2-w4-c86e9a-kserve-595f64764d-br8kc\" (UID: \"a8d7e63a-20e0-4b78-aade-5009e35214fe\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"
Sep 17 15:51:06.680293 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:06.680256    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsjlm\" (UniqueName: \"kubernetes.io/projected/a8d7e63a-20e0-4b78-aade-5009e35214fe-kube-api-access-wsjlm\") pod \"distinct-2-w4-c86e9a-kserve-595f64764d-br8kc\" (UID: \"a8d7e63a-20e0-4b78-aade-5009e35214fe\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"
Sep 17 15:51:06.680481 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:06.680323    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8d7e63a-20e0-4b78-aade-5009e35214fe-tls-certs\") pod \"distinct-2-w4-c86e9a-kserve-595f64764d-br8kc\" (UID: \"a8d7e63a-20e0-4b78-aade-5009e35214fe\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"
Sep 17 15:51:06.680481 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:06.680349    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8d7e63a-20e0-4b78-aade-5009e35214fe-tmp-dir\") pod \"distinct-2-w4-c86e9a-kserve-595f64764d-br8kc\" (UID: \"a8d7e63a-20e0-4b78-aade-5009e35214fe\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"
Sep 17 15:51:06.680481 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:06.680364    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8d7e63a-20e0-4b78-aade-5009e35214fe-dshm\") pod \"distinct-2-w4-c86e9a-kserve-595f64764d-br8kc\" (UID: \"a8d7e63a-20e0-4b78-aade-5009e35214fe\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"
Sep 17 15:51:06.680481 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:06.680410    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8d7e63a-20e0-4b78-aade-5009e35214fe-home\") pod \"distinct-2-w4-c86e9a-kserve-595f64764d-br8kc\" (UID: \"a8d7e63a-20e0-4b78-aade-5009e35214fe\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"
Sep 17 15:51:06.680481 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:06.680454    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8d7e63a-20e0-4b78-aade-5009e35214fe-model-cache\") pod \"distinct-2-w4-c86e9a-kserve-595f64764d-br8kc\" (UID: \"a8d7e63a-20e0-4b78-aade-5009e35214fe\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"
Sep 17 15:51:06.680780 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:06.680759    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8d7e63a-20e0-4b78-aade-5009e35214fe-tmp-dir\") pod \"distinct-2-w4-c86e9a-kserve-595f64764d-br8kc\" (UID: \"a8d7e63a-20e0-4b78-aade-5009e35214fe\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"
Sep 17 15:51:06.680843 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:06.680791    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8d7e63a-20e0-4b78-aade-5009e35214fe-model-cache\") pod \"distinct-2-w4-c86e9a-kserve-595f64764d-br8kc\" (UID: \"a8d7e63a-20e0-4b78-aade-5009e35214fe\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"
Sep 17 15:51:06.680893 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:06.680876    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8d7e63a-20e0-4b78-aade-5009e35214fe-home\") pod \"distinct-2-w4-c86e9a-kserve-595f64764d-br8kc\" (UID: \"a8d7e63a-20e0-4b78-aade-5009e35214fe\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"
Sep 17 15:51:06.682651 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:06.682629    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8d7e63a-20e0-4b78-aade-5009e35214fe-dshm\") pod \"distinct-2-w4-c86e9a-kserve-595f64764d-br8kc\" (UID: \"a8d7e63a-20e0-4b78-aade-5009e35214fe\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"
Sep 17 15:51:06.682853 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:06.682835    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8d7e63a-20e0-4b78-aade-5009e35214fe-tls-certs\") pod \"distinct-2-w4-c86e9a-kserve-595f64764d-br8kc\" (UID: \"a8d7e63a-20e0-4b78-aade-5009e35214fe\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"
Sep 17 15:51:06.690480 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:06.690446    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsjlm\" (UniqueName: \"kubernetes.io/projected/a8d7e63a-20e0-4b78-aade-5009e35214fe-kube-api-access-wsjlm\") pod \"distinct-2-w4-c86e9a-kserve-595f64764d-br8kc\" (UID: \"a8d7e63a-20e0-4b78-aade-5009e35214fe\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"
Sep 17 15:51:06.766478 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:06.766414    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"
Sep 17 15:51:07.044390 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:07.044248    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"]
Sep 17 15:51:07.047108 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:51:07.047074    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8d7e63a_20e0_4b78_aade_5009e35214fe.slice/crio-01e4dbfce8c97ce1442d89a269b0ca97205cf9b0eec16c49d7599c976206f792 WatchSource:0}: Error finding container 01e4dbfce8c97ce1442d89a269b0ca97205cf9b0eec16c49d7599c976206f792: Status 404 returned error can't find the container with id 01e4dbfce8c97ce1442d89a269b0ca97205cf9b0eec16c49d7599c976206f792
Sep 17 15:51:07.568823 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:07.568786    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc" event={"ID":"a8d7e63a-20e0-4b78-aade-5009e35214fe","Type":"ContainerStarted","Data":"4947e3d0bc4aa3ea8a05f722104c7cc56421c0e17048c7c38c4caa9baf99bfa5"}
Sep 17 15:51:07.569236 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:07.568831    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc" event={"ID":"a8d7e63a-20e0-4b78-aade-5009e35214fe","Type":"ContainerStarted","Data":"01e4dbfce8c97ce1442d89a269b0ca97205cf9b0eec16c49d7599c976206f792"}
Sep 17 15:51:07.607529 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:07.607475    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc" podStartSLOduration=1.6074579610000002 podStartE2EDuration="1.607457961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:51:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:51:07.605622208 +0000 UTC m=+1289.985639987" watchObservedRunningTime="2026-09-17 15:51:07.607457961 +0000 UTC m=+1289.987475716"
Sep 17 15:51:16.767389 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:16.767347    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"
Sep 17 15:51:16.767389 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:16.767396    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"
Sep 17 15:51:16.781213 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:16.781193    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"
Sep 17 15:51:17.636418 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:51:17.636388    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"
Sep 17 15:54:24.825855 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:24.825814    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"]
Sep 17 15:54:24.836584 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:24.836554    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"]
Sep 17 15:54:38.176312 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:38.176282    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-45zt7_2c069c9e-0856-46b0-ba10-67829c05ac94/console-operator/1.log"
Sep 17 15:54:38.176991 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:38.176971    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-45zt7_2c069c9e-0856-46b0-ba10-67829c05ac94/console-operator/1.log"
Sep 17 15:54:38.518939 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:38.518860    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z"]
Sep 17 15:54:38.519114 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:38.519094    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z" podUID="c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c" containerName="payload-processing" containerID="cri-o://4073959c7243eb8eab6b826b280e760c3039126b50683ae055a5fa3ee10b771c" gracePeriod=30
Sep 17 15:54:38.530504 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:38.530481    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc"]
Sep 17 15:54:38.530696 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:38.530675    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc" podUID="f98d8a52-f711-4229-ac7a-59ddc0ad68a4" containerName="payload-pre-processing" containerID="cri-o://4c4e2c3d2e8af6ed156be54a58e34c24ac3577378fa3a98bd7331af27604c260" gracePeriod=30
Sep 17 15:54:38.718522 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:38.718490    2556 generic.go:356] "Generic (PLEG): container finished" podID="c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c" containerID="4073959c7243eb8eab6b826b280e760c3039126b50683ae055a5fa3ee10b771c" exitCode=0
Sep 17 15:54:38.718699 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:38.718588    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z" event={"ID":"c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c","Type":"ContainerDied","Data":"4073959c7243eb8eab6b826b280e760c3039126b50683ae055a5fa3ee10b771c"}
Sep 17 15:54:38.720540 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:38.720423    2556 generic.go:356] "Generic (PLEG): container finished" podID="f98d8a52-f711-4229-ac7a-59ddc0ad68a4" containerID="4c4e2c3d2e8af6ed156be54a58e34c24ac3577378fa3a98bd7331af27604c260" exitCode=0
Sep 17 15:54:38.720540 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:38.720462    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc" event={"ID":"f98d8a52-f711-4229-ac7a-59ddc0ad68a4","Type":"ContainerDied","Data":"4c4e2c3d2e8af6ed156be54a58e34c24ac3577378fa3a98bd7331af27604c260"}
Sep 17 15:54:38.787357 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:38.787329    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z"
Sep 17 15:54:38.823155 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:38.823125    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc"
Sep 17 15:54:38.896968 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:38.896697    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c-plugins-config-volume\") pod \"c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c\" (UID: \"c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c\") "
Sep 17 15:54:38.896968 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:38.896786    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gfb6\" (UniqueName: \"kubernetes.io/projected/c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c-kube-api-access-8gfb6\") pod \"c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c\" (UID: \"c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c\") "
Sep 17 15:54:38.896968 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:38.896851    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98d8a52-f711-4229-ac7a-59ddc0ad68a4-plugins-config-volume\") pod \"f98d8a52-f711-4229-ac7a-59ddc0ad68a4\" (UID: \"f98d8a52-f711-4229-ac7a-59ddc0ad68a4\") "
Sep 17 15:54:38.896968 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:38.896962    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fdkw\" (UniqueName: \"kubernetes.io/projected/f98d8a52-f711-4229-ac7a-59ddc0ad68a4-kube-api-access-9fdkw\") pod \"f98d8a52-f711-4229-ac7a-59ddc0ad68a4\" (UID: \"f98d8a52-f711-4229-ac7a-59ddc0ad68a4\") "
Sep 17 15:54:38.897357 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:38.897132    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c" (UID: "c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:54:38.897357 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:38.897330    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f98d8a52-f711-4229-ac7a-59ddc0ad68a4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f98d8a52-f711-4229-ac7a-59ddc0ad68a4" (UID: "f98d8a52-f711-4229-ac7a-59ddc0ad68a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:54:38.897357 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:38.897341    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c-plugins-config-volume\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:54:38.899309 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:38.899276    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98d8a52-f711-4229-ac7a-59ddc0ad68a4-kube-api-access-9fdkw" (OuterVolumeSpecName: "kube-api-access-9fdkw") pod "f98d8a52-f711-4229-ac7a-59ddc0ad68a4" (UID: "f98d8a52-f711-4229-ac7a-59ddc0ad68a4"). InnerVolumeSpecName "kube-api-access-9fdkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:54:38.899471 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:38.899436    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c-kube-api-access-8gfb6" (OuterVolumeSpecName: "kube-api-access-8gfb6") pod "c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c" (UID: "c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c"). InnerVolumeSpecName "kube-api-access-8gfb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:54:38.998310 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:38.998279    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fdkw\" (UniqueName: \"kubernetes.io/projected/f98d8a52-f711-4229-ac7a-59ddc0ad68a4-kube-api-access-9fdkw\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:54:38.998310 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:38.998306    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gfb6\" (UniqueName: \"kubernetes.io/projected/c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c-kube-api-access-8gfb6\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:54:38.998310 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:38.998316    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98d8a52-f711-4229-ac7a-59ddc0ad68a4-plugins-config-volume\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:54:39.189892 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:39.189857    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"]
Sep 17 15:54:39.726843 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:39.726811    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z"
Sep 17 15:54:39.726843 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:39.726833    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z" event={"ID":"c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c","Type":"ContainerDied","Data":"c24812414b1e094ffe6012056316f291bf8b3d7ecf9528ed5dd1aaf735c94609"}
Sep 17 15:54:39.727133 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:39.726902    2556 scope.go:117] "RemoveContainer" containerID="4073959c7243eb8eab6b826b280e760c3039126b50683ae055a5fa3ee10b771c"
Sep 17 15:54:39.728207 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:39.728193    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc"
Sep 17 15:54:39.728274 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:39.728222    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc" event={"ID":"f98d8a52-f711-4229-ac7a-59ddc0ad68a4","Type":"ContainerDied","Data":"e2279b6f6fd894539ad70262bb7d605fc3507c7127ce52ec7dfc785882d39bde"}
Sep 17 15:54:39.735933 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:39.735916    2556 scope.go:117] "RemoveContainer" containerID="4c4e2c3d2e8af6ed156be54a58e34c24ac3577378fa3a98bd7331af27604c260"
Sep 17 15:54:39.774623 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:39.774590    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z"]
Sep 17 15:54:39.788988 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:39.788906    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c86e9a-d5b9884b7-g4k7z"]
Sep 17 15:54:39.804669 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:39.804639    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc"]
Sep 17 15:54:39.813455 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:39.813433    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c86e9a-bbb5599b6-fkwzc"]
Sep 17 15:54:39.845516 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:39.845481    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc" podUID="a8d7e63a-20e0-4b78-aade-5009e35214fe" containerName="main" containerID="cri-o://4947e3d0bc4aa3ea8a05f722104c7cc56421c0e17048c7c38c4caa9baf99bfa5" gracePeriod=45
Sep 17 15:54:39.854909 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:39.854878    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n" podUID="1341d441-3541-4f1a-8e1a-9a8fe9f94398" containerName="main" containerID="cri-o://afcc73c029cfb69166aa73f25be0b7ea8d641e98568ddcbeca2c53d2b089506b" gracePeriod=45
Sep 17 15:54:40.103139 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.103108    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"
Sep 17 15:54:40.124328 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.124300    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"
Sep 17 15:54:40.135620 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.135594    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c" path="/var/lib/kubelet/pods/c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c/volumes"
Sep 17 15:54:40.135954 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.135942    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f98d8a52-f711-4229-ac7a-59ddc0ad68a4" path="/var/lib/kubelet/pods/f98d8a52-f711-4229-ac7a-59ddc0ad68a4/volumes"
Sep 17 15:54:40.209429 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.209398    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8d7e63a-20e0-4b78-aade-5009e35214fe-home\") pod \"a8d7e63a-20e0-4b78-aade-5009e35214fe\" (UID: \"a8d7e63a-20e0-4b78-aade-5009e35214fe\") "
Sep 17 15:54:40.209429 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.209428    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8d7e63a-20e0-4b78-aade-5009e35214fe-model-cache\") pod \"a8d7e63a-20e0-4b78-aade-5009e35214fe\" (UID: \"a8d7e63a-20e0-4b78-aade-5009e35214fe\") "
Sep 17 15:54:40.209928 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.209454    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1341d441-3541-4f1a-8e1a-9a8fe9f94398-dshm\") pod \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\" (UID: \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\") "
Sep 17 15:54:40.209928 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.209475    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnd6c\" (UniqueName: \"kubernetes.io/projected/1341d441-3541-4f1a-8e1a-9a8fe9f94398-kube-api-access-mnd6c\") pod \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\" (UID: \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\") "
Sep 17 15:54:40.209928 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.209556    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8d7e63a-20e0-4b78-aade-5009e35214fe-tls-certs\") pod \"a8d7e63a-20e0-4b78-aade-5009e35214fe\" (UID: \"a8d7e63a-20e0-4b78-aade-5009e35214fe\") "
Sep 17 15:54:40.209928 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.209588    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1341d441-3541-4f1a-8e1a-9a8fe9f94398-model-cache\") pod \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\" (UID: \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\") "
Sep 17 15:54:40.209928 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.209612    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8d7e63a-20e0-4b78-aade-5009e35214fe-dshm\") pod \"a8d7e63a-20e0-4b78-aade-5009e35214fe\" (UID: \"a8d7e63a-20e0-4b78-aade-5009e35214fe\") "
Sep 17 15:54:40.209928 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.209644    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1341d441-3541-4f1a-8e1a-9a8fe9f94398-home\") pod \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\" (UID: \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\") "
Sep 17 15:54:40.209928 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.209690    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8d7e63a-20e0-4b78-aade-5009e35214fe-tmp-dir\") pod \"a8d7e63a-20e0-4b78-aade-5009e35214fe\" (UID: \"a8d7e63a-20e0-4b78-aade-5009e35214fe\") "
Sep 17 15:54:40.209928 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.209728    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d7e63a-20e0-4b78-aade-5009e35214fe-model-cache" (OuterVolumeSpecName: "model-cache") pod "a8d7e63a-20e0-4b78-aade-5009e35214fe" (UID: "a8d7e63a-20e0-4b78-aade-5009e35214fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:40.209928 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.209743    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1341d441-3541-4f1a-8e1a-9a8fe9f94398-tmp-dir\") pod \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\" (UID: \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\") "
Sep 17 15:54:40.209928 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.209780    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsjlm\" (UniqueName: \"kubernetes.io/projected/a8d7e63a-20e0-4b78-aade-5009e35214fe-kube-api-access-wsjlm\") pod \"a8d7e63a-20e0-4b78-aade-5009e35214fe\" (UID: \"a8d7e63a-20e0-4b78-aade-5009e35214fe\") "
Sep 17 15:54:40.209928 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.209824    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1341d441-3541-4f1a-8e1a-9a8fe9f94398-tls-certs\") pod \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\" (UID: \"1341d441-3541-4f1a-8e1a-9a8fe9f94398\") "
Sep 17 15:54:40.210593 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.209993    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d7e63a-20e0-4b78-aade-5009e35214fe-home" (OuterVolumeSpecName: "home") pod "a8d7e63a-20e0-4b78-aade-5009e35214fe" (UID: "a8d7e63a-20e0-4b78-aade-5009e35214fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:40.210593 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.210199    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8d7e63a-20e0-4b78-aade-5009e35214fe-home\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:54:40.210593 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.210219    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8d7e63a-20e0-4b78-aade-5009e35214fe-model-cache\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:54:40.210593 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.210241    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1341d441-3541-4f1a-8e1a-9a8fe9f94398-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1341d441-3541-4f1a-8e1a-9a8fe9f94398" (UID: "1341d441-3541-4f1a-8e1a-9a8fe9f94398"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:40.210593 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.210433    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1341d441-3541-4f1a-8e1a-9a8fe9f94398-home" (OuterVolumeSpecName: "home") pod "1341d441-3541-4f1a-8e1a-9a8fe9f94398" (UID: "1341d441-3541-4f1a-8e1a-9a8fe9f94398"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:40.210860 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.210606    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d7e63a-20e0-4b78-aade-5009e35214fe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a8d7e63a-20e0-4b78-aade-5009e35214fe" (UID: "a8d7e63a-20e0-4b78-aade-5009e35214fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:40.210860 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.210743    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1341d441-3541-4f1a-8e1a-9a8fe9f94398-model-cache" (OuterVolumeSpecName: "model-cache") pod "1341d441-3541-4f1a-8e1a-9a8fe9f94398" (UID: "1341d441-3541-4f1a-8e1a-9a8fe9f94398"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:40.212379 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.212349    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8d7e63a-20e0-4b78-aade-5009e35214fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a8d7e63a-20e0-4b78-aade-5009e35214fe" (UID: "a8d7e63a-20e0-4b78-aade-5009e35214fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:54:40.212486 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.212420    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1341d441-3541-4f1a-8e1a-9a8fe9f94398-kube-api-access-mnd6c" (OuterVolumeSpecName: "kube-api-access-mnd6c") pod "1341d441-3541-4f1a-8e1a-9a8fe9f94398" (UID: "1341d441-3541-4f1a-8e1a-9a8fe9f94398"). InnerVolumeSpecName "kube-api-access-mnd6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:54:40.212681 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.212648    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d7e63a-20e0-4b78-aade-5009e35214fe-dshm" (OuterVolumeSpecName: "dshm") pod "a8d7e63a-20e0-4b78-aade-5009e35214fe" (UID: "a8d7e63a-20e0-4b78-aade-5009e35214fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:40.212887 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.212868    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1341d441-3541-4f1a-8e1a-9a8fe9f94398-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1341d441-3541-4f1a-8e1a-9a8fe9f94398" (UID: "1341d441-3541-4f1a-8e1a-9a8fe9f94398"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:54:40.213039 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.213011    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d7e63a-20e0-4b78-aade-5009e35214fe-kube-api-access-wsjlm" (OuterVolumeSpecName: "kube-api-access-wsjlm") pod "a8d7e63a-20e0-4b78-aade-5009e35214fe" (UID: "a8d7e63a-20e0-4b78-aade-5009e35214fe"). InnerVolumeSpecName "kube-api-access-wsjlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:54:40.213140 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.213011    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1341d441-3541-4f1a-8e1a-9a8fe9f94398-dshm" (OuterVolumeSpecName: "dshm") pod "1341d441-3541-4f1a-8e1a-9a8fe9f94398" (UID: "1341d441-3541-4f1a-8e1a-9a8fe9f94398"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:40.311201 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.311122    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8d7e63a-20e0-4b78-aade-5009e35214fe-tmp-dir\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:54:40.311201 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.311149    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1341d441-3541-4f1a-8e1a-9a8fe9f94398-tmp-dir\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:54:40.311201 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.311158    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsjlm\" (UniqueName: \"kubernetes.io/projected/a8d7e63a-20e0-4b78-aade-5009e35214fe-kube-api-access-wsjlm\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:54:40.311201 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.311166    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1341d441-3541-4f1a-8e1a-9a8fe9f94398-tls-certs\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:54:40.311201 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.311174    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1341d441-3541-4f1a-8e1a-9a8fe9f94398-dshm\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:54:40.311201 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.311182    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnd6c\" (UniqueName: \"kubernetes.io/projected/1341d441-3541-4f1a-8e1a-9a8fe9f94398-kube-api-access-mnd6c\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:54:40.311201 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.311192    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8d7e63a-20e0-4b78-aade-5009e35214fe-tls-certs\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:54:40.311201 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.311200    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1341d441-3541-4f1a-8e1a-9a8fe9f94398-model-cache\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:54:40.311572 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.311210    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8d7e63a-20e0-4b78-aade-5009e35214fe-dshm\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:54:40.311572 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.311217    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1341d441-3541-4f1a-8e1a-9a8fe9f94398-home\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:54:40.737211 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.737174    2556 generic.go:356] "Generic (PLEG): container finished" podID="1341d441-3541-4f1a-8e1a-9a8fe9f94398" containerID="afcc73c029cfb69166aa73f25be0b7ea8d641e98568ddcbeca2c53d2b089506b" exitCode=0
Sep 17 15:54:40.737383 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.737229    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"
Sep 17 15:54:40.737383 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.737259    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n" event={"ID":"1341d441-3541-4f1a-8e1a-9a8fe9f94398","Type":"ContainerDied","Data":"afcc73c029cfb69166aa73f25be0b7ea8d641e98568ddcbeca2c53d2b089506b"}
Sep 17 15:54:40.737383 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.737302    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n" event={"ID":"1341d441-3541-4f1a-8e1a-9a8fe9f94398","Type":"ContainerDied","Data":"8349346827bcfd65948ad83d314d94c884af99e89e8133ddafdf30583e43a261"}
Sep 17 15:54:40.737383 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.737324    2556 scope.go:117] "RemoveContainer" containerID="afcc73c029cfb69166aa73f25be0b7ea8d641e98568ddcbeca2c53d2b089506b"
Sep 17 15:54:40.738738 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.738718    2556 generic.go:356] "Generic (PLEG): container finished" podID="a8d7e63a-20e0-4b78-aade-5009e35214fe" containerID="4947e3d0bc4aa3ea8a05f722104c7cc56421c0e17048c7c38c4caa9baf99bfa5" exitCode=0
Sep 17 15:54:40.738817 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.738763    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc" event={"ID":"a8d7e63a-20e0-4b78-aade-5009e35214fe","Type":"ContainerDied","Data":"4947e3d0bc4aa3ea8a05f722104c7cc56421c0e17048c7c38c4caa9baf99bfa5"}
Sep 17 15:54:40.738817 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.738771    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"
Sep 17 15:54:40.738817 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.738786    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc" event={"ID":"a8d7e63a-20e0-4b78-aade-5009e35214fe","Type":"ContainerDied","Data":"01e4dbfce8c97ce1442d89a269b0ca97205cf9b0eec16c49d7599c976206f792"}
Sep 17 15:54:40.746597 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.746566    2556 scope.go:117] "RemoveContainer" containerID="afcc73c029cfb69166aa73f25be0b7ea8d641e98568ddcbeca2c53d2b089506b"
Sep 17 15:54:40.746952 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:54:40.746931    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afcc73c029cfb69166aa73f25be0b7ea8d641e98568ddcbeca2c53d2b089506b\": container with ID starting with afcc73c029cfb69166aa73f25be0b7ea8d641e98568ddcbeca2c53d2b089506b not found: ID does not exist" containerID="afcc73c029cfb69166aa73f25be0b7ea8d641e98568ddcbeca2c53d2b089506b"
Sep 17 15:54:40.747047 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.746962    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afcc73c029cfb69166aa73f25be0b7ea8d641e98568ddcbeca2c53d2b089506b"} err="failed to get container status \"afcc73c029cfb69166aa73f25be0b7ea8d641e98568ddcbeca2c53d2b089506b\": rpc error: code = NotFound desc = could not find container \"afcc73c029cfb69166aa73f25be0b7ea8d641e98568ddcbeca2c53d2b089506b\": container with ID starting with afcc73c029cfb69166aa73f25be0b7ea8d641e98568ddcbeca2c53d2b089506b not found: ID does not exist"
Sep 17 15:54:40.747047 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.746982    2556 scope.go:117] "RemoveContainer" containerID="4947e3d0bc4aa3ea8a05f722104c7cc56421c0e17048c7c38c4caa9baf99bfa5"
Sep 17 15:54:40.755882 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.755863    2556 scope.go:117] "RemoveContainer" containerID="4947e3d0bc4aa3ea8a05f722104c7cc56421c0e17048c7c38c4caa9baf99bfa5"
Sep 17 15:54:40.756179 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:54:40.756160    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4947e3d0bc4aa3ea8a05f722104c7cc56421c0e17048c7c38c4caa9baf99bfa5\": container with ID starting with 4947e3d0bc4aa3ea8a05f722104c7cc56421c0e17048c7c38c4caa9baf99bfa5 not found: ID does not exist" containerID="4947e3d0bc4aa3ea8a05f722104c7cc56421c0e17048c7c38c4caa9baf99bfa5"
Sep 17 15:54:40.756238 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.756187    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4947e3d0bc4aa3ea8a05f722104c7cc56421c0e17048c7c38c4caa9baf99bfa5"} err="failed to get container status \"4947e3d0bc4aa3ea8a05f722104c7cc56421c0e17048c7c38c4caa9baf99bfa5\": rpc error: code = NotFound desc = could not find container \"4947e3d0bc4aa3ea8a05f722104c7cc56421c0e17048c7c38c4caa9baf99bfa5\": container with ID starting with 4947e3d0bc4aa3ea8a05f722104c7cc56421c0e17048c7c38c4caa9baf99bfa5 not found: ID does not exist"
Sep 17 15:54:40.764908 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.764881    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"]
Sep 17 15:54:40.769264 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.769240    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/distinct-2-w4-c86e9a-kserve-595f64764d-br8kc"]
Sep 17 15:54:40.783179 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.783151    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"]
Sep 17 15:54:40.785164 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:40.785144    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/premium-w4-c86e9a-kserve-856c688968-4777n"]
Sep 17 15:54:41.868134 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:41.868105    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m"]
Sep 17 15:54:41.868579 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:41.868565    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8d7e63a-20e0-4b78-aade-5009e35214fe" containerName="main"
Sep 17 15:54:41.868623 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:41.868580    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8d7e63a-20e0-4b78-aade-5009e35214fe" containerName="main"
Sep 17 15:54:41.868623 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:41.868591    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c" containerName="payload-processing"
Sep 17 15:54:41.868623 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:41.868596    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c" containerName="payload-processing"
Sep 17 15:54:41.868623 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:41.868612    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1341d441-3541-4f1a-8e1a-9a8fe9f94398" containerName="main"
Sep 17 15:54:41.868623 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:41.868617    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1341d441-3541-4f1a-8e1a-9a8fe9f94398" containerName="main"
Sep 17 15:54:41.868774 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:41.868629    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f98d8a52-f711-4229-ac7a-59ddc0ad68a4" containerName="payload-pre-processing"
Sep 17 15:54:41.868774 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:41.868634    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f98d8a52-f711-4229-ac7a-59ddc0ad68a4" containerName="payload-pre-processing"
Sep 17 15:54:41.868774 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:41.868693    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1341d441-3541-4f1a-8e1a-9a8fe9f94398" containerName="main"
Sep 17 15:54:41.868774 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:41.868701    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3924af3-fbb7-4781-88c7-e6a2d5cb3a9c" containerName="payload-processing"
Sep 17 15:54:41.868774 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:41.868708    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8d7e63a-20e0-4b78-aade-5009e35214fe" containerName="main"
Sep 17 15:54:41.868774 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:41.868716    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f98d8a52-f711-4229-ac7a-59ddc0ad68a4" containerName="payload-pre-processing"
Sep 17 15:54:41.873264 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:41.873247    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m"
Sep 17 15:54:41.875486 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:41.875465    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c686d0-serving-cert\""
Sep 17 15:54:41.885103 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:41.885082    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m"]
Sep 17 15:54:42.029324 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:42.029288    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e50e2d-0cc1-4b82-ac28-f28586d5ce18-openshift-service-ca\") pod \"maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m\" (UID: \"16e50e2d-0cc1-4b82-ac28-f28586d5ce18\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m"
Sep 17 15:54:42.029501 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:42.029422    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e50e2d-0cc1-4b82-ac28-f28586d5ce18-maas-api-tls\") pod \"maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m\" (UID: \"16e50e2d-0cc1-4b82-ac28-f28586d5ce18\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m"
Sep 17 15:54:42.029568 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:42.029499    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k458s\" (UniqueName: \"kubernetes.io/projected/16e50e2d-0cc1-4b82-ac28-f28586d5ce18-kube-api-access-k458s\") pod \"maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m\" (UID: \"16e50e2d-0cc1-4b82-ac28-f28586d5ce18\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m"
Sep 17 15:54:42.029568 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:42.029544    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e50e2d-0cc1-4b82-ac28-f28586d5ce18-metrics-certs\") pod \"maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m\" (UID: \"16e50e2d-0cc1-4b82-ac28-f28586d5ce18\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m"
Sep 17 15:54:42.130710 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:42.130619    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e50e2d-0cc1-4b82-ac28-f28586d5ce18-maas-api-tls\") pod \"maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m\" (UID: \"16e50e2d-0cc1-4b82-ac28-f28586d5ce18\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m"
Sep 17 15:54:42.130710 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:42.130664    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k458s\" (UniqueName: \"kubernetes.io/projected/16e50e2d-0cc1-4b82-ac28-f28586d5ce18-kube-api-access-k458s\") pod \"maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m\" (UID: \"16e50e2d-0cc1-4b82-ac28-f28586d5ce18\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m"
Sep 17 15:54:42.130710 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:42.130684    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e50e2d-0cc1-4b82-ac28-f28586d5ce18-metrics-certs\") pod \"maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m\" (UID: \"16e50e2d-0cc1-4b82-ac28-f28586d5ce18\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m"
Sep 17 15:54:42.131005 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:42.130743    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e50e2d-0cc1-4b82-ac28-f28586d5ce18-openshift-service-ca\") pod \"maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m\" (UID: \"16e50e2d-0cc1-4b82-ac28-f28586d5ce18\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m"
Sep 17 15:54:42.131005 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:54:42.130863    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-serving-cert: secret "maas-api-e2e-api-a-c686d0-serving-cert" not found
Sep 17 15:54:42.131005 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:54:42.130954    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16e50e2d-0cc1-4b82-ac28-f28586d5ce18-maas-api-tls podName:16e50e2d-0cc1-4b82-ac28-f28586d5ce18 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:42.630931427 +0000 UTC m=+1505.010949164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/16e50e2d-0cc1-4b82-ac28-f28586d5ce18-maas-api-tls") pod "maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m" (UID: "16e50e2d-0cc1-4b82-ac28-f28586d5ce18") : secret "maas-api-e2e-api-a-c686d0-serving-cert" not found
Sep 17 15:54:42.131401 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:42.131377    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e50e2d-0cc1-4b82-ac28-f28586d5ce18-openshift-service-ca\") pod \"maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m\" (UID: \"16e50e2d-0cc1-4b82-ac28-f28586d5ce18\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m"
Sep 17 15:54:42.133659 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:42.133628    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e50e2d-0cc1-4b82-ac28-f28586d5ce18-metrics-certs\") pod \"maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m\" (UID: \"16e50e2d-0cc1-4b82-ac28-f28586d5ce18\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m"
Sep 17 15:54:42.138755 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:42.138728    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1341d441-3541-4f1a-8e1a-9a8fe9f94398" path="/var/lib/kubelet/pods/1341d441-3541-4f1a-8e1a-9a8fe9f94398/volumes"
Sep 17 15:54:42.139304 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:42.139288    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8d7e63a-20e0-4b78-aade-5009e35214fe" path="/var/lib/kubelet/pods/a8d7e63a-20e0-4b78-aade-5009e35214fe/volumes"
Sep 17 15:54:42.142220 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:42.142194    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k458s\" (UniqueName: \"kubernetes.io/projected/16e50e2d-0cc1-4b82-ac28-f28586d5ce18-kube-api-access-k458s\") pod \"maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m\" (UID: \"16e50e2d-0cc1-4b82-ac28-f28586d5ce18\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m"
Sep 17 15:54:42.636722 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:42.636679    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e50e2d-0cc1-4b82-ac28-f28586d5ce18-maas-api-tls\") pod \"maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m\" (UID: \"16e50e2d-0cc1-4b82-ac28-f28586d5ce18\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m"
Sep 17 15:54:42.639183 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:42.639158    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e50e2d-0cc1-4b82-ac28-f28586d5ce18-maas-api-tls\") pod \"maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m\" (UID: \"16e50e2d-0cc1-4b82-ac28-f28586d5ce18\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m"
Sep 17 15:54:42.785471 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:42.785434    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m"
Sep 17 15:54:43.062250 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:43.062165    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m"]
Sep 17 15:54:43.066906 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:54:43.066880    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e50e2d_0cc1_4b82_ac28_f28586d5ce18.slice/crio-a2b86e98e5e791c2263e20213f392c5e2ca4f71ea91352cb5a483df3256d8022 WatchSource:0}: Error finding container a2b86e98e5e791c2263e20213f392c5e2ca4f71ea91352cb5a483df3256d8022: Status 404 returned error can't find the container with id a2b86e98e5e791c2263e20213f392c5e2ca4f71ea91352cb5a483df3256d8022
Sep 17 15:54:43.068124 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:43.068107    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:54:43.764607 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:43.764569    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m" event={"ID":"16e50e2d-0cc1-4b82-ac28-f28586d5ce18","Type":"ContainerStarted","Data":"d870e08ce6c247b73d4fb47fa323818839c78c3d23741103aef59dd991c0f585"}
Sep 17 15:54:43.764799 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:43.764626    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m" event={"ID":"16e50e2d-0cc1-4b82-ac28-f28586d5ce18","Type":"ContainerStarted","Data":"a2b86e98e5e791c2263e20213f392c5e2ca4f71ea91352cb5a483df3256d8022"}
Sep 17 15:54:43.764799 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:43.764736    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m"
Sep 17 15:54:43.803235 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:43.803118    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m" podStartSLOduration=2.327607704 podStartE2EDuration="2.803100442s" totalImagesPullingTime="475.492738ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:54:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:43.80119184 +0000 UTC m=+1506.181209594" watchObservedRunningTime="2026-09-17 15:54:43.803100442 +0000 UTC m=+1506.183118196"
Sep 17 15:54:49.775229 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:49.775196    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m"
Sep 17 15:54:52.650846 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:52.650811    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9"]
Sep 17 15:54:52.651252 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:52.651038    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9" podUID="a573b2a2-1835-471b-a875-195912e1d592" containerName="maas-api" containerID="cri-o://735259c2bc3dd4c8f8908c0a655d687e6b10df9f347082b96833d846328890cf" gracePeriod=30
Sep 17 15:54:52.718567 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:52.718527    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6"]
Sep 17 15:54:52.719075 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:52.719019    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6" podUID="55be1385-28d7-4c8f-9d17-7b87c852a726" containerName="payload-processing" containerID="cri-o://7ab5ecd293918e3ce1dd45b59579f3a509c3fc8ea484500e5b14bf617fc7be52" gracePeriod=30
Sep 17 15:54:52.816105 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:52.816074    2556 generic.go:356] "Generic (PLEG): container finished" podID="a573b2a2-1835-471b-a875-195912e1d592" containerID="735259c2bc3dd4c8f8908c0a655d687e6b10df9f347082b96833d846328890cf" exitCode=0
Sep 17 15:54:52.816275 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:52.816126    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9" event={"ID":"a573b2a2-1835-471b-a875-195912e1d592","Type":"ContainerDied","Data":"735259c2bc3dd4c8f8908c0a655d687e6b10df9f347082b96833d846328890cf"}
Sep 17 15:54:52.972427 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:52.972400    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9"
Sep 17 15:54:53.005038 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.005003    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6"
Sep 17 15:54:53.049957 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.049926    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vxrl\" (UniqueName: \"kubernetes.io/projected/a573b2a2-1835-471b-a875-195912e1d592-kube-api-access-2vxrl\") pod \"a573b2a2-1835-471b-a875-195912e1d592\" (UID: \"a573b2a2-1835-471b-a875-195912e1d592\") "
Sep 17 15:54:53.049957 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.049964    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55be1385-28d7-4c8f-9d17-7b87c852a726-plugins-config-volume\") pod \"55be1385-28d7-4c8f-9d17-7b87c852a726\" (UID: \"55be1385-28d7-4c8f-9d17-7b87c852a726\") "
Sep 17 15:54:53.050229 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.050000    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwzcg\" (UniqueName: \"kubernetes.io/projected/55be1385-28d7-4c8f-9d17-7b87c852a726-kube-api-access-fwzcg\") pod \"55be1385-28d7-4c8f-9d17-7b87c852a726\" (UID: \"55be1385-28d7-4c8f-9d17-7b87c852a726\") "
Sep 17 15:54:53.050229 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.050028    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a573b2a2-1835-471b-a875-195912e1d592-openshift-service-ca\") pod \"a573b2a2-1835-471b-a875-195912e1d592\" (UID: \"a573b2a2-1835-471b-a875-195912e1d592\") "
Sep 17 15:54:53.050229 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.050122    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a573b2a2-1835-471b-a875-195912e1d592-metrics-certs\") pod \"a573b2a2-1835-471b-a875-195912e1d592\" (UID: \"a573b2a2-1835-471b-a875-195912e1d592\") "
Sep 17 15:54:53.050229 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.050171    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a573b2a2-1835-471b-a875-195912e1d592-maas-api-tls\") pod \"a573b2a2-1835-471b-a875-195912e1d592\" (UID: \"a573b2a2-1835-471b-a875-195912e1d592\") "
Sep 17 15:54:53.051423 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.051374    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a573b2a2-1835-471b-a875-195912e1d592-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a573b2a2-1835-471b-a875-195912e1d592" (UID: "a573b2a2-1835-471b-a875-195912e1d592"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:54:53.051769 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.051734    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55be1385-28d7-4c8f-9d17-7b87c852a726-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "55be1385-28d7-4c8f-9d17-7b87c852a726" (UID: "55be1385-28d7-4c8f-9d17-7b87c852a726"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:54:53.052990 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.052961    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a573b2a2-1835-471b-a875-195912e1d592-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a573b2a2-1835-471b-a875-195912e1d592" (UID: "a573b2a2-1835-471b-a875-195912e1d592"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:54:53.053151 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.053113    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a573b2a2-1835-471b-a875-195912e1d592-kube-api-access-2vxrl" (OuterVolumeSpecName: "kube-api-access-2vxrl") pod "a573b2a2-1835-471b-a875-195912e1d592" (UID: "a573b2a2-1835-471b-a875-195912e1d592"). InnerVolumeSpecName "kube-api-access-2vxrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:54:53.053385 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.053363    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55be1385-28d7-4c8f-9d17-7b87c852a726-kube-api-access-fwzcg" (OuterVolumeSpecName: "kube-api-access-fwzcg") pod "55be1385-28d7-4c8f-9d17-7b87c852a726" (UID: "55be1385-28d7-4c8f-9d17-7b87c852a726"). InnerVolumeSpecName "kube-api-access-fwzcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:54:53.053814 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.053789    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a573b2a2-1835-471b-a875-195912e1d592-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a573b2a2-1835-471b-a875-195912e1d592" (UID: "a573b2a2-1835-471b-a875-195912e1d592"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:54:53.151628 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.151581    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwzcg\" (UniqueName: \"kubernetes.io/projected/55be1385-28d7-4c8f-9d17-7b87c852a726-kube-api-access-fwzcg\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:54:53.151628 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.151620    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a573b2a2-1835-471b-a875-195912e1d592-openshift-service-ca\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:54:53.151628 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.151631    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a573b2a2-1835-471b-a875-195912e1d592-metrics-certs\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:54:53.151628 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.151640    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a573b2a2-1835-471b-a875-195912e1d592-maas-api-tls\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:54:53.151906 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.151648    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vxrl\" (UniqueName: \"kubernetes.io/projected/a573b2a2-1835-471b-a875-195912e1d592-kube-api-access-2vxrl\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:54:53.151906 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.151657    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55be1385-28d7-4c8f-9d17-7b87c852a726-plugins-config-volume\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:54:53.822711 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.822681    2556 generic.go:356] "Generic (PLEG): container finished" podID="55be1385-28d7-4c8f-9d17-7b87c852a726" containerID="7ab5ecd293918e3ce1dd45b59579f3a509c3fc8ea484500e5b14bf617fc7be52" exitCode=0
Sep 17 15:54:53.823206 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.822728    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6"
Sep 17 15:54:53.823206 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.822773    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6" event={"ID":"55be1385-28d7-4c8f-9d17-7b87c852a726","Type":"ContainerDied","Data":"7ab5ecd293918e3ce1dd45b59579f3a509c3fc8ea484500e5b14bf617fc7be52"}
Sep 17 15:54:53.823206 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.822834    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6" event={"ID":"55be1385-28d7-4c8f-9d17-7b87c852a726","Type":"ContainerDied","Data":"9a17349f2c9b829435d7e72cc6197ce5ebb4dfee6297fb89303e918a058381fa"}
Sep 17 15:54:53.823206 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.822868    2556 scope.go:117] "RemoveContainer" containerID="7ab5ecd293918e3ce1dd45b59579f3a509c3fc8ea484500e5b14bf617fc7be52"
Sep 17 15:54:53.824261 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.824226    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9" event={"ID":"a573b2a2-1835-471b-a875-195912e1d592","Type":"ContainerDied","Data":"093d7cc484560a3716e1f0068d7fc6fd490772c2387b6f6f98805d67da207fa8"}
Sep 17 15:54:53.824334 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.824265    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9"
Sep 17 15:54:53.832635 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.832617    2556 scope.go:117] "RemoveContainer" containerID="7ab5ecd293918e3ce1dd45b59579f3a509c3fc8ea484500e5b14bf617fc7be52"
Sep 17 15:54:53.832971 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:54:53.832947    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ab5ecd293918e3ce1dd45b59579f3a509c3fc8ea484500e5b14bf617fc7be52\": container with ID starting with 7ab5ecd293918e3ce1dd45b59579f3a509c3fc8ea484500e5b14bf617fc7be52 not found: ID does not exist" containerID="7ab5ecd293918e3ce1dd45b59579f3a509c3fc8ea484500e5b14bf617fc7be52"
Sep 17 15:54:53.833127 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.832990    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ab5ecd293918e3ce1dd45b59579f3a509c3fc8ea484500e5b14bf617fc7be52"} err="failed to get container status \"7ab5ecd293918e3ce1dd45b59579f3a509c3fc8ea484500e5b14bf617fc7be52\": rpc error: code = NotFound desc = could not find container \"7ab5ecd293918e3ce1dd45b59579f3a509c3fc8ea484500e5b14bf617fc7be52\": container with ID starting with 7ab5ecd293918e3ce1dd45b59579f3a509c3fc8ea484500e5b14bf617fc7be52 not found: ID does not exist"
Sep 17 15:54:53.833127 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.833013    2556 scope.go:117] "RemoveContainer" containerID="735259c2bc3dd4c8f8908c0a655d687e6b10df9f347082b96833d846328890cf"
Sep 17 15:54:53.851727 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.851703    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6"]
Sep 17 15:54:53.855191 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.855170    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-849b71-6848bc846f-lr6r6"]
Sep 17 15:54:53.868804 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.868784    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9"]
Sep 17 15:54:53.873143 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:53.873123    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-849b71-84ccb79477-6pxk9"]
Sep 17 15:54:54.136214 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.136185    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55be1385-28d7-4c8f-9d17-7b87c852a726" path="/var/lib/kubelet/pods/55be1385-28d7-4c8f-9d17-7b87c852a726/volumes"
Sep 17 15:54:54.136630 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.136616    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a573b2a2-1835-471b-a875-195912e1d592" path="/var/lib/kubelet/pods/a573b2a2-1835-471b-a875-195912e1d592/volumes"
Sep 17 15:54:54.208608 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.208570    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68" podUID="6e2e1a5a-48d7-4164-adae-4d10011730fe" containerName="main" containerID="cri-o://efec482158f292cb20d951c4b791fd398a6c83c159fd54263fc19bc4763ef1ea" gracePeriod=45
Sep 17 15:54:54.458668 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.458644    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"
Sep 17 15:54:54.565137 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.565107    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e2e1a5a-48d7-4164-adae-4d10011730fe-model-cache\") pod \"6e2e1a5a-48d7-4164-adae-4d10011730fe\" (UID: \"6e2e1a5a-48d7-4164-adae-4d10011730fe\") "
Sep 17 15:54:54.565324 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.565160    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e2e1a5a-48d7-4164-adae-4d10011730fe-dshm\") pod \"6e2e1a5a-48d7-4164-adae-4d10011730fe\" (UID: \"6e2e1a5a-48d7-4164-adae-4d10011730fe\") "
Sep 17 15:54:54.565324 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.565220    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tt4r\" (UniqueName: \"kubernetes.io/projected/6e2e1a5a-48d7-4164-adae-4d10011730fe-kube-api-access-2tt4r\") pod \"6e2e1a5a-48d7-4164-adae-4d10011730fe\" (UID: \"6e2e1a5a-48d7-4164-adae-4d10011730fe\") "
Sep 17 15:54:54.565324 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.565285    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e2e1a5a-48d7-4164-adae-4d10011730fe-tls-certs\") pod \"6e2e1a5a-48d7-4164-adae-4d10011730fe\" (UID: \"6e2e1a5a-48d7-4164-adae-4d10011730fe\") "
Sep 17 15:54:54.565485 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.565360    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e2e1a5a-48d7-4164-adae-4d10011730fe-tmp-dir\") pod \"6e2e1a5a-48d7-4164-adae-4d10011730fe\" (UID: \"6e2e1a5a-48d7-4164-adae-4d10011730fe\") "
Sep 17 15:54:54.565485 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.565385    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e2e1a5a-48d7-4164-adae-4d10011730fe-home\") pod \"6e2e1a5a-48d7-4164-adae-4d10011730fe\" (UID: \"6e2e1a5a-48d7-4164-adae-4d10011730fe\") "
Sep 17 15:54:54.565853 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.565820    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e2e1a5a-48d7-4164-adae-4d10011730fe-model-cache" (OuterVolumeSpecName: "model-cache") pod "6e2e1a5a-48d7-4164-adae-4d10011730fe" (UID: "6e2e1a5a-48d7-4164-adae-4d10011730fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:54.565966 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.565845    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e2e1a5a-48d7-4164-adae-4d10011730fe-home" (OuterVolumeSpecName: "home") pod "6e2e1a5a-48d7-4164-adae-4d10011730fe" (UID: "6e2e1a5a-48d7-4164-adae-4d10011730fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:54.566030 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.566018    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e2e1a5a-48d7-4164-adae-4d10011730fe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6e2e1a5a-48d7-4164-adae-4d10011730fe" (UID: "6e2e1a5a-48d7-4164-adae-4d10011730fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:54.568344 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.568318    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e2e1a5a-48d7-4164-adae-4d10011730fe-dshm" (OuterVolumeSpecName: "dshm") pod "6e2e1a5a-48d7-4164-adae-4d10011730fe" (UID: "6e2e1a5a-48d7-4164-adae-4d10011730fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:54.568575 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.568548    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e2e1a5a-48d7-4164-adae-4d10011730fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6e2e1a5a-48d7-4164-adae-4d10011730fe" (UID: "6e2e1a5a-48d7-4164-adae-4d10011730fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:54:54.568683 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.568556    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e2e1a5a-48d7-4164-adae-4d10011730fe-kube-api-access-2tt4r" (OuterVolumeSpecName: "kube-api-access-2tt4r") pod "6e2e1a5a-48d7-4164-adae-4d10011730fe" (UID: "6e2e1a5a-48d7-4164-adae-4d10011730fe"). InnerVolumeSpecName "kube-api-access-2tt4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:54:54.667051 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.667020    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e2e1a5a-48d7-4164-adae-4d10011730fe-tls-certs\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:54:54.667051 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.667045    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e2e1a5a-48d7-4164-adae-4d10011730fe-tmp-dir\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:54:54.667051 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.667054    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e2e1a5a-48d7-4164-adae-4d10011730fe-home\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:54:54.667307 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.667093    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e2e1a5a-48d7-4164-adae-4d10011730fe-model-cache\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:54:54.667307 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.667102    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e2e1a5a-48d7-4164-adae-4d10011730fe-dshm\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:54:54.667307 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.667111    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tt4r\" (UniqueName: \"kubernetes.io/projected/6e2e1a5a-48d7-4164-adae-4d10011730fe-kube-api-access-2tt4r\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:54:54.832985 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.832899    2556 generic.go:356] "Generic (PLEG): container finished" podID="6e2e1a5a-48d7-4164-adae-4d10011730fe" containerID="efec482158f292cb20d951c4b791fd398a6c83c159fd54263fc19bc4763ef1ea" exitCode=0
Sep 17 15:54:54.832985 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.832936    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68" event={"ID":"6e2e1a5a-48d7-4164-adae-4d10011730fe","Type":"ContainerDied","Data":"efec482158f292cb20d951c4b791fd398a6c83c159fd54263fc19bc4763ef1ea"}
Sep 17 15:54:54.832985 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.832956    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"
Sep 17 15:54:54.832985 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.832970    2556 scope.go:117] "RemoveContainer" containerID="efec482158f292cb20d951c4b791fd398a6c83c159fd54263fc19bc4763ef1ea"
Sep 17 15:54:54.833645 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.832960    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68" event={"ID":"6e2e1a5a-48d7-4164-adae-4d10011730fe","Type":"ContainerDied","Data":"6310d026a7779fa95c932b27fe7a3c7333ecec200cb88e04c4dcfd1d89ccbe1b"}
Sep 17 15:54:54.841964 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.841944    2556 scope.go:117] "RemoveContainer" containerID="efec482158f292cb20d951c4b791fd398a6c83c159fd54263fc19bc4763ef1ea"
Sep 17 15:54:54.842247 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:54:54.842229    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efec482158f292cb20d951c4b791fd398a6c83c159fd54263fc19bc4763ef1ea\": container with ID starting with efec482158f292cb20d951c4b791fd398a6c83c159fd54263fc19bc4763ef1ea not found: ID does not exist" containerID="efec482158f292cb20d951c4b791fd398a6c83c159fd54263fc19bc4763ef1ea"
Sep 17 15:54:54.842318 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.842257    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efec482158f292cb20d951c4b791fd398a6c83c159fd54263fc19bc4763ef1ea"} err="failed to get container status \"efec482158f292cb20d951c4b791fd398a6c83c159fd54263fc19bc4763ef1ea\": rpc error: code = NotFound desc = could not find container \"efec482158f292cb20d951c4b791fd398a6c83c159fd54263fc19bc4763ef1ea\": container with ID starting with efec482158f292cb20d951c4b791fd398a6c83c159fd54263fc19bc4763ef1ea not found: ID does not exist"
Sep 17 15:54:54.863272 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.863241    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"]
Sep 17 15:54:54.867645 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:54.867622    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-849b71/premium-w0-849b71-kserve-6bb4fcbcbf-8fl68"]
Sep 17 15:54:56.136160 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:54:56.136130    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e2e1a5a-48d7-4164-adae-4d10011730fe" path="/var/lib/kubelet/pods/6e2e1a5a-48d7-4164-adae-4d10011730fe/volumes"
Sep 17 15:55:06.776613 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:06.776497    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj"]
Sep 17 15:55:06.777133 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:06.777115    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e2e1a5a-48d7-4164-adae-4d10011730fe" containerName="main"
Sep 17 15:55:06.777133 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:06.777134    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e2e1a5a-48d7-4164-adae-4d10011730fe" containerName="main"
Sep 17 15:55:06.777249 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:06.777151    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a573b2a2-1835-471b-a875-195912e1d592" containerName="maas-api"
Sep 17 15:55:06.777249 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:06.777157    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a573b2a2-1835-471b-a875-195912e1d592" containerName="maas-api"
Sep 17 15:55:06.777249 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:06.777164    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55be1385-28d7-4c8f-9d17-7b87c852a726" containerName="payload-processing"
Sep 17 15:55:06.777249 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:06.777170    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="55be1385-28d7-4c8f-9d17-7b87c852a726" containerName="payload-processing"
Sep 17 15:55:06.777378 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:06.777256    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e2e1a5a-48d7-4164-adae-4d10011730fe" containerName="main"
Sep 17 15:55:06.777378 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:06.777267    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="55be1385-28d7-4c8f-9d17-7b87c852a726" containerName="payload-processing"
Sep 17 15:55:06.777378 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:06.777274    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a573b2a2-1835-471b-a875-195912e1d592" containerName="maas-api"
Sep 17 15:55:06.781853 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:06.781829    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj"
Sep 17 15:55:06.784385 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:06.784364    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-8f99a1-serving-cert\""
Sep 17 15:55:06.793206 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:06.793153    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj"]
Sep 17 15:55:06.883308 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:06.883268    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9tb2\" (UniqueName: \"kubernetes.io/projected/c2ec914d-76d3-4adf-ba43-8290317b6bd9-kube-api-access-k9tb2\") pod \"maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj\" (UID: \"c2ec914d-76d3-4adf-ba43-8290317b6bd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj"
Sep 17 15:55:06.883502 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:06.883317    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2ec914d-76d3-4adf-ba43-8290317b6bd9-openshift-service-ca\") pod \"maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj\" (UID: \"c2ec914d-76d3-4adf-ba43-8290317b6bd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj"
Sep 17 15:55:06.883502 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:06.883379    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2ec914d-76d3-4adf-ba43-8290317b6bd9-metrics-certs\") pod \"maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj\" (UID: \"c2ec914d-76d3-4adf-ba43-8290317b6bd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj"
Sep 17 15:55:06.883502 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:06.883409    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2ec914d-76d3-4adf-ba43-8290317b6bd9-maas-api-tls\") pod \"maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj\" (UID: \"c2ec914d-76d3-4adf-ba43-8290317b6bd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj"
Sep 17 15:55:06.984582 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:06.984545    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9tb2\" (UniqueName: \"kubernetes.io/projected/c2ec914d-76d3-4adf-ba43-8290317b6bd9-kube-api-access-k9tb2\") pod \"maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj\" (UID: \"c2ec914d-76d3-4adf-ba43-8290317b6bd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj"
Sep 17 15:55:06.984853 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:06.984588    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2ec914d-76d3-4adf-ba43-8290317b6bd9-openshift-service-ca\") pod \"maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj\" (UID: \"c2ec914d-76d3-4adf-ba43-8290317b6bd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj"
Sep 17 15:55:06.984853 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:06.984635    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2ec914d-76d3-4adf-ba43-8290317b6bd9-metrics-certs\") pod \"maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj\" (UID: \"c2ec914d-76d3-4adf-ba43-8290317b6bd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj"
Sep 17 15:55:06.984853 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:06.984665    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2ec914d-76d3-4adf-ba43-8290317b6bd9-maas-api-tls\") pod \"maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj\" (UID: \"c2ec914d-76d3-4adf-ba43-8290317b6bd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj"
Sep 17 15:55:06.985400 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:06.985379    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2ec914d-76d3-4adf-ba43-8290317b6bd9-openshift-service-ca\") pod \"maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj\" (UID: \"c2ec914d-76d3-4adf-ba43-8290317b6bd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj"
Sep 17 15:55:06.987251 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:06.987221    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2ec914d-76d3-4adf-ba43-8290317b6bd9-metrics-certs\") pod \"maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj\" (UID: \"c2ec914d-76d3-4adf-ba43-8290317b6bd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj"
Sep 17 15:55:06.987412 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:06.987388    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2ec914d-76d3-4adf-ba43-8290317b6bd9-maas-api-tls\") pod \"maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj\" (UID: \"c2ec914d-76d3-4adf-ba43-8290317b6bd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj"
Sep 17 15:55:06.998238 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:06.998204    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9tb2\" (UniqueName: \"kubernetes.io/projected/c2ec914d-76d3-4adf-ba43-8290317b6bd9-kube-api-access-k9tb2\") pod \"maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj\" (UID: \"c2ec914d-76d3-4adf-ba43-8290317b6bd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj"
Sep 17 15:55:07.094089 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:07.093975    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj"
Sep 17 15:55:07.371184 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:07.371158    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj"]
Sep 17 15:55:07.373386 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:55:07.373363    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ec914d_76d3_4adf_ba43_8290317b6bd9.slice/crio-53f8b834e6d2e23993a9b7f9cff0dd7663d0be7be663de6ac91400db0b771be0 WatchSource:0}: Error finding container 53f8b834e6d2e23993a9b7f9cff0dd7663d0be7be663de6ac91400db0b771be0: Status 404 returned error can't find the container with id 53f8b834e6d2e23993a9b7f9cff0dd7663d0be7be663de6ac91400db0b771be0
Sep 17 15:55:07.903379 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:07.903341    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj" event={"ID":"c2ec914d-76d3-4adf-ba43-8290317b6bd9","Type":"ContainerStarted","Data":"1a6c454902aa068226185a0201c664d4cf43ff796690236a8241088dc0a5c094"}
Sep 17 15:55:07.903809 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:07.903541    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj"
Sep 17 15:55:07.903809 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:07.903581    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj" event={"ID":"c2ec914d-76d3-4adf-ba43-8290317b6bd9","Type":"ContainerStarted","Data":"53f8b834e6d2e23993a9b7f9cff0dd7663d0be7be663de6ac91400db0b771be0"}
Sep 17 15:55:07.925737 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:07.925693    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj" podStartSLOduration=1.6151132879999999 podStartE2EDuration="1.92567777s" totalImagesPullingTime="310.564482ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:55:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:07.924396063 +0000 UTC m=+1530.304413818" watchObservedRunningTime="2026-09-17 15:55:07.92567777 +0000 UTC m=+1530.305695519"
Sep 17 15:55:13.914468 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:13.914434    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj"
Sep 17 15:55:50.365465 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:50.365430    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m"]
Sep 17 15:55:50.366005 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:50.365624    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m" podUID="16e50e2d-0cc1-4b82-ac28-f28586d5ce18" containerName="maas-api" containerID="cri-o://d870e08ce6c247b73d4fb47fa323818839c78c3d23741103aef59dd991c0f585" gracePeriod=30
Sep 17 15:55:50.622849 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:50.622792    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m"
Sep 17 15:55:50.693672 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:50.693641    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k458s\" (UniqueName: \"kubernetes.io/projected/16e50e2d-0cc1-4b82-ac28-f28586d5ce18-kube-api-access-k458s\") pod \"16e50e2d-0cc1-4b82-ac28-f28586d5ce18\" (UID: \"16e50e2d-0cc1-4b82-ac28-f28586d5ce18\") "
Sep 17 15:55:50.693807 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:50.693722    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e50e2d-0cc1-4b82-ac28-f28586d5ce18-maas-api-tls\") pod \"16e50e2d-0cc1-4b82-ac28-f28586d5ce18\" (UID: \"16e50e2d-0cc1-4b82-ac28-f28586d5ce18\") "
Sep 17 15:55:50.693807 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:50.693770    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e50e2d-0cc1-4b82-ac28-f28586d5ce18-metrics-certs\") pod \"16e50e2d-0cc1-4b82-ac28-f28586d5ce18\" (UID: \"16e50e2d-0cc1-4b82-ac28-f28586d5ce18\") "
Sep 17 15:55:50.693807 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:50.693789    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e50e2d-0cc1-4b82-ac28-f28586d5ce18-openshift-service-ca\") pod \"16e50e2d-0cc1-4b82-ac28-f28586d5ce18\" (UID: \"16e50e2d-0cc1-4b82-ac28-f28586d5ce18\") "
Sep 17 15:55:50.694244 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:50.694218    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e50e2d-0cc1-4b82-ac28-f28586d5ce18-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "16e50e2d-0cc1-4b82-ac28-f28586d5ce18" (UID: "16e50e2d-0cc1-4b82-ac28-f28586d5ce18"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:55:50.695773 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:50.695749    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e50e2d-0cc1-4b82-ac28-f28586d5ce18-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "16e50e2d-0cc1-4b82-ac28-f28586d5ce18" (UID: "16e50e2d-0cc1-4b82-ac28-f28586d5ce18"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:55:50.695849 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:50.695774    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e50e2d-0cc1-4b82-ac28-f28586d5ce18-kube-api-access-k458s" (OuterVolumeSpecName: "kube-api-access-k458s") pod "16e50e2d-0cc1-4b82-ac28-f28586d5ce18" (UID: "16e50e2d-0cc1-4b82-ac28-f28586d5ce18"). InnerVolumeSpecName "kube-api-access-k458s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:55:50.695849 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:50.695830    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e50e2d-0cc1-4b82-ac28-f28586d5ce18-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "16e50e2d-0cc1-4b82-ac28-f28586d5ce18" (UID: "16e50e2d-0cc1-4b82-ac28-f28586d5ce18"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:55:50.794793 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:50.794762    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16e50e2d-0cc1-4b82-ac28-f28586d5ce18-maas-api-tls\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:55:50.794793 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:50.794795    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16e50e2d-0cc1-4b82-ac28-f28586d5ce18-metrics-certs\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:55:50.795260 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:50.794810    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e50e2d-0cc1-4b82-ac28-f28586d5ce18-openshift-service-ca\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:55:50.795260 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:50.794825    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k458s\" (UniqueName: \"kubernetes.io/projected/16e50e2d-0cc1-4b82-ac28-f28586d5ce18-kube-api-access-k458s\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:55:51.127586 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:51.127553    2556 generic.go:356] "Generic (PLEG): container finished" podID="16e50e2d-0cc1-4b82-ac28-f28586d5ce18" containerID="d870e08ce6c247b73d4fb47fa323818839c78c3d23741103aef59dd991c0f585" exitCode=0
Sep 17 15:55:51.127749 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:51.127605    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m"
Sep 17 15:55:51.127749 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:51.127638    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m" event={"ID":"16e50e2d-0cc1-4b82-ac28-f28586d5ce18","Type":"ContainerDied","Data":"d870e08ce6c247b73d4fb47fa323818839c78c3d23741103aef59dd991c0f585"}
Sep 17 15:55:51.127749 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:51.127715    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m" event={"ID":"16e50e2d-0cc1-4b82-ac28-f28586d5ce18","Type":"ContainerDied","Data":"a2b86e98e5e791c2263e20213f392c5e2ca4f71ea91352cb5a483df3256d8022"}
Sep 17 15:55:51.127749 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:51.127737    2556 scope.go:117] "RemoveContainer" containerID="d870e08ce6c247b73d4fb47fa323818839c78c3d23741103aef59dd991c0f585"
Sep 17 15:55:51.137104 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:51.137087    2556 scope.go:117] "RemoveContainer" containerID="d870e08ce6c247b73d4fb47fa323818839c78c3d23741103aef59dd991c0f585"
Sep 17 15:55:51.137355 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:55:51.137336    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d870e08ce6c247b73d4fb47fa323818839c78c3d23741103aef59dd991c0f585\": container with ID starting with d870e08ce6c247b73d4fb47fa323818839c78c3d23741103aef59dd991c0f585 not found: ID does not exist" containerID="d870e08ce6c247b73d4fb47fa323818839c78c3d23741103aef59dd991c0f585"
Sep 17 15:55:51.137403 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:51.137364    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d870e08ce6c247b73d4fb47fa323818839c78c3d23741103aef59dd991c0f585"} err="failed to get container status \"d870e08ce6c247b73d4fb47fa323818839c78c3d23741103aef59dd991c0f585\": rpc error: code = NotFound desc = could not find container \"d870e08ce6c247b73d4fb47fa323818839c78c3d23741103aef59dd991c0f585\": container with ID starting with d870e08ce6c247b73d4fb47fa323818839c78c3d23741103aef59dd991c0f585 not found: ID does not exist"
Sep 17 15:55:51.152597 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:51.152572    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m"]
Sep 17 15:55:51.156579 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:51.156560    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c686d0-758dc88d49-rpq9m"]
Sep 17 15:55:52.136413 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:55:52.136378    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16e50e2d-0cc1-4b82-ac28-f28586d5ce18" path="/var/lib/kubelet/pods/16e50e2d-0cc1-4b82-ac28-f28586d5ce18/volumes"
Sep 17 15:56:14.646759 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:14.646717    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj"]
Sep 17 15:56:14.647411 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:14.646979    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj" podUID="c2ec914d-76d3-4adf-ba43-8290317b6bd9" containerName="maas-api" containerID="cri-o://1a6c454902aa068226185a0201c664d4cf43ff796690236a8241088dc0a5c094" gracePeriod=30
Sep 17 15:56:14.908016 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:14.907954    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj"
Sep 17 15:56:15.037250 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:15.037215    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2ec914d-76d3-4adf-ba43-8290317b6bd9-openshift-service-ca\") pod \"c2ec914d-76d3-4adf-ba43-8290317b6bd9\" (UID: \"c2ec914d-76d3-4adf-ba43-8290317b6bd9\") "
Sep 17 15:56:15.037250 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:15.037257    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2ec914d-76d3-4adf-ba43-8290317b6bd9-metrics-certs\") pod \"c2ec914d-76d3-4adf-ba43-8290317b6bd9\" (UID: \"c2ec914d-76d3-4adf-ba43-8290317b6bd9\") "
Sep 17 15:56:15.037476 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:15.037327    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2ec914d-76d3-4adf-ba43-8290317b6bd9-maas-api-tls\") pod \"c2ec914d-76d3-4adf-ba43-8290317b6bd9\" (UID: \"c2ec914d-76d3-4adf-ba43-8290317b6bd9\") "
Sep 17 15:56:15.037476 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:15.037351    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9tb2\" (UniqueName: \"kubernetes.io/projected/c2ec914d-76d3-4adf-ba43-8290317b6bd9-kube-api-access-k9tb2\") pod \"c2ec914d-76d3-4adf-ba43-8290317b6bd9\" (UID: \"c2ec914d-76d3-4adf-ba43-8290317b6bd9\") "
Sep 17 15:56:15.037621 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:15.037589    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2ec914d-76d3-4adf-ba43-8290317b6bd9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c2ec914d-76d3-4adf-ba43-8290317b6bd9" (UID: "c2ec914d-76d3-4adf-ba43-8290317b6bd9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:56:15.037741 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:15.037723    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2ec914d-76d3-4adf-ba43-8290317b6bd9-openshift-service-ca\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:56:15.039602 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:15.039578    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2ec914d-76d3-4adf-ba43-8290317b6bd9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c2ec914d-76d3-4adf-ba43-8290317b6bd9" (UID: "c2ec914d-76d3-4adf-ba43-8290317b6bd9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:56:15.039602 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:15.039596    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2ec914d-76d3-4adf-ba43-8290317b6bd9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c2ec914d-76d3-4adf-ba43-8290317b6bd9" (UID: "c2ec914d-76d3-4adf-ba43-8290317b6bd9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:56:15.039727 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:15.039583    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ec914d-76d3-4adf-ba43-8290317b6bd9-kube-api-access-k9tb2" (OuterVolumeSpecName: "kube-api-access-k9tb2") pod "c2ec914d-76d3-4adf-ba43-8290317b6bd9" (UID: "c2ec914d-76d3-4adf-ba43-8290317b6bd9"). InnerVolumeSpecName "kube-api-access-k9tb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:56:15.139227 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:15.139190    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2ec914d-76d3-4adf-ba43-8290317b6bd9-metrics-certs\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:56:15.139227 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:15.139218    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2ec914d-76d3-4adf-ba43-8290317b6bd9-maas-api-tls\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:56:15.139227 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:15.139227    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9tb2\" (UniqueName: \"kubernetes.io/projected/c2ec914d-76d3-4adf-ba43-8290317b6bd9-kube-api-access-k9tb2\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:56:15.255658 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:15.255576    2556 generic.go:356] "Generic (PLEG): container finished" podID="c2ec914d-76d3-4adf-ba43-8290317b6bd9" containerID="1a6c454902aa068226185a0201c664d4cf43ff796690236a8241088dc0a5c094" exitCode=0
Sep 17 15:56:15.255658 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:15.255617    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj"
Sep 17 15:56:15.255844 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:15.255662    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj" event={"ID":"c2ec914d-76d3-4adf-ba43-8290317b6bd9","Type":"ContainerDied","Data":"1a6c454902aa068226185a0201c664d4cf43ff796690236a8241088dc0a5c094"}
Sep 17 15:56:15.255844 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:15.255714    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj" event={"ID":"c2ec914d-76d3-4adf-ba43-8290317b6bd9","Type":"ContainerDied","Data":"53f8b834e6d2e23993a9b7f9cff0dd7663d0be7be663de6ac91400db0b771be0"}
Sep 17 15:56:15.255844 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:15.255741    2556 scope.go:117] "RemoveContainer" containerID="1a6c454902aa068226185a0201c664d4cf43ff796690236a8241088dc0a5c094"
Sep 17 15:56:15.265286 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:15.265268    2556 scope.go:117] "RemoveContainer" containerID="1a6c454902aa068226185a0201c664d4cf43ff796690236a8241088dc0a5c094"
Sep 17 15:56:15.265561 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:56:15.265539    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a6c454902aa068226185a0201c664d4cf43ff796690236a8241088dc0a5c094\": container with ID starting with 1a6c454902aa068226185a0201c664d4cf43ff796690236a8241088dc0a5c094 not found: ID does not exist" containerID="1a6c454902aa068226185a0201c664d4cf43ff796690236a8241088dc0a5c094"
Sep 17 15:56:15.265645 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:15.265573    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a6c454902aa068226185a0201c664d4cf43ff796690236a8241088dc0a5c094"} err="failed to get container status \"1a6c454902aa068226185a0201c664d4cf43ff796690236a8241088dc0a5c094\": rpc error: code = NotFound desc = could not find container \"1a6c454902aa068226185a0201c664d4cf43ff796690236a8241088dc0a5c094\": container with ID starting with 1a6c454902aa068226185a0201c664d4cf43ff796690236a8241088dc0a5c094 not found: ID does not exist"
Sep 17 15:56:15.303075 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:15.303028    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj"]
Sep 17 15:56:15.306720 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:15.306695    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8f99a1-7c4c76778c-6bspj"]
Sep 17 15:56:16.136234 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:16.136200    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2ec914d-76d3-4adf-ba43-8290317b6bd9" path="/var/lib/kubelet/pods/c2ec914d-76d3-4adf-ba43-8290317b6bd9/volumes"
Sep 17 15:56:36.109213 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:36.109133    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf"]
Sep 17 15:56:36.109592 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:36.109342    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf" podUID="7227a335-5e51-4adf-a705-dafc223fd4fa" containerName="maas-api" containerID="cri-o://01e4d3ce1137865cd08709d09069e9d0a0251773b99ce47180dccbcca8bee492" gracePeriod=30
Sep 17 15:56:36.364145 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:36.363991    2556 generic.go:356] "Generic (PLEG): container finished" podID="7227a335-5e51-4adf-a705-dafc223fd4fa" containerID="01e4d3ce1137865cd08709d09069e9d0a0251773b99ce47180dccbcca8bee492" exitCode=0
Sep 17 15:56:36.364145 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:36.364073    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf" event={"ID":"7227a335-5e51-4adf-a705-dafc223fd4fa","Type":"ContainerDied","Data":"01e4d3ce1137865cd08709d09069e9d0a0251773b99ce47180dccbcca8bee492"}
Sep 17 15:56:36.364145 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:36.364126    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf" event={"ID":"7227a335-5e51-4adf-a705-dafc223fd4fa","Type":"ContainerDied","Data":"ee44edf8b4b18022844035a0e7041d0b5ccedbb2608dec0a4df1757b4f1a9254"}
Sep 17 15:56:36.364344 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:36.364154    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee44edf8b4b18022844035a0e7041d0b5ccedbb2608dec0a4df1757b4f1a9254"
Sep 17 15:56:36.373707 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:36.373681    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf"
Sep 17 15:56:36.435126 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:36.435091    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7227a335-5e51-4adf-a705-dafc223fd4fa-metrics-certs\") pod \"7227a335-5e51-4adf-a705-dafc223fd4fa\" (UID: \"7227a335-5e51-4adf-a705-dafc223fd4fa\") "
Sep 17 15:56:36.435311 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:36.435147    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6s74\" (UniqueName: \"kubernetes.io/projected/7227a335-5e51-4adf-a705-dafc223fd4fa-kube-api-access-z6s74\") pod \"7227a335-5e51-4adf-a705-dafc223fd4fa\" (UID: \"7227a335-5e51-4adf-a705-dafc223fd4fa\") "
Sep 17 15:56:36.435311 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:36.435265    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7227a335-5e51-4adf-a705-dafc223fd4fa-openshift-service-ca\") pod \"7227a335-5e51-4adf-a705-dafc223fd4fa\" (UID: \"7227a335-5e51-4adf-a705-dafc223fd4fa\") "
Sep 17 15:56:36.435447 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:36.435319    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7227a335-5e51-4adf-a705-dafc223fd4fa-maas-api-tls\") pod \"7227a335-5e51-4adf-a705-dafc223fd4fa\" (UID: \"7227a335-5e51-4adf-a705-dafc223fd4fa\") "
Sep 17 15:56:36.435943 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:36.435916    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7227a335-5e51-4adf-a705-dafc223fd4fa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7227a335-5e51-4adf-a705-dafc223fd4fa" (UID: "7227a335-5e51-4adf-a705-dafc223fd4fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:56:36.437438 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:36.437414    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7227a335-5e51-4adf-a705-dafc223fd4fa-kube-api-access-z6s74" (OuterVolumeSpecName: "kube-api-access-z6s74") pod "7227a335-5e51-4adf-a705-dafc223fd4fa" (UID: "7227a335-5e51-4adf-a705-dafc223fd4fa"). InnerVolumeSpecName "kube-api-access-z6s74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:56:36.437711 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:36.437693    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7227a335-5e51-4adf-a705-dafc223fd4fa-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7227a335-5e51-4adf-a705-dafc223fd4fa" (UID: "7227a335-5e51-4adf-a705-dafc223fd4fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:56:36.437772 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:36.437721    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7227a335-5e51-4adf-a705-dafc223fd4fa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7227a335-5e51-4adf-a705-dafc223fd4fa" (UID: "7227a335-5e51-4adf-a705-dafc223fd4fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:56:36.536231 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:36.536201    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7227a335-5e51-4adf-a705-dafc223fd4fa-openshift-service-ca\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:56:36.536231 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:36.536227    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7227a335-5e51-4adf-a705-dafc223fd4fa-maas-api-tls\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:56:36.536231 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:36.536236    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7227a335-5e51-4adf-a705-dafc223fd4fa-metrics-certs\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:56:36.536456 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:36.536244    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6s74\" (UniqueName: \"kubernetes.io/projected/7227a335-5e51-4adf-a705-dafc223fd4fa-kube-api-access-z6s74\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:56:37.369393 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:37.369352    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf"
Sep 17 15:56:37.398597 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:37.398557    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf"]
Sep 17 15:56:37.407153 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:37.407117    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2b23c7-9bb8fd678-wldtf"]
Sep 17 15:56:38.138704 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:38.138673    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7227a335-5e51-4adf-a705-dafc223fd4fa" path="/var/lib/kubelet/pods/7227a335-5e51-4adf-a705-dafc223fd4fa/volumes"
Sep 17 15:56:38.284941 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:38.284905    2556 scope.go:117] "RemoveContainer" containerID="01e4d3ce1137865cd08709d09069e9d0a0251773b99ce47180dccbcca8bee492"
Sep 17 15:56:55.881396 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:55.876714    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp"]
Sep 17 15:56:55.881396 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:55.877029    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp" podUID="42caa626-be8b-4d3b-9bdc-34eea6c345b9" containerName="maas-api" containerID="cri-o://070259a8dd59c6d35b6a3ae285bb56f2ee0fcd1d3cee7af8ee4734dcf485acac" gracePeriod=30
Sep 17 15:56:56.127729 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:56.127702    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp"
Sep 17 15:56:56.228087 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:56.227970    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42caa626-be8b-4d3b-9bdc-34eea6c345b9-maas-api-tls\") pod \"42caa626-be8b-4d3b-9bdc-34eea6c345b9\" (UID: \"42caa626-be8b-4d3b-9bdc-34eea6c345b9\") "
Sep 17 15:56:56.228087 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:56.228087    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42caa626-be8b-4d3b-9bdc-34eea6c345b9-metrics-certs\") pod \"42caa626-be8b-4d3b-9bdc-34eea6c345b9\" (UID: \"42caa626-be8b-4d3b-9bdc-34eea6c345b9\") "
Sep 17 15:56:56.228333 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:56.228113    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjt2g\" (UniqueName: \"kubernetes.io/projected/42caa626-be8b-4d3b-9bdc-34eea6c345b9-kube-api-access-rjt2g\") pod \"42caa626-be8b-4d3b-9bdc-34eea6c345b9\" (UID: \"42caa626-be8b-4d3b-9bdc-34eea6c345b9\") "
Sep 17 15:56:56.228333 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:56.228153    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42caa626-be8b-4d3b-9bdc-34eea6c345b9-openshift-service-ca\") pod \"42caa626-be8b-4d3b-9bdc-34eea6c345b9\" (UID: \"42caa626-be8b-4d3b-9bdc-34eea6c345b9\") "
Sep 17 15:56:56.228868 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:56.228504    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42caa626-be8b-4d3b-9bdc-34eea6c345b9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "42caa626-be8b-4d3b-9bdc-34eea6c345b9" (UID: "42caa626-be8b-4d3b-9bdc-34eea6c345b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:56:56.228868 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:56.228571    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42caa626-be8b-4d3b-9bdc-34eea6c345b9-openshift-service-ca\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:56:56.230422 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:56.230396    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42caa626-be8b-4d3b-9bdc-34eea6c345b9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "42caa626-be8b-4d3b-9bdc-34eea6c345b9" (UID: "42caa626-be8b-4d3b-9bdc-34eea6c345b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:56:56.230708 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:56.230680    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42caa626-be8b-4d3b-9bdc-34eea6c345b9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "42caa626-be8b-4d3b-9bdc-34eea6c345b9" (UID: "42caa626-be8b-4d3b-9bdc-34eea6c345b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:56:56.230782 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:56.230686    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42caa626-be8b-4d3b-9bdc-34eea6c345b9-kube-api-access-rjt2g" (OuterVolumeSpecName: "kube-api-access-rjt2g") pod "42caa626-be8b-4d3b-9bdc-34eea6c345b9" (UID: "42caa626-be8b-4d3b-9bdc-34eea6c345b9"). InnerVolumeSpecName "kube-api-access-rjt2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:56:56.329730 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:56.329699    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42caa626-be8b-4d3b-9bdc-34eea6c345b9-maas-api-tls\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:56:56.329730 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:56.329727    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42caa626-be8b-4d3b-9bdc-34eea6c345b9-metrics-certs\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:56:56.329730 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:56.329736    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjt2g\" (UniqueName: \"kubernetes.io/projected/42caa626-be8b-4d3b-9bdc-34eea6c345b9-kube-api-access-rjt2g\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:56:56.462530 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:56.462497    2556 generic.go:356] "Generic (PLEG): container finished" podID="42caa626-be8b-4d3b-9bdc-34eea6c345b9" containerID="070259a8dd59c6d35b6a3ae285bb56f2ee0fcd1d3cee7af8ee4734dcf485acac" exitCode=0
Sep 17 15:56:56.462686 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:56.462538    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp"
Sep 17 15:56:56.462686 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:56.462581    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp" event={"ID":"42caa626-be8b-4d3b-9bdc-34eea6c345b9","Type":"ContainerDied","Data":"070259a8dd59c6d35b6a3ae285bb56f2ee0fcd1d3cee7af8ee4734dcf485acac"}
Sep 17 15:56:56.462686 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:56.462634    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp" event={"ID":"42caa626-be8b-4d3b-9bdc-34eea6c345b9","Type":"ContainerDied","Data":"f7fa3ca49fe18ddc310e005fba082f0cbe3b1b5a4fd9187328466e51b35b4ef2"}
Sep 17 15:56:56.462686 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:56.462656    2556 scope.go:117] "RemoveContainer" containerID="070259a8dd59c6d35b6a3ae285bb56f2ee0fcd1d3cee7af8ee4734dcf485acac"
Sep 17 15:56:56.473944 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:56.473921    2556 scope.go:117] "RemoveContainer" containerID="070259a8dd59c6d35b6a3ae285bb56f2ee0fcd1d3cee7af8ee4734dcf485acac"
Sep 17 15:56:56.474766 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:56:56.474745    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"070259a8dd59c6d35b6a3ae285bb56f2ee0fcd1d3cee7af8ee4734dcf485acac\": container with ID starting with 070259a8dd59c6d35b6a3ae285bb56f2ee0fcd1d3cee7af8ee4734dcf485acac not found: ID does not exist" containerID="070259a8dd59c6d35b6a3ae285bb56f2ee0fcd1d3cee7af8ee4734dcf485acac"
Sep 17 15:56:56.474904 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:56.474772    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"070259a8dd59c6d35b6a3ae285bb56f2ee0fcd1d3cee7af8ee4734dcf485acac"} err="failed to get container status \"070259a8dd59c6d35b6a3ae285bb56f2ee0fcd1d3cee7af8ee4734dcf485acac\": rpc error: code = NotFound desc = could not find container \"070259a8dd59c6d35b6a3ae285bb56f2ee0fcd1d3cee7af8ee4734dcf485acac\": container with ID starting with 070259a8dd59c6d35b6a3ae285bb56f2ee0fcd1d3cee7af8ee4734dcf485acac not found: ID does not exist"
Sep 17 15:56:56.484713 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:56.484662    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp"]
Sep 17 15:56:56.488945 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:56.488921    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-608b97-77c9cbf86b-qqfkp"]
Sep 17 15:56:58.136518 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:56:58.136486    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42caa626-be8b-4d3b-9bdc-34eea6c345b9" path="/var/lib/kubelet/pods/42caa626-be8b-4d3b-9bdc-34eea6c345b9/volumes"
Sep 17 15:58:20.661020 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.660992    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"]
Sep 17 15:58:20.661449 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.661417    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16e50e2d-0cc1-4b82-ac28-f28586d5ce18" containerName="maas-api"
Sep 17 15:58:20.661449 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.661430    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="16e50e2d-0cc1-4b82-ac28-f28586d5ce18" containerName="maas-api"
Sep 17 15:58:20.661449 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.661440    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42caa626-be8b-4d3b-9bdc-34eea6c345b9" containerName="maas-api"
Sep 17 15:58:20.661449 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.661445    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="42caa626-be8b-4d3b-9bdc-34eea6c345b9" containerName="maas-api"
Sep 17 15:58:20.661578 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.661462    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7227a335-5e51-4adf-a705-dafc223fd4fa" containerName="maas-api"
Sep 17 15:58:20.661578 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.661468    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7227a335-5e51-4adf-a705-dafc223fd4fa" containerName="maas-api"
Sep 17 15:58:20.661578 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.661483    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2ec914d-76d3-4adf-ba43-8290317b6bd9" containerName="maas-api"
Sep 17 15:58:20.661578 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.661488    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2ec914d-76d3-4adf-ba43-8290317b6bd9" containerName="maas-api"
Sep 17 15:58:20.661578 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.661546    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="42caa626-be8b-4d3b-9bdc-34eea6c345b9" containerName="maas-api"
Sep 17 15:58:20.661578 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.661554    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="16e50e2d-0cc1-4b82-ac28-f28586d5ce18" containerName="maas-api"
Sep 17 15:58:20.661578 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.661562    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2ec914d-76d3-4adf-ba43-8290317b6bd9" containerName="maas-api"
Sep 17 15:58:20.661578 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.661568    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7227a335-5e51-4adf-a705-dafc223fd4fa" containerName="maas-api"
Sep 17 15:58:20.664855 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.664834    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"
Sep 17 15:58:20.670426 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.670407    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1d63c0\"/\"kube-root-ca.crt\""
Sep 17 15:58:20.671551 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.671307    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1d63c0\"/\"openshift-service-ca.crt\""
Sep 17 15:58:20.675855 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.675838    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1d63c0\"/\"rate-test-model-1d63c0-kserve-self-signed-certs\""
Sep 17 15:58:20.676837 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.676818    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1d63c0\"/\"default-dockercfg-ng4jw\""
Sep 17 15:58:20.689501 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.689479    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"]
Sep 17 15:58:20.765643 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.765616    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpkrn\" (UniqueName: \"kubernetes.io/projected/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-kube-api-access-dpkrn\") pod \"rate-test-model-1d63c0-kserve-547985c76c-85lmm\" (UID: \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"
Sep 17 15:58:20.765790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.765666    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-tls-certs\") pod \"rate-test-model-1d63c0-kserve-547985c76c-85lmm\" (UID: \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"
Sep 17 15:58:20.765790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.765706    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-home\") pod \"rate-test-model-1d63c0-kserve-547985c76c-85lmm\" (UID: \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"
Sep 17 15:58:20.765790 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.765735    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-tmp-dir\") pod \"rate-test-model-1d63c0-kserve-547985c76c-85lmm\" (UID: \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"
Sep 17 15:58:20.765939 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.765799    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-dshm\") pod \"rate-test-model-1d63c0-kserve-547985c76c-85lmm\" (UID: \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"
Sep 17 15:58:20.765939 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.765820    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-model-cache\") pod \"rate-test-model-1d63c0-kserve-547985c76c-85lmm\" (UID: \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"
Sep 17 15:58:20.866246 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.866214    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-tmp-dir\") pod \"rate-test-model-1d63c0-kserve-547985c76c-85lmm\" (UID: \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"
Sep 17 15:58:20.866408 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.866260    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-dshm\") pod \"rate-test-model-1d63c0-kserve-547985c76c-85lmm\" (UID: \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"
Sep 17 15:58:20.866408 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.866282    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-model-cache\") pod \"rate-test-model-1d63c0-kserve-547985c76c-85lmm\" (UID: \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"
Sep 17 15:58:20.866408 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.866382    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpkrn\" (UniqueName: \"kubernetes.io/projected/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-kube-api-access-dpkrn\") pod \"rate-test-model-1d63c0-kserve-547985c76c-85lmm\" (UID: \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"
Sep 17 15:58:20.866575 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.866425    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-tls-certs\") pod \"rate-test-model-1d63c0-kserve-547985c76c-85lmm\" (UID: \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"
Sep 17 15:58:20.866575 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.866466    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-home\") pod \"rate-test-model-1d63c0-kserve-547985c76c-85lmm\" (UID: \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"
Sep 17 15:58:20.866686 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.866633    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-tmp-dir\") pod \"rate-test-model-1d63c0-kserve-547985c76c-85lmm\" (UID: \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"
Sep 17 15:58:20.866742 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.866708    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-model-cache\") pod \"rate-test-model-1d63c0-kserve-547985c76c-85lmm\" (UID: \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"
Sep 17 15:58:20.866822 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.866803    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-home\") pod \"rate-test-model-1d63c0-kserve-547985c76c-85lmm\" (UID: \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"
Sep 17 15:58:20.868510 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.868483    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-dshm\") pod \"rate-test-model-1d63c0-kserve-547985c76c-85lmm\" (UID: \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"
Sep 17 15:58:20.868910 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.868893    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-tls-certs\") pod \"rate-test-model-1d63c0-kserve-547985c76c-85lmm\" (UID: \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"
Sep 17 15:58:20.874729 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.874708    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpkrn\" (UniqueName: \"kubernetes.io/projected/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-kube-api-access-dpkrn\") pod \"rate-test-model-1d63c0-kserve-547985c76c-85lmm\" (UID: \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"
Sep 17 15:58:20.974113 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:20.974031    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"
Sep 17 15:58:21.252151 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:21.252127    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"]
Sep 17 15:58:21.253771 ip-10-0-134-104 kubenswrapper[2556]: W0917 15:58:21.253738    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ca6a0c1_b6bc_4a11_85a8_8887ea9da216.slice/crio-5e0099e8692c7c8707aafd5a7aefcc747d319593629e8a3f8f66713a473b2334 WatchSource:0}: Error finding container 5e0099e8692c7c8707aafd5a7aefcc747d319593629e8a3f8f66713a473b2334: Status 404 returned error can't find the container with id 5e0099e8692c7c8707aafd5a7aefcc747d319593629e8a3f8f66713a473b2334
Sep 17 15:58:21.878507 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:21.878474    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm" event={"ID":"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216","Type":"ContainerStarted","Data":"09c5008287a1e43d3438a1df1951e25ee38390505a1a42a61d9405091be92a16"}
Sep 17 15:58:21.878846 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:21.878514    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm" event={"ID":"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216","Type":"ContainerStarted","Data":"5e0099e8692c7c8707aafd5a7aefcc747d319593629e8a3f8f66713a473b2334"}
Sep 17 15:58:21.912825 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:21.912782    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm" podStartSLOduration=1.912768636 podStartE2EDuration="1.912768636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:58:21.911247168 +0000 UTC m=+1724.291264922" watchObservedRunningTime="2026-09-17 15:58:21.912768636 +0000 UTC m=+1724.292786390"
Sep 17 15:58:30.974430 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:30.974392    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"
Sep 17 15:58:30.974430 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:30.974437    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"
Sep 17 15:58:30.986750 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:30.986731    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"
Sep 17 15:58:31.939559 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:58:31.939530    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"
Sep 17 15:59:31.547017 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:31.546933    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"]
Sep 17 15:59:38.208219 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:38.208190    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-45zt7_2c069c9e-0856-46b0-ba10-67829c05ac94/console-operator/1.log"
Sep 17 15:59:38.209644 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:38.209620    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-45zt7_2c069c9e-0856-46b0-ba10-67829c05ac94/console-operator/1.log"
Sep 17 15:59:46.565578 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:46.565518    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm" podUID="6ca6a0c1-b6bc-4a11-85a8-8887ea9da216" containerName="main" containerID="cri-o://09c5008287a1e43d3438a1df1951e25ee38390505a1a42a61d9405091be92a16" gracePeriod=45
Sep 17 15:59:46.811120 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:46.811095    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"
Sep 17 15:59:46.852842 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:46.852818    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-tls-certs\") pod \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\" (UID: \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\") "
Sep 17 15:59:46.852974 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:46.852853    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-home\") pod \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\" (UID: \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\") "
Sep 17 15:59:46.852974 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:46.852870    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-tmp-dir\") pod \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\" (UID: \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\") "
Sep 17 15:59:46.852974 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:46.852913    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-model-cache\") pod \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\" (UID: \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\") "
Sep 17 15:59:46.852974 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:46.852952    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpkrn\" (UniqueName: \"kubernetes.io/projected/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-kube-api-access-dpkrn\") pod \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\" (UID: \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\") "
Sep 17 15:59:46.853210 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:46.852981    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-dshm\") pod \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\" (UID: \"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216\") "
Sep 17 15:59:46.853210 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:46.853111    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-home" (OuterVolumeSpecName: "home") pod "6ca6a0c1-b6bc-4a11-85a8-8887ea9da216" (UID: "6ca6a0c1-b6bc-4a11-85a8-8887ea9da216"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:59:46.853333 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:46.853213    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ca6a0c1-b6bc-4a11-85a8-8887ea9da216" (UID: "6ca6a0c1-b6bc-4a11-85a8-8887ea9da216"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:59:46.853377 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:46.853329    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-home\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:59:46.853377 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:46.853346    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-tmp-dir\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:59:46.853462 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:46.853437    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ca6a0c1-b6bc-4a11-85a8-8887ea9da216" (UID: "6ca6a0c1-b6bc-4a11-85a8-8887ea9da216"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:59:46.855180 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:46.855156    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-dshm" (OuterVolumeSpecName: "dshm") pod "6ca6a0c1-b6bc-4a11-85a8-8887ea9da216" (UID: "6ca6a0c1-b6bc-4a11-85a8-8887ea9da216"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:59:46.855462 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:46.855439    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ca6a0c1-b6bc-4a11-85a8-8887ea9da216" (UID: "6ca6a0c1-b6bc-4a11-85a8-8887ea9da216"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:59:46.855550 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:46.855470    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-kube-api-access-dpkrn" (OuterVolumeSpecName: "kube-api-access-dpkrn") pod "6ca6a0c1-b6bc-4a11-85a8-8887ea9da216" (UID: "6ca6a0c1-b6bc-4a11-85a8-8887ea9da216"). InnerVolumeSpecName "kube-api-access-dpkrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:59:46.954486 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:46.954463    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-tls-certs\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:59:46.954486 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:46.954483    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-model-cache\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:59:46.954631 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:46.954492    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpkrn\" (UniqueName: \"kubernetes.io/projected/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-kube-api-access-dpkrn\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:59:46.954631 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:46.954502    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216-dshm\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 15:59:47.306855 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:47.306823    2556 generic.go:356] "Generic (PLEG): container finished" podID="6ca6a0c1-b6bc-4a11-85a8-8887ea9da216" containerID="09c5008287a1e43d3438a1df1951e25ee38390505a1a42a61d9405091be92a16" exitCode=0
Sep 17 15:59:47.307018 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:47.306862    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"
Sep 17 15:59:47.307018 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:47.306873    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm" event={"ID":"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216","Type":"ContainerDied","Data":"09c5008287a1e43d3438a1df1951e25ee38390505a1a42a61d9405091be92a16"}
Sep 17 15:59:47.307018 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:47.306904    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm" event={"ID":"6ca6a0c1-b6bc-4a11-85a8-8887ea9da216","Type":"ContainerDied","Data":"5e0099e8692c7c8707aafd5a7aefcc747d319593629e8a3f8f66713a473b2334"}
Sep 17 15:59:47.307018 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:47.306920    2556 scope.go:117] "RemoveContainer" containerID="09c5008287a1e43d3438a1df1951e25ee38390505a1a42a61d9405091be92a16"
Sep 17 15:59:47.316260 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:47.316241    2556 scope.go:117] "RemoveContainer" containerID="09c5008287a1e43d3438a1df1951e25ee38390505a1a42a61d9405091be92a16"
Sep 17 15:59:47.316524 ip-10-0-134-104 kubenswrapper[2556]: E0917 15:59:47.316505    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09c5008287a1e43d3438a1df1951e25ee38390505a1a42a61d9405091be92a16\": container with ID starting with 09c5008287a1e43d3438a1df1951e25ee38390505a1a42a61d9405091be92a16 not found: ID does not exist" containerID="09c5008287a1e43d3438a1df1951e25ee38390505a1a42a61d9405091be92a16"
Sep 17 15:59:47.316571 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:47.316532    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09c5008287a1e43d3438a1df1951e25ee38390505a1a42a61d9405091be92a16"} err="failed to get container status \"09c5008287a1e43d3438a1df1951e25ee38390505a1a42a61d9405091be92a16\": rpc error: code = NotFound desc = could not find container \"09c5008287a1e43d3438a1df1951e25ee38390505a1a42a61d9405091be92a16\": container with ID starting with 09c5008287a1e43d3438a1df1951e25ee38390505a1a42a61d9405091be92a16 not found: ID does not exist"
Sep 17 15:59:47.363226 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:47.363195    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"]
Sep 17 15:59:47.369847 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:47.369823    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d63c0/rate-test-model-1d63c0-kserve-547985c76c-85lmm"]
Sep 17 15:59:48.136274 ip-10-0-134-104 kubenswrapper[2556]: I0917 15:59:48.136247    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ca6a0c1-b6bc-4a11-85a8-8887ea9da216" path="/var/lib/kubelet/pods/6ca6a0c1-b6bc-4a11-85a8-8887ea9da216/volumes"
Sep 17 16:00:08.086515 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:08.086474    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7"]
Sep 17 16:00:08.087151 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:08.087123    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ca6a0c1-b6bc-4a11-85a8-8887ea9da216" containerName="main"
Sep 17 16:00:08.087151 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:08.087143    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ca6a0c1-b6bc-4a11-85a8-8887ea9da216" containerName="main"
Sep 17 16:00:08.087331 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:08.087235    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ca6a0c1-b6bc-4a11-85a8-8887ea9da216" containerName="main"
Sep 17 16:00:08.092023 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:08.092003    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7"
Sep 17 16:00:08.101380 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:08.101360    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-ddedf5-serving-cert\""
Sep 17 16:00:08.110969 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:08.110945    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7"]
Sep 17 16:00:08.244196 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:08.244161    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/846d4c21-2957-4679-826d-3ae4ed841928-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7\" (UID: \"846d4c21-2957-4679-826d-3ae4ed841928\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7"
Sep 17 16:00:08.244358 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:08.244226    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/846d4c21-2957-4679-826d-3ae4ed841928-metrics-certs\") pod \"maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7\" (UID: \"846d4c21-2957-4679-826d-3ae4ed841928\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7"
Sep 17 16:00:08.244358 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:08.244265    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/846d4c21-2957-4679-826d-3ae4ed841928-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7\" (UID: \"846d4c21-2957-4679-826d-3ae4ed841928\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7"
Sep 17 16:00:08.244486 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:08.244385    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qppd\" (UniqueName: \"kubernetes.io/projected/846d4c21-2957-4679-826d-3ae4ed841928-kube-api-access-9qppd\") pod \"maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7\" (UID: \"846d4c21-2957-4679-826d-3ae4ed841928\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7"
Sep 17 16:00:08.345700 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:08.345638    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qppd\" (UniqueName: \"kubernetes.io/projected/846d4c21-2957-4679-826d-3ae4ed841928-kube-api-access-9qppd\") pod \"maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7\" (UID: \"846d4c21-2957-4679-826d-3ae4ed841928\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7"
Sep 17 16:00:08.345700 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:08.345684    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/846d4c21-2957-4679-826d-3ae4ed841928-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7\" (UID: \"846d4c21-2957-4679-826d-3ae4ed841928\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7"
Sep 17 16:00:08.345846 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:08.345717    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/846d4c21-2957-4679-826d-3ae4ed841928-metrics-certs\") pod \"maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7\" (UID: \"846d4c21-2957-4679-826d-3ae4ed841928\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7"
Sep 17 16:00:08.345911 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:08.345891    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/846d4c21-2957-4679-826d-3ae4ed841928-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7\" (UID: \"846d4c21-2957-4679-826d-3ae4ed841928\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7"
Sep 17 16:00:08.346037 ip-10-0-134-104 kubenswrapper[2556]: E0917 16:00:08.346012    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-serving-cert: secret "maas-api-e2e-ipp-b-ddedf5-serving-cert" not found
Sep 17 16:00:08.346138 ip-10-0-134-104 kubenswrapper[2556]: E0917 16:00:08.346118    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/846d4c21-2957-4679-826d-3ae4ed841928-maas-api-tls podName:846d4c21-2957-4679-826d-3ae4ed841928 nodeName:}" failed. No retries permitted until 2026-09-17 16:00:08.84609499 +0000 UTC m=+1831.226112744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/846d4c21-2957-4679-826d-3ae4ed841928-maas-api-tls") pod "maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7" (UID: "846d4c21-2957-4679-826d-3ae4ed841928") : secret "maas-api-e2e-ipp-b-ddedf5-serving-cert" not found
Sep 17 16:00:08.346405 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:08.346385    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/846d4c21-2957-4679-826d-3ae4ed841928-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7\" (UID: \"846d4c21-2957-4679-826d-3ae4ed841928\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7"
Sep 17 16:00:08.348171 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:08.348153    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/846d4c21-2957-4679-826d-3ae4ed841928-metrics-certs\") pod \"maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7\" (UID: \"846d4c21-2957-4679-826d-3ae4ed841928\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7"
Sep 17 16:00:08.359525 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:08.359498    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qppd\" (UniqueName: \"kubernetes.io/projected/846d4c21-2957-4679-826d-3ae4ed841928-kube-api-access-9qppd\") pod \"maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7\" (UID: \"846d4c21-2957-4679-826d-3ae4ed841928\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7"
Sep 17 16:00:08.850127 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:08.850090    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/846d4c21-2957-4679-826d-3ae4ed841928-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7\" (UID: \"846d4c21-2957-4679-826d-3ae4ed841928\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7"
Sep 17 16:00:08.852600 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:08.852580    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/846d4c21-2957-4679-826d-3ae4ed841928-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7\" (UID: \"846d4c21-2957-4679-826d-3ae4ed841928\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7"
Sep 17 16:00:08.902316 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:08.902288    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z"]
Sep 17 16:00:08.905985 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:08.905970    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z"
Sep 17 16:00:08.909719 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:08.909699    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ddedf5\""
Sep 17 16:00:08.922262 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:08.922241    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z"]
Sep 17 16:00:09.002310 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:09.002288    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7"
Sep 17 16:00:09.052535 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:09.052502    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z\" (UID: \"ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z"
Sep 17 16:00:09.052658 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:09.052608    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zfsr\" (UniqueName: \"kubernetes.io/projected/ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c-kube-api-access-8zfsr\") pod \"payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z\" (UID: \"ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z"
Sep 17 16:00:09.153299 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:09.153214    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zfsr\" (UniqueName: \"kubernetes.io/projected/ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c-kube-api-access-8zfsr\") pod \"payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z\" (UID: \"ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z"
Sep 17 16:00:09.153678 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:09.153333    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z\" (UID: \"ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z"
Sep 17 16:00:09.154044 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:09.154022    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z\" (UID: \"ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z"
Sep 17 16:00:09.163571 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:09.163542    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zfsr\" (UniqueName: \"kubernetes.io/projected/ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c-kube-api-access-8zfsr\") pod \"payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z\" (UID: \"ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z"
Sep 17 16:00:09.216446 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:09.216422    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z"
Sep 17 16:00:09.281779 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:09.281749    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7"]
Sep 17 16:00:09.284048 ip-10-0-134-104 kubenswrapper[2556]: W0917 16:00:09.284017    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846d4c21_2957_4679_826d_3ae4ed841928.slice/crio-4744be0f8885d88e817c2efe8af79e3bc5738b9baba26fcfc1f21603733e8588 WatchSource:0}: Error finding container 4744be0f8885d88e817c2efe8af79e3bc5738b9baba26fcfc1f21603733e8588: Status 404 returned error can't find the container with id 4744be0f8885d88e817c2efe8af79e3bc5738b9baba26fcfc1f21603733e8588
Sep 17 16:00:09.285523 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:09.285503    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:00:09.416917 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:09.416839    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7" event={"ID":"846d4c21-2957-4679-826d-3ae4ed841928","Type":"ContainerStarted","Data":"4744be0f8885d88e817c2efe8af79e3bc5738b9baba26fcfc1f21603733e8588"}
Sep 17 16:00:09.514724 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:09.514694    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z"]
Sep 17 16:00:09.518662 ip-10-0-134-104 kubenswrapper[2556]: W0917 16:00:09.518635    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded7a40bc_f76e_4735_83ca_1a9cd3cbc72c.slice/crio-9398a2a78d4e9cafe9e1fa491cffa44f00f2f08484ad43397a84bfd42f0af167 WatchSource:0}: Error finding container 9398a2a78d4e9cafe9e1fa491cffa44f00f2f08484ad43397a84bfd42f0af167: Status 404 returned error can't find the container with id 9398a2a78d4e9cafe9e1fa491cffa44f00f2f08484ad43397a84bfd42f0af167
Sep 17 16:00:10.424024 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:10.423988    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z" event={"ID":"ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c","Type":"ContainerStarted","Data":"715e8dc6d91090a2af63ee31dc86e312fe4b148ac9824ea608678da3d4ac651c"}
Sep 17 16:00:10.424414 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:10.424045    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z" event={"ID":"ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c","Type":"ContainerStarted","Data":"9398a2a78d4e9cafe9e1fa491cffa44f00f2f08484ad43397a84bfd42f0af167"}
Sep 17 16:00:10.424414 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:10.424099    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z"
Sep 17 16:00:10.449254 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:10.449209    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z" podStartSLOduration=2.449195187 podStartE2EDuration="2.449195187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:00:10.448250158 +0000 UTC m=+1832.828267916" watchObservedRunningTime="2026-09-17 16:00:10.449195187 +0000 UTC m=+1832.829212962"
Sep 17 16:00:11.430302 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:11.430264    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7" event={"ID":"846d4c21-2957-4679-826d-3ae4ed841928","Type":"ContainerStarted","Data":"e100e03914221dd4e8b009264801fa41310ef5e4c47a5c7d6c7738a37b5decf5"}
Sep 17 16:00:11.430692 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:11.430488    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7"
Sep 17 16:00:11.454128 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:11.454072    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7" podStartSLOduration=2.272941899 podStartE2EDuration="3.454040377s" totalImagesPullingTime="1.181098478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:00:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:00:11.451905244 +0000 UTC m=+1833.831923001" watchObservedRunningTime="2026-09-17 16:00:11.454040377 +0000 UTC m=+1833.834058133"
Sep 17 16:00:17.440215 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:17.440182    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7"
Sep 17 16:00:21.432561 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:21.432529    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z"
Sep 17 16:00:35.910220 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:35.910188    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-2wlv5_5bd2083c-fee3-4da3-95dd-1be423dab576/payload-processing/0.log"
Sep 17 16:00:36.077666 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:00:36.077638    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z_ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c/payload-processing/0.log"
Sep 17 16:01:12.614829 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:12.614752    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-2wlv5_5bd2083c-fee3-4da3-95dd-1be423dab576/payload-processing/0.log"
Sep 17 16:01:12.741324 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:12.741299    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z_ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c/payload-processing/0.log"
Sep 17 16:01:12.875469 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:12.875400    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z_ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c/payload-processing/0.log"
Sep 17 16:01:22.250317 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:22.250289    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q"]
Sep 17 16:01:22.254454 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:22.254432    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q"
Sep 17 16:01:22.256759 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:22.256736    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-ec94ed-serving-cert\""
Sep 17 16:01:22.266707 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:22.266679    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q"]
Sep 17 16:01:22.396464 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:22.396430    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqbfr\" (UniqueName: \"kubernetes.io/projected/5011a14d-647a-49b1-bd42-75e36ef20ab3-kube-api-access-kqbfr\") pod \"maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q\" (UID: \"5011a14d-647a-49b1-bd42-75e36ef20ab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q"
Sep 17 16:01:22.396653 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:22.396480    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5011a14d-647a-49b1-bd42-75e36ef20ab3-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q\" (UID: \"5011a14d-647a-49b1-bd42-75e36ef20ab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q"
Sep 17 16:01:22.396653 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:22.396512    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5011a14d-647a-49b1-bd42-75e36ef20ab3-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q\" (UID: \"5011a14d-647a-49b1-bd42-75e36ef20ab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q"
Sep 17 16:01:22.396653 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:22.396613    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5011a14d-647a-49b1-bd42-75e36ef20ab3-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q\" (UID: \"5011a14d-647a-49b1-bd42-75e36ef20ab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q"
Sep 17 16:01:22.497097 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:22.497043    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5011a14d-647a-49b1-bd42-75e36ef20ab3-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q\" (UID: \"5011a14d-647a-49b1-bd42-75e36ef20ab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q"
Sep 17 16:01:22.497283 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:22.497131    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqbfr\" (UniqueName: \"kubernetes.io/projected/5011a14d-647a-49b1-bd42-75e36ef20ab3-kube-api-access-kqbfr\") pod \"maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q\" (UID: \"5011a14d-647a-49b1-bd42-75e36ef20ab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q"
Sep 17 16:01:22.497283 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:22.497165    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5011a14d-647a-49b1-bd42-75e36ef20ab3-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q\" (UID: \"5011a14d-647a-49b1-bd42-75e36ef20ab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q"
Sep 17 16:01:22.497283 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:22.497183    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5011a14d-647a-49b1-bd42-75e36ef20ab3-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q\" (UID: \"5011a14d-647a-49b1-bd42-75e36ef20ab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q"
Sep 17 16:01:22.497776 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:22.497754    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5011a14d-647a-49b1-bd42-75e36ef20ab3-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q\" (UID: \"5011a14d-647a-49b1-bd42-75e36ef20ab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q"
Sep 17 16:01:22.499759 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:22.499731    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5011a14d-647a-49b1-bd42-75e36ef20ab3-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q\" (UID: \"5011a14d-647a-49b1-bd42-75e36ef20ab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q"
Sep 17 16:01:22.499844 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:22.499761    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5011a14d-647a-49b1-bd42-75e36ef20ab3-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q\" (UID: \"5011a14d-647a-49b1-bd42-75e36ef20ab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q"
Sep 17 16:01:22.505592 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:22.505538    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqbfr\" (UniqueName: \"kubernetes.io/projected/5011a14d-647a-49b1-bd42-75e36ef20ab3-kube-api-access-kqbfr\") pod \"maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q\" (UID: \"5011a14d-647a-49b1-bd42-75e36ef20ab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q"
Sep 17 16:01:22.567509 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:22.567483    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q"
Sep 17 16:01:22.860860 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:22.860820    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q"]
Sep 17 16:01:22.865459 ip-10-0-134-104 kubenswrapper[2556]: W0917 16:01:22.865423    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5011a14d_647a_49b1_bd42_75e36ef20ab3.slice/crio-517a7bffe11469d2ca1e00f37dd812e0395eb55721ffd3f42b435d37b1781e2c WatchSource:0}: Error finding container 517a7bffe11469d2ca1e00f37dd812e0395eb55721ffd3f42b435d37b1781e2c: Status 404 returned error can't find the container with id 517a7bffe11469d2ca1e00f37dd812e0395eb55721ffd3f42b435d37b1781e2c
Sep 17 16:01:23.793660 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:23.793621    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q" event={"ID":"5011a14d-647a-49b1-bd42-75e36ef20ab3","Type":"ContainerStarted","Data":"0fc4b758aec57efa9f475ab1cf61ef318543f7402a24d0da5e8ddccb79339e64"}
Sep 17 16:01:23.794034 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:23.793676    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q" event={"ID":"5011a14d-647a-49b1-bd42-75e36ef20ab3","Type":"ContainerStarted","Data":"517a7bffe11469d2ca1e00f37dd812e0395eb55721ffd3f42b435d37b1781e2c"}
Sep 17 16:01:23.794034 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:23.793697    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q"
Sep 17 16:01:23.820247 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:23.820164    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q" podStartSLOduration=1.141718588 podStartE2EDuration="1.820148738s" totalImagesPullingTime="678.43015ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:01:23.817891655 +0000 UTC m=+1906.197909411" watchObservedRunningTime="2026-09-17 16:01:23.820148738 +0000 UTC m=+1906.200166494"
Sep 17 16:01:29.803873 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:29.803842    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q"
Sep 17 16:01:59.173593 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:59.173548    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q"]
Sep 17 16:01:59.174214 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:59.173885    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q" podUID="5011a14d-647a-49b1-bd42-75e36ef20ab3" containerName="maas-api" containerID="cri-o://0fc4b758aec57efa9f475ab1cf61ef318543f7402a24d0da5e8ddccb79339e64" gracePeriod=30
Sep 17 16:01:59.441589 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:59.441563    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q"
Sep 17 16:01:59.534257 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:59.534224    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5011a14d-647a-49b1-bd42-75e36ef20ab3-maas-api-tls\") pod \"5011a14d-647a-49b1-bd42-75e36ef20ab3\" (UID: \"5011a14d-647a-49b1-bd42-75e36ef20ab3\") "
Sep 17 16:01:59.534448 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:59.534275    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqbfr\" (UniqueName: \"kubernetes.io/projected/5011a14d-647a-49b1-bd42-75e36ef20ab3-kube-api-access-kqbfr\") pod \"5011a14d-647a-49b1-bd42-75e36ef20ab3\" (UID: \"5011a14d-647a-49b1-bd42-75e36ef20ab3\") "
Sep 17 16:01:59.534448 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:59.534308    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5011a14d-647a-49b1-bd42-75e36ef20ab3-openshift-service-ca\") pod \"5011a14d-647a-49b1-bd42-75e36ef20ab3\" (UID: \"5011a14d-647a-49b1-bd42-75e36ef20ab3\") "
Sep 17 16:01:59.534579 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:59.534518    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5011a14d-647a-49b1-bd42-75e36ef20ab3-metrics-certs\") pod \"5011a14d-647a-49b1-bd42-75e36ef20ab3\" (UID: \"5011a14d-647a-49b1-bd42-75e36ef20ab3\") "
Sep 17 16:01:59.534691 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:59.534656    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5011a14d-647a-49b1-bd42-75e36ef20ab3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5011a14d-647a-49b1-bd42-75e36ef20ab3" (UID: "5011a14d-647a-49b1-bd42-75e36ef20ab3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:01:59.534906 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:59.534889    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5011a14d-647a-49b1-bd42-75e36ef20ab3-openshift-service-ca\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 16:01:59.536590 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:59.536559    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5011a14d-647a-49b1-bd42-75e36ef20ab3-kube-api-access-kqbfr" (OuterVolumeSpecName: "kube-api-access-kqbfr") pod "5011a14d-647a-49b1-bd42-75e36ef20ab3" (UID: "5011a14d-647a-49b1-bd42-75e36ef20ab3"). InnerVolumeSpecName "kube-api-access-kqbfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:01:59.536677 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:59.536600    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5011a14d-647a-49b1-bd42-75e36ef20ab3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5011a14d-647a-49b1-bd42-75e36ef20ab3" (UID: "5011a14d-647a-49b1-bd42-75e36ef20ab3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:01:59.536719 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:59.536671    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5011a14d-647a-49b1-bd42-75e36ef20ab3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5011a14d-647a-49b1-bd42-75e36ef20ab3" (UID: "5011a14d-647a-49b1-bd42-75e36ef20ab3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:01:59.635983 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:59.635951    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5011a14d-647a-49b1-bd42-75e36ef20ab3-maas-api-tls\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 16:01:59.635983 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:59.635983    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqbfr\" (UniqueName: \"kubernetes.io/projected/5011a14d-647a-49b1-bd42-75e36ef20ab3-kube-api-access-kqbfr\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 16:01:59.636220 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:59.635993    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5011a14d-647a-49b1-bd42-75e36ef20ab3-metrics-certs\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 16:01:59.986876 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:59.986844    2556 generic.go:356] "Generic (PLEG): container finished" podID="5011a14d-647a-49b1-bd42-75e36ef20ab3" containerID="0fc4b758aec57efa9f475ab1cf61ef318543f7402a24d0da5e8ddccb79339e64" exitCode=0
Sep 17 16:01:59.987115 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:59.986894    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q"
Sep 17 16:01:59.987115 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:59.986934    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q" event={"ID":"5011a14d-647a-49b1-bd42-75e36ef20ab3","Type":"ContainerDied","Data":"0fc4b758aec57efa9f475ab1cf61ef318543f7402a24d0da5e8ddccb79339e64"}
Sep 17 16:01:59.987115 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:59.986991    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q" event={"ID":"5011a14d-647a-49b1-bd42-75e36ef20ab3","Type":"ContainerDied","Data":"517a7bffe11469d2ca1e00f37dd812e0395eb55721ffd3f42b435d37b1781e2c"}
Sep 17 16:01:59.987115 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:59.987017    2556 scope.go:117] "RemoveContainer" containerID="0fc4b758aec57efa9f475ab1cf61ef318543f7402a24d0da5e8ddccb79339e64"
Sep 17 16:01:59.998402 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:59.998376    2556 scope.go:117] "RemoveContainer" containerID="0fc4b758aec57efa9f475ab1cf61ef318543f7402a24d0da5e8ddccb79339e64"
Sep 17 16:01:59.998737 ip-10-0-134-104 kubenswrapper[2556]: E0917 16:01:59.998700    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fc4b758aec57efa9f475ab1cf61ef318543f7402a24d0da5e8ddccb79339e64\": container with ID starting with 0fc4b758aec57efa9f475ab1cf61ef318543f7402a24d0da5e8ddccb79339e64 not found: ID does not exist" containerID="0fc4b758aec57efa9f475ab1cf61ef318543f7402a24d0da5e8ddccb79339e64"
Sep 17 16:01:59.998876 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:01:59.998744    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fc4b758aec57efa9f475ab1cf61ef318543f7402a24d0da5e8ddccb79339e64"} err="failed to get container status \"0fc4b758aec57efa9f475ab1cf61ef318543f7402a24d0da5e8ddccb79339e64\": rpc error: code = NotFound desc = could not find container \"0fc4b758aec57efa9f475ab1cf61ef318543f7402a24d0da5e8ddccb79339e64\": container with ID starting with 0fc4b758aec57efa9f475ab1cf61ef318543f7402a24d0da5e8ddccb79339e64 not found: ID does not exist"
Sep 17 16:02:00.013008 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:02:00.012975    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q"]
Sep 17 16:02:00.016087 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:02:00.016048    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ec94ed-dd8864f64-7px9q"]
Sep 17 16:02:00.136233 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:02:00.136199    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5011a14d-647a-49b1-bd42-75e36ef20ab3" path="/var/lib/kubelet/pods/5011a14d-647a-49b1-bd42-75e36ef20ab3/volumes"
Sep 17 16:03:41.087549 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.087511    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7"]
Sep 17 16:03:41.087988 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.087699    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7" podUID="846d4c21-2957-4679-826d-3ae4ed841928" containerName="maas-api" containerID="cri-o://e100e03914221dd4e8b009264801fa41310ef5e4c47a5c7d6c7738a37b5decf5" gracePeriod=30
Sep 17 16:03:41.282801 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.282767    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z"]
Sep 17 16:03:41.283030 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.282983    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z" podUID="ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c" containerName="payload-processing" containerID="cri-o://715e8dc6d91090a2af63ee31dc86e312fe4b148ac9824ea608678da3d4ac651c" gracePeriod=30
Sep 17 16:03:41.410804 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.410782    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7"
Sep 17 16:03:41.431482 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.431442    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z" podUID="ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.66:9004: connect: connection refused"
Sep 17 16:03:41.481319 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.481262    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/846d4c21-2957-4679-826d-3ae4ed841928-maas-api-tls\") pod \"846d4c21-2957-4679-826d-3ae4ed841928\" (UID: \"846d4c21-2957-4679-826d-3ae4ed841928\") "
Sep 17 16:03:41.481800 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.481727    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qppd\" (UniqueName: \"kubernetes.io/projected/846d4c21-2957-4679-826d-3ae4ed841928-kube-api-access-9qppd\") pod \"846d4c21-2957-4679-826d-3ae4ed841928\" (UID: \"846d4c21-2957-4679-826d-3ae4ed841928\") "
Sep 17 16:03:41.482028 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.482003    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/846d4c21-2957-4679-826d-3ae4ed841928-metrics-certs\") pod \"846d4c21-2957-4679-826d-3ae4ed841928\" (UID: \"846d4c21-2957-4679-826d-3ae4ed841928\") "
Sep 17 16:03:41.482180 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.482099    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/846d4c21-2957-4679-826d-3ae4ed841928-openshift-service-ca\") pod \"846d4c21-2957-4679-826d-3ae4ed841928\" (UID: \"846d4c21-2957-4679-826d-3ae4ed841928\") "
Sep 17 16:03:41.482852 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.482806    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/846d4c21-2957-4679-826d-3ae4ed841928-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "846d4c21-2957-4679-826d-3ae4ed841928" (UID: "846d4c21-2957-4679-826d-3ae4ed841928"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:03:41.485624 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.485593    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846d4c21-2957-4679-826d-3ae4ed841928-kube-api-access-9qppd" (OuterVolumeSpecName: "kube-api-access-9qppd") pod "846d4c21-2957-4679-826d-3ae4ed841928" (UID: "846d4c21-2957-4679-826d-3ae4ed841928"). InnerVolumeSpecName "kube-api-access-9qppd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:03:41.485808 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.485782    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/846d4c21-2957-4679-826d-3ae4ed841928-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "846d4c21-2957-4679-826d-3ae4ed841928" (UID: "846d4c21-2957-4679-826d-3ae4ed841928"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:03:41.486212 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.486188    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/846d4c21-2957-4679-826d-3ae4ed841928-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "846d4c21-2957-4679-826d-3ae4ed841928" (UID: "846d4c21-2957-4679-826d-3ae4ed841928"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:03:41.501367 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.501336    2556 generic.go:356] "Generic (PLEG): container finished" podID="846d4c21-2957-4679-826d-3ae4ed841928" containerID="e100e03914221dd4e8b009264801fa41310ef5e4c47a5c7d6c7738a37b5decf5" exitCode=0
Sep 17 16:03:41.501498 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.501432    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7"
Sep 17 16:03:41.501498 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.501478    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7" event={"ID":"846d4c21-2957-4679-826d-3ae4ed841928","Type":"ContainerDied","Data":"e100e03914221dd4e8b009264801fa41310ef5e4c47a5c7d6c7738a37b5decf5"}
Sep 17 16:03:41.501620 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.501539    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7" event={"ID":"846d4c21-2957-4679-826d-3ae4ed841928","Type":"ContainerDied","Data":"4744be0f8885d88e817c2efe8af79e3bc5738b9baba26fcfc1f21603733e8588"}
Sep 17 16:03:41.501620 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.501572    2556 scope.go:117] "RemoveContainer" containerID="e100e03914221dd4e8b009264801fa41310ef5e4c47a5c7d6c7738a37b5decf5"
Sep 17 16:03:41.503481 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.503456    2556 generic.go:356] "Generic (PLEG): container finished" podID="ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c" containerID="715e8dc6d91090a2af63ee31dc86e312fe4b148ac9824ea608678da3d4ac651c" exitCode=0
Sep 17 16:03:41.503599 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.503515    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z" event={"ID":"ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c","Type":"ContainerDied","Data":"715e8dc6d91090a2af63ee31dc86e312fe4b148ac9824ea608678da3d4ac651c"}
Sep 17 16:03:41.512027 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.512013    2556 scope.go:117] "RemoveContainer" containerID="e100e03914221dd4e8b009264801fa41310ef5e4c47a5c7d6c7738a37b5decf5"
Sep 17 16:03:41.512359 ip-10-0-134-104 kubenswrapper[2556]: E0917 16:03:41.512337    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e100e03914221dd4e8b009264801fa41310ef5e4c47a5c7d6c7738a37b5decf5\": container with ID starting with e100e03914221dd4e8b009264801fa41310ef5e4c47a5c7d6c7738a37b5decf5 not found: ID does not exist" containerID="e100e03914221dd4e8b009264801fa41310ef5e4c47a5c7d6c7738a37b5decf5"
Sep 17 16:03:41.512446 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.512363    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e100e03914221dd4e8b009264801fa41310ef5e4c47a5c7d6c7738a37b5decf5"} err="failed to get container status \"e100e03914221dd4e8b009264801fa41310ef5e4c47a5c7d6c7738a37b5decf5\": rpc error: code = NotFound desc = could not find container \"e100e03914221dd4e8b009264801fa41310ef5e4c47a5c7d6c7738a37b5decf5\": container with ID starting with e100e03914221dd4e8b009264801fa41310ef5e4c47a5c7d6c7738a37b5decf5 not found: ID does not exist"
Sep 17 16:03:41.531705 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.531688    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z"
Sep 17 16:03:41.545116 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.543702    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7"]
Sep 17 16:03:41.553951 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.553926    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddedf5-76c489bbf4-vstr7"]
Sep 17 16:03:41.583384 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.583351    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qppd\" (UniqueName: \"kubernetes.io/projected/846d4c21-2957-4679-826d-3ae4ed841928-kube-api-access-9qppd\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 16:03:41.583384 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.583385    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/846d4c21-2957-4679-826d-3ae4ed841928-metrics-certs\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 16:03:41.583618 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.583399    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/846d4c21-2957-4679-826d-3ae4ed841928-openshift-service-ca\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 16:03:41.583618 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.583412    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/846d4c21-2957-4679-826d-3ae4ed841928-maas-api-tls\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 16:03:41.684133 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.684033    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c-plugins-config-volume\") pod \"ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c\" (UID: \"ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c\") "
Sep 17 16:03:41.684294 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.684131    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zfsr\" (UniqueName: \"kubernetes.io/projected/ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c-kube-api-access-8zfsr\") pod \"ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c\" (UID: \"ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c\") "
Sep 17 16:03:41.684398 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.684375    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c" (UID: "ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:03:41.686098 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.686079    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c-kube-api-access-8zfsr" (OuterVolumeSpecName: "kube-api-access-8zfsr") pod "ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c" (UID: "ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c"). InnerVolumeSpecName "kube-api-access-8zfsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:03:41.785820 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.785790    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c-plugins-config-volume\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 16:03:41.785820 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:41.785815    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zfsr\" (UniqueName: \"kubernetes.io/projected/ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c-kube-api-access-8zfsr\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 16:03:42.136580 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:42.136547    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="846d4c21-2957-4679-826d-3ae4ed841928" path="/var/lib/kubelet/pods/846d4c21-2957-4679-826d-3ae4ed841928/volumes"
Sep 17 16:03:42.511350 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:42.511262    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z" event={"ID":"ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c","Type":"ContainerDied","Data":"9398a2a78d4e9cafe9e1fa491cffa44f00f2f08484ad43397a84bfd42f0af167"}
Sep 17 16:03:42.511350 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:42.511309    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z"
Sep 17 16:03:42.511350 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:42.511322    2556 scope.go:117] "RemoveContainer" containerID="715e8dc6d91090a2af63ee31dc86e312fe4b148ac9824ea608678da3d4ac651c"
Sep 17 16:03:42.552856 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:42.552820    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z"]
Sep 17 16:03:42.563481 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:42.563452    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ddedf5-78c48df79f-lxd9z"]
Sep 17 16:03:44.135902 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:03:44.135872    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c" path="/var/lib/kubelet/pods/ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c/volumes"
Sep 17 16:04:02.581676 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:02.581598    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84"]
Sep 17 16:04:02.582234 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:02.581823    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84" podUID="b85ecfe7-6d52-4877-898b-22614e00eb8a" containerName="maas-api" containerID="cri-o://e1cce7be627d879b0aca3111255d5bb131fcb1d7bfe4b2b7d4c9e6d00b3139e6" gracePeriod=30
Sep 17 16:04:02.838033 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:02.837970    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84"
Sep 17 16:04:02.878957 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:02.878927    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b85ecfe7-6d52-4877-898b-22614e00eb8a-metrics-certs\") pod \"b85ecfe7-6d52-4877-898b-22614e00eb8a\" (UID: \"b85ecfe7-6d52-4877-898b-22614e00eb8a\") "
Sep 17 16:04:02.879149 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:02.878978    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b85ecfe7-6d52-4877-898b-22614e00eb8a-maas-api-tls\") pod \"b85ecfe7-6d52-4877-898b-22614e00eb8a\" (UID: \"b85ecfe7-6d52-4877-898b-22614e00eb8a\") "
Sep 17 16:04:02.879149 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:02.879092    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85ecfe7-6d52-4877-898b-22614e00eb8a-openshift-service-ca\") pod \"b85ecfe7-6d52-4877-898b-22614e00eb8a\" (UID: \"b85ecfe7-6d52-4877-898b-22614e00eb8a\") "
Sep 17 16:04:02.879244 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:02.879148    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfdg5\" (UniqueName: \"kubernetes.io/projected/b85ecfe7-6d52-4877-898b-22614e00eb8a-kube-api-access-tfdg5\") pod \"b85ecfe7-6d52-4877-898b-22614e00eb8a\" (UID: \"b85ecfe7-6d52-4877-898b-22614e00eb8a\") "
Sep 17 16:04:02.879518 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:02.879493    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b85ecfe7-6d52-4877-898b-22614e00eb8a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b85ecfe7-6d52-4877-898b-22614e00eb8a" (UID: "b85ecfe7-6d52-4877-898b-22614e00eb8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:04:02.881146 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:02.881116    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b85ecfe7-6d52-4877-898b-22614e00eb8a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b85ecfe7-6d52-4877-898b-22614e00eb8a" (UID: "b85ecfe7-6d52-4877-898b-22614e00eb8a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:04:02.881262 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:02.881171    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b85ecfe7-6d52-4877-898b-22614e00eb8a-kube-api-access-tfdg5" (OuterVolumeSpecName: "kube-api-access-tfdg5") pod "b85ecfe7-6d52-4877-898b-22614e00eb8a" (UID: "b85ecfe7-6d52-4877-898b-22614e00eb8a"). InnerVolumeSpecName "kube-api-access-tfdg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:04:02.881262 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:02.881216    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b85ecfe7-6d52-4877-898b-22614e00eb8a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b85ecfe7-6d52-4877-898b-22614e00eb8a" (UID: "b85ecfe7-6d52-4877-898b-22614e00eb8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:04:02.980222 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:02.980187    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfdg5\" (UniqueName: \"kubernetes.io/projected/b85ecfe7-6d52-4877-898b-22614e00eb8a-kube-api-access-tfdg5\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 16:04:02.980222 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:02.980219    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b85ecfe7-6d52-4877-898b-22614e00eb8a-metrics-certs\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 16:04:02.980222 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:02.980229    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b85ecfe7-6d52-4877-898b-22614e00eb8a-maas-api-tls\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 16:04:02.980464 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:02.980238    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85ecfe7-6d52-4877-898b-22614e00eb8a-openshift-service-ca\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 16:04:03.619759 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:03.619721    2556 generic.go:356] "Generic (PLEG): container finished" podID="b85ecfe7-6d52-4877-898b-22614e00eb8a" containerID="e1cce7be627d879b0aca3111255d5bb131fcb1d7bfe4b2b7d4c9e6d00b3139e6" exitCode=0
Sep 17 16:04:03.620184 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:03.619797    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84" event={"ID":"b85ecfe7-6d52-4877-898b-22614e00eb8a","Type":"ContainerDied","Data":"e1cce7be627d879b0aca3111255d5bb131fcb1d7bfe4b2b7d4c9e6d00b3139e6"}
Sep 17 16:04:03.620184 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:03.619822    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84"
Sep 17 16:04:03.620184 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:03.619846    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84" event={"ID":"b85ecfe7-6d52-4877-898b-22614e00eb8a","Type":"ContainerDied","Data":"20ac4454146f7a965a4d616f5bab5013477ff07a3d5da2da2425d730ed86e361"}
Sep 17 16:04:03.620184 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:03.619868    2556 scope.go:117] "RemoveContainer" containerID="e1cce7be627d879b0aca3111255d5bb131fcb1d7bfe4b2b7d4c9e6d00b3139e6"
Sep 17 16:04:03.631966 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:03.631943    2556 scope.go:117] "RemoveContainer" containerID="e1cce7be627d879b0aca3111255d5bb131fcb1d7bfe4b2b7d4c9e6d00b3139e6"
Sep 17 16:04:03.632280 ip-10-0-134-104 kubenswrapper[2556]: E0917 16:04:03.632262    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1cce7be627d879b0aca3111255d5bb131fcb1d7bfe4b2b7d4c9e6d00b3139e6\": container with ID starting with e1cce7be627d879b0aca3111255d5bb131fcb1d7bfe4b2b7d4c9e6d00b3139e6 not found: ID does not exist" containerID="e1cce7be627d879b0aca3111255d5bb131fcb1d7bfe4b2b7d4c9e6d00b3139e6"
Sep 17 16:04:03.632352 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:03.632287    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1cce7be627d879b0aca3111255d5bb131fcb1d7bfe4b2b7d4c9e6d00b3139e6"} err="failed to get container status \"e1cce7be627d879b0aca3111255d5bb131fcb1d7bfe4b2b7d4c9e6d00b3139e6\": rpc error: code = NotFound desc = could not find container \"e1cce7be627d879b0aca3111255d5bb131fcb1d7bfe4b2b7d4c9e6d00b3139e6\": container with ID starting with e1cce7be627d879b0aca3111255d5bb131fcb1d7bfe4b2b7d4c9e6d00b3139e6 not found: ID does not exist"
Sep 17 16:04:03.663002 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:03.662956    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84"]
Sep 17 16:04:03.664757 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:03.664731    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d63c0-bd8d5f869-67l84"]
Sep 17 16:04:04.136309 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:04.136277    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b85ecfe7-6d52-4877-898b-22614e00eb8a" path="/var/lib/kubelet/pods/b85ecfe7-6d52-4877-898b-22614e00eb8a/volumes"
Sep 17 16:04:22.349778 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:22.349743    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72"]
Sep 17 16:04:22.350176 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:22.349992    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72" podUID="6b6920f1-5a16-4b44-a613-d067ee36c8bd" containerName="maas-api" containerID="cri-o://5e5303458ffcda4f67307b3824f08b070a9ed71a36ba06880279e61579f3516b" gracePeriod=30
Sep 17 16:04:22.602633 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:22.602569    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72"
Sep 17 16:04:22.659209 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:22.659172    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b6920f1-5a16-4b44-a613-d067ee36c8bd-openshift-service-ca\") pod \"6b6920f1-5a16-4b44-a613-d067ee36c8bd\" (UID: \"6b6920f1-5a16-4b44-a613-d067ee36c8bd\") "
Sep 17 16:04:22.659386 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:22.659250    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7x59h\" (UniqueName: \"kubernetes.io/projected/6b6920f1-5a16-4b44-a613-d067ee36c8bd-kube-api-access-7x59h\") pod \"6b6920f1-5a16-4b44-a613-d067ee36c8bd\" (UID: \"6b6920f1-5a16-4b44-a613-d067ee36c8bd\") "
Sep 17 16:04:22.659386 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:22.659280    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6920f1-5a16-4b44-a613-d067ee36c8bd-metrics-certs\") pod \"6b6920f1-5a16-4b44-a613-d067ee36c8bd\" (UID: \"6b6920f1-5a16-4b44-a613-d067ee36c8bd\") "
Sep 17 16:04:22.659386 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:22.659376    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6920f1-5a16-4b44-a613-d067ee36c8bd-maas-api-tls\") pod \"6b6920f1-5a16-4b44-a613-d067ee36c8bd\" (UID: \"6b6920f1-5a16-4b44-a613-d067ee36c8bd\") "
Sep 17 16:04:22.660008 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:22.659618    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b6920f1-5a16-4b44-a613-d067ee36c8bd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6b6920f1-5a16-4b44-a613-d067ee36c8bd" (UID: "6b6920f1-5a16-4b44-a613-d067ee36c8bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:04:22.661738 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:22.661712    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b6920f1-5a16-4b44-a613-d067ee36c8bd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6b6920f1-5a16-4b44-a613-d067ee36c8bd" (UID: "6b6920f1-5a16-4b44-a613-d067ee36c8bd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:04:22.661918 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:22.661897    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6920f1-5a16-4b44-a613-d067ee36c8bd-kube-api-access-7x59h" (OuterVolumeSpecName: "kube-api-access-7x59h") pod "6b6920f1-5a16-4b44-a613-d067ee36c8bd" (UID: "6b6920f1-5a16-4b44-a613-d067ee36c8bd"). InnerVolumeSpecName "kube-api-access-7x59h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:04:22.661962 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:22.661900    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b6920f1-5a16-4b44-a613-d067ee36c8bd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6b6920f1-5a16-4b44-a613-d067ee36c8bd" (UID: "6b6920f1-5a16-4b44-a613-d067ee36c8bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:04:22.717130 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:22.717099    2556 generic.go:356] "Generic (PLEG): container finished" podID="6b6920f1-5a16-4b44-a613-d067ee36c8bd" containerID="5e5303458ffcda4f67307b3824f08b070a9ed71a36ba06880279e61579f3516b" exitCode=0
Sep 17 16:04:22.717289 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:22.717163    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72"
Sep 17 16:04:22.717289 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:22.717185    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72" event={"ID":"6b6920f1-5a16-4b44-a613-d067ee36c8bd","Type":"ContainerDied","Data":"5e5303458ffcda4f67307b3824f08b070a9ed71a36ba06880279e61579f3516b"}
Sep 17 16:04:22.717289 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:22.717242    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72" event={"ID":"6b6920f1-5a16-4b44-a613-d067ee36c8bd","Type":"ContainerDied","Data":"62c31e66ecf1ec81ccacbb6f6880ea6d2d937881d3eb5cd756a7efd48ea4a2aa"}
Sep 17 16:04:22.717289 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:22.717264    2556 scope.go:117] "RemoveContainer" containerID="5e5303458ffcda4f67307b3824f08b070a9ed71a36ba06880279e61579f3516b"
Sep 17 16:04:22.726870 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:22.726853    2556 scope.go:117] "RemoveContainer" containerID="5e5303458ffcda4f67307b3824f08b070a9ed71a36ba06880279e61579f3516b"
Sep 17 16:04:22.727217 ip-10-0-134-104 kubenswrapper[2556]: E0917 16:04:22.727184    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e5303458ffcda4f67307b3824f08b070a9ed71a36ba06880279e61579f3516b\": container with ID starting with 5e5303458ffcda4f67307b3824f08b070a9ed71a36ba06880279e61579f3516b not found: ID does not exist" containerID="5e5303458ffcda4f67307b3824f08b070a9ed71a36ba06880279e61579f3516b"
Sep 17 16:04:22.727316 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:22.727227    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e5303458ffcda4f67307b3824f08b070a9ed71a36ba06880279e61579f3516b"} err="failed to get container status \"5e5303458ffcda4f67307b3824f08b070a9ed71a36ba06880279e61579f3516b\": rpc error: code = NotFound desc = could not find container \"5e5303458ffcda4f67307b3824f08b070a9ed71a36ba06880279e61579f3516b\": container with ID starting with 5e5303458ffcda4f67307b3824f08b070a9ed71a36ba06880279e61579f3516b not found: ID does not exist"
Sep 17 16:04:22.748522 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:22.748495    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72"]
Sep 17 16:04:22.753634 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:22.753613    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5818c9-6f5cbcfd5c-gdc72"]
Sep 17 16:04:22.760229 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:22.760208    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6920f1-5a16-4b44-a613-d067ee36c8bd-maas-api-tls\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 16:04:22.760229 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:22.760227    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b6920f1-5a16-4b44-a613-d067ee36c8bd-openshift-service-ca\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 16:04:22.760320 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:22.760237    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x59h\" (UniqueName: \"kubernetes.io/projected/6b6920f1-5a16-4b44-a613-d067ee36c8bd-kube-api-access-7x59h\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 16:04:22.760320 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:22.760247    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6920f1-5a16-4b44-a613-d067ee36c8bd-metrics-certs\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 16:04:24.136135 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:24.136098    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b6920f1-5a16-4b44-a613-d067ee36c8bd" path="/var/lib/kubelet/pods/6b6920f1-5a16-4b44-a613-d067ee36c8bd/volumes"
Sep 17 16:04:34.852440 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:34.852396    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b9f9c646d-lw4ph"]
Sep 17 16:04:34.852999 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:34.852600    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5b9f9c646d-lw4ph" podUID="d04bbc03-6252-4a48-80b1-26aaf6956426" containerName="manager" containerID="cri-o://fdfbf25f2d1880ef09a2b7a1168273ee76a7ffeb90d48c8c45723c8f5ca3c12b" gracePeriod=30
Sep 17 16:04:35.097116 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:35.097091    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9f9c646d-lw4ph"
Sep 17 16:04:35.177438 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:35.177358    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgr49\" (UniqueName: \"kubernetes.io/projected/d04bbc03-6252-4a48-80b1-26aaf6956426-kube-api-access-lgr49\") pod \"d04bbc03-6252-4a48-80b1-26aaf6956426\" (UID: \"d04bbc03-6252-4a48-80b1-26aaf6956426\") "
Sep 17 16:04:35.177605 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:35.177463    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d04bbc03-6252-4a48-80b1-26aaf6956426-metrics-certs\") pod \"d04bbc03-6252-4a48-80b1-26aaf6956426\" (UID: \"d04bbc03-6252-4a48-80b1-26aaf6956426\") "
Sep 17 16:04:35.177605 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:35.177481    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d04bbc03-6252-4a48-80b1-26aaf6956426-webhook-cert\") pod \"d04bbc03-6252-4a48-80b1-26aaf6956426\" (UID: \"d04bbc03-6252-4a48-80b1-26aaf6956426\") "
Sep 17 16:04:35.179608 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:35.179575    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d04bbc03-6252-4a48-80b1-26aaf6956426-kube-api-access-lgr49" (OuterVolumeSpecName: "kube-api-access-lgr49") pod "d04bbc03-6252-4a48-80b1-26aaf6956426" (UID: "d04bbc03-6252-4a48-80b1-26aaf6956426"). InnerVolumeSpecName "kube-api-access-lgr49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:04:35.179716 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:35.179601    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d04bbc03-6252-4a48-80b1-26aaf6956426-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d04bbc03-6252-4a48-80b1-26aaf6956426" (UID: "d04bbc03-6252-4a48-80b1-26aaf6956426"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:04:35.179716 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:35.179649    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d04bbc03-6252-4a48-80b1-26aaf6956426-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d04bbc03-6252-4a48-80b1-26aaf6956426" (UID: "d04bbc03-6252-4a48-80b1-26aaf6956426"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:04:35.279045 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:35.278999    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgr49\" (UniqueName: \"kubernetes.io/projected/d04bbc03-6252-4a48-80b1-26aaf6956426-kube-api-access-lgr49\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 16:04:35.279045 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:35.279035    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d04bbc03-6252-4a48-80b1-26aaf6956426-metrics-certs\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 16:04:35.279045 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:35.279049    2556 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d04bbc03-6252-4a48-80b1-26aaf6956426-webhook-cert\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 16:04:35.785537 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:35.785507    2556 generic.go:356] "Generic (PLEG): container finished" podID="d04bbc03-6252-4a48-80b1-26aaf6956426" containerID="fdfbf25f2d1880ef09a2b7a1168273ee76a7ffeb90d48c8c45723c8f5ca3c12b" exitCode=0
Sep 17 16:04:35.785731 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:35.785566    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9f9c646d-lw4ph"
Sep 17 16:04:35.785731 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:35.785587    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9f9c646d-lw4ph" event={"ID":"d04bbc03-6252-4a48-80b1-26aaf6956426","Type":"ContainerDied","Data":"fdfbf25f2d1880ef09a2b7a1168273ee76a7ffeb90d48c8c45723c8f5ca3c12b"}
Sep 17 16:04:35.785731 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:35.785653    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9f9c646d-lw4ph" event={"ID":"d04bbc03-6252-4a48-80b1-26aaf6956426","Type":"ContainerDied","Data":"d45c1ed255d4de52e5a866e74230c1f03342bc695b0848141d540f1fd437e5cf"}
Sep 17 16:04:35.785731 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:35.785682    2556 scope.go:117] "RemoveContainer" containerID="fdfbf25f2d1880ef09a2b7a1168273ee76a7ffeb90d48c8c45723c8f5ca3c12b"
Sep 17 16:04:35.795011 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:35.794994    2556 scope.go:117] "RemoveContainer" containerID="fdfbf25f2d1880ef09a2b7a1168273ee76a7ffeb90d48c8c45723c8f5ca3c12b"
Sep 17 16:04:35.795361 ip-10-0-134-104 kubenswrapper[2556]: E0917 16:04:35.795339    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdfbf25f2d1880ef09a2b7a1168273ee76a7ffeb90d48c8c45723c8f5ca3c12b\": container with ID starting with fdfbf25f2d1880ef09a2b7a1168273ee76a7ffeb90d48c8c45723c8f5ca3c12b not found: ID does not exist" containerID="fdfbf25f2d1880ef09a2b7a1168273ee76a7ffeb90d48c8c45723c8f5ca3c12b"
Sep 17 16:04:35.795441 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:35.795373    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdfbf25f2d1880ef09a2b7a1168273ee76a7ffeb90d48c8c45723c8f5ca3c12b"} err="failed to get container status \"fdfbf25f2d1880ef09a2b7a1168273ee76a7ffeb90d48c8c45723c8f5ca3c12b\": rpc error: code = NotFound desc = could not find container \"fdfbf25f2d1880ef09a2b7a1168273ee76a7ffeb90d48c8c45723c8f5ca3c12b\": container with ID starting with fdfbf25f2d1880ef09a2b7a1168273ee76a7ffeb90d48c8c45723c8f5ca3c12b not found: ID does not exist"
Sep 17 16:04:35.813963 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:35.813934    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b9f9c646d-lw4ph"]
Sep 17 16:04:35.818774 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:35.818747    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5b9f9c646d-lw4ph"]
Sep 17 16:04:36.135429 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:36.135399    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d04bbc03-6252-4a48-80b1-26aaf6956426" path="/var/lib/kubelet/pods/d04bbc03-6252-4a48-80b1-26aaf6956426/volumes"
Sep 17 16:04:38.237485 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:38.237456    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-45zt7_2c069c9e-0856-46b0-ba10-67829c05ac94/console-operator/1.log"
Sep 17 16:04:38.239302 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:04:38.239278    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-45zt7_2c069c9e-0856-46b0-ba10-67829c05ac94/console-operator/1.log"
Sep 17 16:05:36.751303 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:36.751220    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b9f9c646d-lttcj"]
Sep 17 16:05:36.751711 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:36.751595    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="846d4c21-2957-4679-826d-3ae4ed841928" containerName="maas-api"
Sep 17 16:05:36.751711 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:36.751606    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="846d4c21-2957-4679-826d-3ae4ed841928" containerName="maas-api"
Sep 17 16:05:36.751711 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:36.751616    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5011a14d-647a-49b1-bd42-75e36ef20ab3" containerName="maas-api"
Sep 17 16:05:36.751711 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:36.751623    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5011a14d-647a-49b1-bd42-75e36ef20ab3" containerName="maas-api"
Sep 17 16:05:36.751711 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:36.751636    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b6920f1-5a16-4b44-a613-d067ee36c8bd" containerName="maas-api"
Sep 17 16:05:36.751711 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:36.751641    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b6920f1-5a16-4b44-a613-d067ee36c8bd" containerName="maas-api"
Sep 17 16:05:36.751711 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:36.751649    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b85ecfe7-6d52-4877-898b-22614e00eb8a" containerName="maas-api"
Sep 17 16:05:36.751711 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:36.751654    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b85ecfe7-6d52-4877-898b-22614e00eb8a" containerName="maas-api"
Sep 17 16:05:36.751711 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:36.751662    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c" containerName="payload-processing"
Sep 17 16:05:36.751711 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:36.751667    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c" containerName="payload-processing"
Sep 17 16:05:36.751711 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:36.751677    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d04bbc03-6252-4a48-80b1-26aaf6956426" containerName="manager"
Sep 17 16:05:36.751711 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:36.751682    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d04bbc03-6252-4a48-80b1-26aaf6956426" containerName="manager"
Sep 17 16:05:36.752114 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:36.751737    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b6920f1-5a16-4b44-a613-d067ee36c8bd" containerName="maas-api"
Sep 17 16:05:36.752114 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:36.751746    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="846d4c21-2957-4679-826d-3ae4ed841928" containerName="maas-api"
Sep 17 16:05:36.752114 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:36.751753    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed7a40bc-f76e-4735-83ca-1a9cd3cbc72c" containerName="payload-processing"
Sep 17 16:05:36.752114 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:36.751759    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b85ecfe7-6d52-4877-898b-22614e00eb8a" containerName="maas-api"
Sep 17 16:05:36.752114 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:36.751767    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d04bbc03-6252-4a48-80b1-26aaf6956426" containerName="manager"
Sep 17 16:05:36.752114 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:36.751773    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5011a14d-647a-49b1-bd42-75e36ef20ab3" containerName="maas-api"
Sep 17 16:05:36.754819 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:36.754803    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9f9c646d-lttcj"
Sep 17 16:05:36.760105 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:36.760083    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 17 16:05:36.760428 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:36.760389    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mf6ws\""
Sep 17 16:05:36.760603 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:36.760577    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 17 16:05:36.760687 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:36.760607    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 17 16:05:36.775931 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:36.775905    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b9f9c646d-lttcj"]
Sep 17 16:05:36.916705 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:36.916661    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnth8\" (UniqueName: \"kubernetes.io/projected/4212d21e-e459-4d91-88fc-4a50f84f1844-kube-api-access-gnth8\") pod \"maas-controller-5b9f9c646d-lttcj\" (UID: \"4212d21e-e459-4d91-88fc-4a50f84f1844\") " pod="opendatahub/maas-controller-5b9f9c646d-lttcj"
Sep 17 16:05:36.916705 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:36.916711    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4212d21e-e459-4d91-88fc-4a50f84f1844-metrics-certs\") pod \"maas-controller-5b9f9c646d-lttcj\" (UID: \"4212d21e-e459-4d91-88fc-4a50f84f1844\") " pod="opendatahub/maas-controller-5b9f9c646d-lttcj"
Sep 17 16:05:36.916940 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:36.916778    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4212d21e-e459-4d91-88fc-4a50f84f1844-webhook-cert\") pod \"maas-controller-5b9f9c646d-lttcj\" (UID: \"4212d21e-e459-4d91-88fc-4a50f84f1844\") " pod="opendatahub/maas-controller-5b9f9c646d-lttcj"
Sep 17 16:05:37.017543 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:37.017458    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnth8\" (UniqueName: \"kubernetes.io/projected/4212d21e-e459-4d91-88fc-4a50f84f1844-kube-api-access-gnth8\") pod \"maas-controller-5b9f9c646d-lttcj\" (UID: \"4212d21e-e459-4d91-88fc-4a50f84f1844\") " pod="opendatahub/maas-controller-5b9f9c646d-lttcj"
Sep 17 16:05:37.017543 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:37.017502    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4212d21e-e459-4d91-88fc-4a50f84f1844-metrics-certs\") pod \"maas-controller-5b9f9c646d-lttcj\" (UID: \"4212d21e-e459-4d91-88fc-4a50f84f1844\") " pod="opendatahub/maas-controller-5b9f9c646d-lttcj"
Sep 17 16:05:37.017731 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:37.017550    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4212d21e-e459-4d91-88fc-4a50f84f1844-webhook-cert\") pod \"maas-controller-5b9f9c646d-lttcj\" (UID: \"4212d21e-e459-4d91-88fc-4a50f84f1844\") " pod="opendatahub/maas-controller-5b9f9c646d-lttcj"
Sep 17 16:05:37.020045 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:37.020012    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4212d21e-e459-4d91-88fc-4a50f84f1844-webhook-cert\") pod \"maas-controller-5b9f9c646d-lttcj\" (UID: \"4212d21e-e459-4d91-88fc-4a50f84f1844\") " pod="opendatahub/maas-controller-5b9f9c646d-lttcj"
Sep 17 16:05:37.020045 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:37.020043    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4212d21e-e459-4d91-88fc-4a50f84f1844-metrics-certs\") pod \"maas-controller-5b9f9c646d-lttcj\" (UID: \"4212d21e-e459-4d91-88fc-4a50f84f1844\") " pod="opendatahub/maas-controller-5b9f9c646d-lttcj"
Sep 17 16:05:37.030322 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:37.030300    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnth8\" (UniqueName: \"kubernetes.io/projected/4212d21e-e459-4d91-88fc-4a50f84f1844-kube-api-access-gnth8\") pod \"maas-controller-5b9f9c646d-lttcj\" (UID: \"4212d21e-e459-4d91-88fc-4a50f84f1844\") " pod="opendatahub/maas-controller-5b9f9c646d-lttcj"
Sep 17 16:05:37.065366 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:37.065328    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9f9c646d-lttcj"
Sep 17 16:05:37.350044 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:37.349952    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b9f9c646d-lttcj"]
Sep 17 16:05:37.353476 ip-10-0-134-104 kubenswrapper[2556]: W0917 16:05:37.353448    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4212d21e_e459_4d91_88fc_4a50f84f1844.slice/crio-d7c5e6148c8fdc5cc60b341d1e37a36368b4405aabc8185b1a20adfeb6564efc WatchSource:0}: Error finding container d7c5e6148c8fdc5cc60b341d1e37a36368b4405aabc8185b1a20adfeb6564efc: Status 404 returned error can't find the container with id d7c5e6148c8fdc5cc60b341d1e37a36368b4405aabc8185b1a20adfeb6564efc
Sep 17 16:05:37.354776 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:37.354757    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:05:38.099385 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:38.099343    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9f9c646d-lttcj" event={"ID":"4212d21e-e459-4d91-88fc-4a50f84f1844","Type":"ContainerStarted","Data":"62020ddaf5bfcf9c9ca34f2cc32f0edeae5d4b052c9f9b4ca897a52979da9059"}
Sep 17 16:05:38.099848 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:38.099438    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9f9c646d-lttcj" event={"ID":"4212d21e-e459-4d91-88fc-4a50f84f1844","Type":"ContainerStarted","Data":"d7c5e6148c8fdc5cc60b341d1e37a36368b4405aabc8185b1a20adfeb6564efc"}
Sep 17 16:05:38.099848 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:38.099462    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b9f9c646d-lttcj"
Sep 17 16:05:38.122988 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:38.122936    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b9f9c646d-lttcj" podStartSLOduration=1.549496634 podStartE2EDuration="2.122920895s" totalImagesPullingTime="573.424261ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:05:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:05:38.121177584 +0000 UTC m=+2160.501195340" watchObservedRunningTime="2026-09-17 16:05:38.122920895 +0000 UTC m=+2160.502938650"
Sep 17 16:05:49.106759 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:49.106726    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b9f9c646d-lttcj"
Sep 17 16:05:56.472215 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:56.472174    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b9f9c646d-lttcj"]
Sep 17 16:05:56.472699 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:56.472411    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5b9f9c646d-lttcj" podUID="4212d21e-e459-4d91-88fc-4a50f84f1844" containerName="manager" containerID="cri-o://62020ddaf5bfcf9c9ca34f2cc32f0edeae5d4b052c9f9b4ca897a52979da9059" gracePeriod=30
Sep 17 16:05:56.723474 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:56.723416    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9f9c646d-lttcj"
Sep 17 16:05:56.790045 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:56.790014    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4212d21e-e459-4d91-88fc-4a50f84f1844-webhook-cert\") pod \"4212d21e-e459-4d91-88fc-4a50f84f1844\" (UID: \"4212d21e-e459-4d91-88fc-4a50f84f1844\") "
Sep 17 16:05:56.790045 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:56.790053    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4212d21e-e459-4d91-88fc-4a50f84f1844-metrics-certs\") pod \"4212d21e-e459-4d91-88fc-4a50f84f1844\" (UID: \"4212d21e-e459-4d91-88fc-4a50f84f1844\") "
Sep 17 16:05:56.790258 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:56.790138    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnth8\" (UniqueName: \"kubernetes.io/projected/4212d21e-e459-4d91-88fc-4a50f84f1844-kube-api-access-gnth8\") pod \"4212d21e-e459-4d91-88fc-4a50f84f1844\" (UID: \"4212d21e-e459-4d91-88fc-4a50f84f1844\") "
Sep 17 16:05:56.792355 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:56.792322    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4212d21e-e459-4d91-88fc-4a50f84f1844-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "4212d21e-e459-4d91-88fc-4a50f84f1844" (UID: "4212d21e-e459-4d91-88fc-4a50f84f1844"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:05:56.792478 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:56.792333    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4212d21e-e459-4d91-88fc-4a50f84f1844-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4212d21e-e459-4d91-88fc-4a50f84f1844" (UID: "4212d21e-e459-4d91-88fc-4a50f84f1844"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:05:56.792478 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:56.792384    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4212d21e-e459-4d91-88fc-4a50f84f1844-kube-api-access-gnth8" (OuterVolumeSpecName: "kube-api-access-gnth8") pod "4212d21e-e459-4d91-88fc-4a50f84f1844" (UID: "4212d21e-e459-4d91-88fc-4a50f84f1844"). InnerVolumeSpecName "kube-api-access-gnth8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:05:56.891094 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:56.891025    2556 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4212d21e-e459-4d91-88fc-4a50f84f1844-webhook-cert\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 16:05:56.891094 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:56.891087    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4212d21e-e459-4d91-88fc-4a50f84f1844-metrics-certs\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 16:05:56.891094 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:56.891098    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnth8\" (UniqueName: \"kubernetes.io/projected/4212d21e-e459-4d91-88fc-4a50f84f1844-kube-api-access-gnth8\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 16:05:57.190163 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:57.190128    2556 generic.go:356] "Generic (PLEG): container finished" podID="4212d21e-e459-4d91-88fc-4a50f84f1844" containerID="62020ddaf5bfcf9c9ca34f2cc32f0edeae5d4b052c9f9b4ca897a52979da9059" exitCode=0
Sep 17 16:05:57.190381 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:57.190177    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9f9c646d-lttcj"
Sep 17 16:05:57.190381 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:57.190213    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9f9c646d-lttcj" event={"ID":"4212d21e-e459-4d91-88fc-4a50f84f1844","Type":"ContainerDied","Data":"62020ddaf5bfcf9c9ca34f2cc32f0edeae5d4b052c9f9b4ca897a52979da9059"}
Sep 17 16:05:57.190381 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:57.190264    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9f9c646d-lttcj" event={"ID":"4212d21e-e459-4d91-88fc-4a50f84f1844","Type":"ContainerDied","Data":"d7c5e6148c8fdc5cc60b341d1e37a36368b4405aabc8185b1a20adfeb6564efc"}
Sep 17 16:05:57.190381 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:57.190287    2556 scope.go:117] "RemoveContainer" containerID="62020ddaf5bfcf9c9ca34f2cc32f0edeae5d4b052c9f9b4ca897a52979da9059"
Sep 17 16:05:57.199681 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:57.199663    2556 scope.go:117] "RemoveContainer" containerID="62020ddaf5bfcf9c9ca34f2cc32f0edeae5d4b052c9f9b4ca897a52979da9059"
Sep 17 16:05:57.199972 ip-10-0-134-104 kubenswrapper[2556]: E0917 16:05:57.199947    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62020ddaf5bfcf9c9ca34f2cc32f0edeae5d4b052c9f9b4ca897a52979da9059\": container with ID starting with 62020ddaf5bfcf9c9ca34f2cc32f0edeae5d4b052c9f9b4ca897a52979da9059 not found: ID does not exist" containerID="62020ddaf5bfcf9c9ca34f2cc32f0edeae5d4b052c9f9b4ca897a52979da9059"
Sep 17 16:05:57.200040 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:57.199979    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62020ddaf5bfcf9c9ca34f2cc32f0edeae5d4b052c9f9b4ca897a52979da9059"} err="failed to get container status \"62020ddaf5bfcf9c9ca34f2cc32f0edeae5d4b052c9f9b4ca897a52979da9059\": rpc error: code = NotFound desc = could not find container \"62020ddaf5bfcf9c9ca34f2cc32f0edeae5d4b052c9f9b4ca897a52979da9059\": container with ID starting with 62020ddaf5bfcf9c9ca34f2cc32f0edeae5d4b052c9f9b4ca897a52979da9059 not found: ID does not exist"
Sep 17 16:05:57.214107 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:57.214073    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b9f9c646d-lttcj"]
Sep 17 16:05:57.220227 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:57.220197    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5b9f9c646d-lttcj"]
Sep 17 16:05:58.066929 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:58.066893    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b9f9c646d-w244f"]
Sep 17 16:05:58.067347 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:58.067301    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4212d21e-e459-4d91-88fc-4a50f84f1844" containerName="manager"
Sep 17 16:05:58.067347 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:58.067315    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4212d21e-e459-4d91-88fc-4a50f84f1844" containerName="manager"
Sep 17 16:05:58.067422 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:58.067386    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4212d21e-e459-4d91-88fc-4a50f84f1844" containerName="manager"
Sep 17 16:05:58.071902 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:58.071882    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9f9c646d-w244f"
Sep 17 16:05:58.076200 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:58.076176    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mf6ws\""
Sep 17 16:05:58.076200 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:58.076199    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 17 16:05:58.076444 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:58.076199    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 17 16:05:58.076444 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:58.076176    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 17 16:05:58.096020 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:58.095989    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b9f9c646d-w244f"]
Sep 17 16:05:58.103198 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:58.103160    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhrfm\" (UniqueName: \"kubernetes.io/projected/f5028686-ff8b-44ba-9bdb-59de4c69aa12-kube-api-access-hhrfm\") pod \"maas-controller-5b9f9c646d-w244f\" (UID: \"f5028686-ff8b-44ba-9bdb-59de4c69aa12\") " pod="opendatahub/maas-controller-5b9f9c646d-w244f"
Sep 17 16:05:58.103331 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:58.103222    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f5028686-ff8b-44ba-9bdb-59de4c69aa12-webhook-cert\") pod \"maas-controller-5b9f9c646d-w244f\" (UID: \"f5028686-ff8b-44ba-9bdb-59de4c69aa12\") " pod="opendatahub/maas-controller-5b9f9c646d-w244f"
Sep 17 16:05:58.103401 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:58.103375    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5028686-ff8b-44ba-9bdb-59de4c69aa12-metrics-certs\") pod \"maas-controller-5b9f9c646d-w244f\" (UID: \"f5028686-ff8b-44ba-9bdb-59de4c69aa12\") " pod="opendatahub/maas-controller-5b9f9c646d-w244f"
Sep 17 16:05:58.136250 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:58.136216    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4212d21e-e459-4d91-88fc-4a50f84f1844" path="/var/lib/kubelet/pods/4212d21e-e459-4d91-88fc-4a50f84f1844/volumes"
Sep 17 16:05:58.204392 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:58.204361    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhrfm\" (UniqueName: \"kubernetes.io/projected/f5028686-ff8b-44ba-9bdb-59de4c69aa12-kube-api-access-hhrfm\") pod \"maas-controller-5b9f9c646d-w244f\" (UID: \"f5028686-ff8b-44ba-9bdb-59de4c69aa12\") " pod="opendatahub/maas-controller-5b9f9c646d-w244f"
Sep 17 16:05:58.204540 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:58.204399    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f5028686-ff8b-44ba-9bdb-59de4c69aa12-webhook-cert\") pod \"maas-controller-5b9f9c646d-w244f\" (UID: \"f5028686-ff8b-44ba-9bdb-59de4c69aa12\") " pod="opendatahub/maas-controller-5b9f9c646d-w244f"
Sep 17 16:05:58.204540 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:58.204463    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5028686-ff8b-44ba-9bdb-59de4c69aa12-metrics-certs\") pod \"maas-controller-5b9f9c646d-w244f\" (UID: \"f5028686-ff8b-44ba-9bdb-59de4c69aa12\") " pod="opendatahub/maas-controller-5b9f9c646d-w244f"
Sep 17 16:05:58.206913 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:58.206886    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f5028686-ff8b-44ba-9bdb-59de4c69aa12-webhook-cert\") pod \"maas-controller-5b9f9c646d-w244f\" (UID: \"f5028686-ff8b-44ba-9bdb-59de4c69aa12\") " pod="opendatahub/maas-controller-5b9f9c646d-w244f"
Sep 17 16:05:58.207018 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:58.206888    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5028686-ff8b-44ba-9bdb-59de4c69aa12-metrics-certs\") pod \"maas-controller-5b9f9c646d-w244f\" (UID: \"f5028686-ff8b-44ba-9bdb-59de4c69aa12\") " pod="opendatahub/maas-controller-5b9f9c646d-w244f"
Sep 17 16:05:58.218245 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:58.218215    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhrfm\" (UniqueName: \"kubernetes.io/projected/f5028686-ff8b-44ba-9bdb-59de4c69aa12-kube-api-access-hhrfm\") pod \"maas-controller-5b9f9c646d-w244f\" (UID: \"f5028686-ff8b-44ba-9bdb-59de4c69aa12\") " pod="opendatahub/maas-controller-5b9f9c646d-w244f"
Sep 17 16:05:58.388550 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:58.388506    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9f9c646d-w244f"
Sep 17 16:05:58.669775 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:58.669748    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b9f9c646d-w244f"]
Sep 17 16:05:58.672268 ip-10-0-134-104 kubenswrapper[2556]: W0917 16:05:58.672237    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5028686_ff8b_44ba_9bdb_59de4c69aa12.slice/crio-350c0db0ba23bca6d4fe36e46c3ca86ef92a191379eedac88885ac38097b5900 WatchSource:0}: Error finding container 350c0db0ba23bca6d4fe36e46c3ca86ef92a191379eedac88885ac38097b5900: Status 404 returned error can't find the container with id 350c0db0ba23bca6d4fe36e46c3ca86ef92a191379eedac88885ac38097b5900
Sep 17 16:05:59.202918 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:05:59.202886    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9f9c646d-w244f" event={"ID":"f5028686-ff8b-44ba-9bdb-59de4c69aa12","Type":"ContainerStarted","Data":"350c0db0ba23bca6d4fe36e46c3ca86ef92a191379eedac88885ac38097b5900"}
Sep 17 16:06:00.210259 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:06:00.210223    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9f9c646d-w244f" event={"ID":"f5028686-ff8b-44ba-9bdb-59de4c69aa12","Type":"ContainerStarted","Data":"dd514f3508f17c808cab416fb35e7fe83f2215005524a3cb25268f15e8589c4d"}
Sep 17 16:06:00.210827 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:06:00.210346    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b9f9c646d-w244f"
Sep 17 16:06:00.240446 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:06:00.240391    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b9f9c646d-w244f" podStartSLOduration=1.750842944 podStartE2EDuration="2.240375428s" totalImagesPullingTime="489.532484ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:05:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:06:00.239655178 +0000 UTC m=+2182.619672934" watchObservedRunningTime="2026-09-17 16:06:00.240375428 +0000 UTC m=+2182.620393183"
Sep 17 16:06:11.218667 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:06:11.218629    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b9f9c646d-w244f"
Sep 17 16:09:38.267408 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:09:38.267376    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-45zt7_2c069c9e-0856-46b0-ba10-67829c05ac94/console-operator/1.log"
Sep 17 16:09:38.270302 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:09:38.270279    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-45zt7_2c069c9e-0856-46b0-ba10-67829c05ac94/console-operator/1.log"
Sep 17 16:10:10.611605 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:10.611568    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7698c4fdd9-snbnt"]
Sep 17 16:10:10.616072 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:10.616039    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-snbnt"
Sep 17 16:10:10.626924 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:10.626898    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7698c4fdd9-snbnt"]
Sep 17 16:10:10.661693 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:10.661652    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dee9aa02-de3f-4d37-bf0a-5902eb06bdd4-maas-api-tls\") pod \"maas-api-7698c4fdd9-snbnt\" (UID: \"dee9aa02-de3f-4d37-bf0a-5902eb06bdd4\") " pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-snbnt"
Sep 17 16:10:10.661693 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:10.661696    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqs2m\" (UniqueName: \"kubernetes.io/projected/dee9aa02-de3f-4d37-bf0a-5902eb06bdd4-kube-api-access-pqs2m\") pod \"maas-api-7698c4fdd9-snbnt\" (UID: \"dee9aa02-de3f-4d37-bf0a-5902eb06bdd4\") " pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-snbnt"
Sep 17 16:10:10.661949 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:10.661848    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dee9aa02-de3f-4d37-bf0a-5902eb06bdd4-openshift-service-ca\") pod \"maas-api-7698c4fdd9-snbnt\" (UID: \"dee9aa02-de3f-4d37-bf0a-5902eb06bdd4\") " pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-snbnt"
Sep 17 16:10:10.661992 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:10.661940    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dee9aa02-de3f-4d37-bf0a-5902eb06bdd4-metrics-certs\") pod \"maas-api-7698c4fdd9-snbnt\" (UID: \"dee9aa02-de3f-4d37-bf0a-5902eb06bdd4\") " pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-snbnt"
Sep 17 16:10:10.763154 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:10.763116    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dee9aa02-de3f-4d37-bf0a-5902eb06bdd4-openshift-service-ca\") pod \"maas-api-7698c4fdd9-snbnt\" (UID: \"dee9aa02-de3f-4d37-bf0a-5902eb06bdd4\") " pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-snbnt"
Sep 17 16:10:10.763361 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:10.763176    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dee9aa02-de3f-4d37-bf0a-5902eb06bdd4-metrics-certs\") pod \"maas-api-7698c4fdd9-snbnt\" (UID: \"dee9aa02-de3f-4d37-bf0a-5902eb06bdd4\") " pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-snbnt"
Sep 17 16:10:10.763361 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:10.763199    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dee9aa02-de3f-4d37-bf0a-5902eb06bdd4-maas-api-tls\") pod \"maas-api-7698c4fdd9-snbnt\" (UID: \"dee9aa02-de3f-4d37-bf0a-5902eb06bdd4\") " pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-snbnt"
Sep 17 16:10:10.763361 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:10.763223    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqs2m\" (UniqueName: \"kubernetes.io/projected/dee9aa02-de3f-4d37-bf0a-5902eb06bdd4-kube-api-access-pqs2m\") pod \"maas-api-7698c4fdd9-snbnt\" (UID: \"dee9aa02-de3f-4d37-bf0a-5902eb06bdd4\") " pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-snbnt"
Sep 17 16:10:10.763782 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:10.763758    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dee9aa02-de3f-4d37-bf0a-5902eb06bdd4-openshift-service-ca\") pod \"maas-api-7698c4fdd9-snbnt\" (UID: \"dee9aa02-de3f-4d37-bf0a-5902eb06bdd4\") " pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-snbnt"
Sep 17 16:10:10.765828 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:10.765802    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dee9aa02-de3f-4d37-bf0a-5902eb06bdd4-maas-api-tls\") pod \"maas-api-7698c4fdd9-snbnt\" (UID: \"dee9aa02-de3f-4d37-bf0a-5902eb06bdd4\") " pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-snbnt"
Sep 17 16:10:10.765929 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:10.765802    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dee9aa02-de3f-4d37-bf0a-5902eb06bdd4-metrics-certs\") pod \"maas-api-7698c4fdd9-snbnt\" (UID: \"dee9aa02-de3f-4d37-bf0a-5902eb06bdd4\") " pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-snbnt"
Sep 17 16:10:10.772468 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:10.772450    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqs2m\" (UniqueName: \"kubernetes.io/projected/dee9aa02-de3f-4d37-bf0a-5902eb06bdd4-kube-api-access-pqs2m\") pod \"maas-api-7698c4fdd9-snbnt\" (UID: \"dee9aa02-de3f-4d37-bf0a-5902eb06bdd4\") " pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-snbnt"
Sep 17 16:10:10.928203 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:10.928110    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-snbnt"
Sep 17 16:10:11.212550 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:11.212509    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7698c4fdd9-snbnt"]
Sep 17 16:10:11.214963 ip-10-0-134-104 kubenswrapper[2556]: W0917 16:10:11.214934    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee9aa02_de3f_4d37_bf0a_5902eb06bdd4.slice/crio-f1c857972d35d32b60507264729693082c8a50b8c3433b61d0dd5719d794925e WatchSource:0}: Error finding container f1c857972d35d32b60507264729693082c8a50b8c3433b61d0dd5719d794925e: Status 404 returned error can't find the container with id f1c857972d35d32b60507264729693082c8a50b8c3433b61d0dd5719d794925e
Sep 17 16:10:11.436233 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:11.436199    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-snbnt" event={"ID":"dee9aa02-de3f-4d37-bf0a-5902eb06bdd4","Type":"ContainerStarted","Data":"f1c857972d35d32b60507264729693082c8a50b8c3433b61d0dd5719d794925e"}
Sep 17 16:10:12.443397 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:12.443360    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-snbnt" event={"ID":"dee9aa02-de3f-4d37-bf0a-5902eb06bdd4","Type":"ContainerStarted","Data":"86fcc03107348aacb217429fde20a2a65c88a45da59cf96c75cd3f725b1d38f6"}
Sep 17 16:10:12.443835 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:12.443488    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-snbnt"
Sep 17 16:10:18.454605 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:18.454575    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-snbnt"
Sep 17 16:10:18.482001 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:18.481946    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-snbnt" podStartSLOduration=7.920020662 podStartE2EDuration="8.481928952s" totalImagesPullingTime="561.90829ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:10:12.468439301 +0000 UTC m=+2434.848457057" watchObservedRunningTime="2026-09-17 16:10:18.481928952 +0000 UTC m=+2440.861946802"
Sep 17 16:10:18.544950 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:18.544906    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh"]
Sep 17 16:10:18.545195 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:18.545171    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh" podUID="ca972d44-9d36-420c-a530-aa9d6add83ac" containerName="maas-api" containerID="cri-o://27b9efe78d0ec174bea518226d8b6416fe4012b8e48a1da4fbf056a710e21474" gracePeriod=30
Sep 17 16:10:19.803317 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:19.803287    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh"
Sep 17 16:10:19.848717 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:19.848681    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca972d44-9d36-420c-a530-aa9d6add83ac-metrics-certs\") pod \"ca972d44-9d36-420c-a530-aa9d6add83ac\" (UID: \"ca972d44-9d36-420c-a530-aa9d6add83ac\") "
Sep 17 16:10:19.848913 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:19.848769    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca972d44-9d36-420c-a530-aa9d6add83ac-maas-api-tls\") pod \"ca972d44-9d36-420c-a530-aa9d6add83ac\" (UID: \"ca972d44-9d36-420c-a530-aa9d6add83ac\") "
Sep 17 16:10:19.848913 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:19.848859    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca972d44-9d36-420c-a530-aa9d6add83ac-openshift-service-ca\") pod \"ca972d44-9d36-420c-a530-aa9d6add83ac\" (UID: \"ca972d44-9d36-420c-a530-aa9d6add83ac\") "
Sep 17 16:10:19.849051 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:19.848971    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pf4nt\" (UniqueName: \"kubernetes.io/projected/ca972d44-9d36-420c-a530-aa9d6add83ac-kube-api-access-pf4nt\") pod \"ca972d44-9d36-420c-a530-aa9d6add83ac\" (UID: \"ca972d44-9d36-420c-a530-aa9d6add83ac\") "
Sep 17 16:10:19.849324 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:19.849289    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca972d44-9d36-420c-a530-aa9d6add83ac-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ca972d44-9d36-420c-a530-aa9d6add83ac" (UID: "ca972d44-9d36-420c-a530-aa9d6add83ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:10:19.850964 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:19.850940    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca972d44-9d36-420c-a530-aa9d6add83ac-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ca972d44-9d36-420c-a530-aa9d6add83ac" (UID: "ca972d44-9d36-420c-a530-aa9d6add83ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:10:19.851095 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:19.850988    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca972d44-9d36-420c-a530-aa9d6add83ac-kube-api-access-pf4nt" (OuterVolumeSpecName: "kube-api-access-pf4nt") pod "ca972d44-9d36-420c-a530-aa9d6add83ac" (UID: "ca972d44-9d36-420c-a530-aa9d6add83ac"). InnerVolumeSpecName "kube-api-access-pf4nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:10:19.851095 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:19.851043    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca972d44-9d36-420c-a530-aa9d6add83ac-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ca972d44-9d36-420c-a530-aa9d6add83ac" (UID: "ca972d44-9d36-420c-a530-aa9d6add83ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:10:19.950788 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:19.950671    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca972d44-9d36-420c-a530-aa9d6add83ac-metrics-certs\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 16:10:19.950788 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:19.950719    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca972d44-9d36-420c-a530-aa9d6add83ac-maas-api-tls\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 16:10:19.950788 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:19.950732    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca972d44-9d36-420c-a530-aa9d6add83ac-openshift-service-ca\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 16:10:19.950788 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:19.950747    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf4nt\" (UniqueName: \"kubernetes.io/projected/ca972d44-9d36-420c-a530-aa9d6add83ac-kube-api-access-pf4nt\") on node \"ip-10-0-134-104.ec2.internal\" DevicePath \"\""
Sep 17 16:10:20.492214 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:20.492176    2556 generic.go:356] "Generic (PLEG): container finished" podID="ca972d44-9d36-420c-a530-aa9d6add83ac" containerID="27b9efe78d0ec174bea518226d8b6416fe4012b8e48a1da4fbf056a710e21474" exitCode=0
Sep 17 16:10:20.492367 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:20.492229    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh"
Sep 17 16:10:20.492367 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:20.492235    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh" event={"ID":"ca972d44-9d36-420c-a530-aa9d6add83ac","Type":"ContainerDied","Data":"27b9efe78d0ec174bea518226d8b6416fe4012b8e48a1da4fbf056a710e21474"}
Sep 17 16:10:20.492367 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:20.492358    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh" event={"ID":"ca972d44-9d36-420c-a530-aa9d6add83ac","Type":"ContainerDied","Data":"73e8cbcb5e775921be153009f05daccbb6b9e27ab7f4bc8badaaab4438fb4a38"}
Sep 17 16:10:20.492512 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:20.492380    2556 scope.go:117] "RemoveContainer" containerID="27b9efe78d0ec174bea518226d8b6416fe4012b8e48a1da4fbf056a710e21474"
Sep 17 16:10:20.501429 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:20.501399    2556 scope.go:117] "RemoveContainer" containerID="27b9efe78d0ec174bea518226d8b6416fe4012b8e48a1da4fbf056a710e21474"
Sep 17 16:10:20.501754 ip-10-0-134-104 kubenswrapper[2556]: E0917 16:10:20.501731    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27b9efe78d0ec174bea518226d8b6416fe4012b8e48a1da4fbf056a710e21474\": container with ID starting with 27b9efe78d0ec174bea518226d8b6416fe4012b8e48a1da4fbf056a710e21474 not found: ID does not exist" containerID="27b9efe78d0ec174bea518226d8b6416fe4012b8e48a1da4fbf056a710e21474"
Sep 17 16:10:20.501851 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:20.501758    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27b9efe78d0ec174bea518226d8b6416fe4012b8e48a1da4fbf056a710e21474"} err="failed to get container status \"27b9efe78d0ec174bea518226d8b6416fe4012b8e48a1da4fbf056a710e21474\": rpc error: code = NotFound desc = could not find container \"27b9efe78d0ec174bea518226d8b6416fe4012b8e48a1da4fbf056a710e21474\": container with ID starting with 27b9efe78d0ec174bea518226d8b6416fe4012b8e48a1da4fbf056a710e21474 not found: ID does not exist"
Sep 17 16:10:20.514274 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:20.514240    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh"]
Sep 17 16:10:20.517145 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:20.517120    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-d5fdc6457-2swjh"]
Sep 17 16:10:22.135821 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:22.135791    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca972d44-9d36-420c-a530-aa9d6add83ac" path="/var/lib/kubelet/pods/ca972d44-9d36-420c-a530-aa9d6add83ac/volumes"
Sep 17 16:10:38.469023 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:38.468996    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-lq9sm_89181401-93fb-4dac-8941-57ebe347292a/manager/0.log"
Sep 17 16:10:38.618025 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:38.617996    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b9f9c646d-w244f_f5028686-ff8b-44ba-9bdb-59de4c69aa12/manager/0.log"
Sep 17 16:10:38.746580 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:38.746492    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-4g2kw_693bf480-ee10-4366-9335-ef0c2651f980/server/0.log"
Sep 17 16:10:39.020343 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:39.020262    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5df9996869-86l8g_fa8daaea-084c-4cb6-9c92-2b5a61aac914/copy-manifests/0.log"
Sep 17 16:10:39.035640 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:39.035609    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5df9996869-86l8g_fa8daaea-084c-4cb6-9c92-2b5a61aac914/manager/0.log"
Sep 17 16:10:39.799487 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:39.799457    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7698c4fdd9-snbnt_dee9aa02-de3f-4d37-bf0a-5902eb06bdd4/maas-api/0.log"
Sep 17 16:10:40.182643 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:40.182612    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-2mqcs_9ba42d1c-e8db-4178-a9db-b6b04b961f72/postgres/0.log"
Sep 17 16:10:41.250843 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:41.250815    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-s45bq_e8a4c613-d01f-42ed-b771-0744a7e9f2e9/manager/0.log"
Sep 17 16:10:42.814681 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:42.814645    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-btzhg_2ec252e5-1112-40e2-b579-966af538ac4f/payload-pre-processing/0.log"
Sep 17 16:10:42.941435 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:42.941403    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-2wlv5_5bd2083c-fee3-4da3-95dd-1be423dab576/payload-processing/0.log"
Sep 17 16:10:43.068732 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:43.068656    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85cf78879b-z2qnj_8b73fab2-e530-4b68-acf0-c71d1943cb7c/router/0.log"
Sep 17 16:10:43.452915 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:43.452882    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-mscr4_ebc729c3-e744-442a-afef-a32261d00baf/main/0.log"
Sep 17 16:10:43.825093 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:43.824993    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-sdz2t_7e21c1f2-ff9e-4eaf-85ef-fe9fbd40860f/main/0.log"
Sep 17 16:10:43.949367 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:43.949338    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fghhcd_cd6f7039-09e8-4a44-9fe9-212ee9d8b772/main/0.log"
Sep 17 16:10:44.066869 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:44.066838    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-csnjg_0f96f508-5ac7-42f1-ab3a-d6d2ab6e24ce/main/0.log"
Sep 17 16:10:53.700986 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:53.700958    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-r5jgb_84728120-5cde-4b79-944d-7554db40cf41/global-pull-secret-syncer/0.log"
Sep 17 16:10:53.813953 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:53.813917    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nh9lk_69c5bd0b-4c06-45dc-9b1b-a9af91086fcd/konnectivity-agent/0.log"
Sep 17 16:10:53.953915 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:53.953836    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-104.ec2.internal_4965a8a85b93317838f07968164cf5cd/haproxy/0.log"
Sep 17 16:10:58.299982 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:10:58.299957    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-s45bq_e8a4c613-d01f-42ed-b771-0744a7e9f2e9/manager/0.log"
Sep 17 16:11:00.510132 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:00.510103    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-t58k8_8107c705-1f01-4194-bcde-679259e3ea51/cluster-monitoring-operator/0.log"
Sep 17 16:11:00.847436 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:00.847343    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kprs2_9ae480d7-2e41-41e3-9923-656e5e9e82ae/node-exporter/0.log"
Sep 17 16:11:00.880370 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:00.880341    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kprs2_9ae480d7-2e41-41e3-9923-656e5e9e82ae/kube-rbac-proxy/0.log"
Sep 17 16:11:00.906925 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:00.906899    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kprs2_9ae480d7-2e41-41e3-9923-656e5e9e82ae/init-textfile/0.log"
Sep 17 16:11:03.226698 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:03.226666    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-65bdt_12788045-cf19-4320-8007-927007d2eb7c/networking-console-plugin/0.log"
Sep 17 16:11:03.609948 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:03.609916    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-25lkm/perf-node-gather-daemonset-csqmq"]
Sep 17 16:11:03.610537 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:03.610519    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca972d44-9d36-420c-a530-aa9d6add83ac" containerName="maas-api"
Sep 17 16:11:03.610587 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:03.610541    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca972d44-9d36-420c-a530-aa9d6add83ac" containerName="maas-api"
Sep 17 16:11:03.610653 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:03.610643    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca972d44-9d36-420c-a530-aa9d6add83ac" containerName="maas-api"
Sep 17 16:11:03.614178 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:03.614156    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-csqmq"
Sep 17 16:11:03.616873 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:03.616846    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-25lkm\"/\"openshift-service-ca.crt\""
Sep 17 16:11:03.617379 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:03.617356    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-25lkm\"/\"default-dockercfg-bmf5g\""
Sep 17 16:11:03.617486 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:03.617366    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-25lkm\"/\"kube-root-ca.crt\""
Sep 17 16:11:03.632848 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:03.632813    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-25lkm/perf-node-gather-daemonset-csqmq"]
Sep 17 16:11:03.638485 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:03.638451    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gt58\" (UniqueName: \"kubernetes.io/projected/b256234b-64f7-4d2f-9403-e84aaf928e35-kube-api-access-4gt58\") pod \"perf-node-gather-daemonset-csqmq\" (UID: \"b256234b-64f7-4d2f-9403-e84aaf928e35\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-csqmq"
Sep 17 16:11:03.638615 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:03.638536    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b256234b-64f7-4d2f-9403-e84aaf928e35-sys\") pod \"perf-node-gather-daemonset-csqmq\" (UID: \"b256234b-64f7-4d2f-9403-e84aaf928e35\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-csqmq"
Sep 17 16:11:03.638615 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:03.638593    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b256234b-64f7-4d2f-9403-e84aaf928e35-podres\") pod \"perf-node-gather-daemonset-csqmq\" (UID: \"b256234b-64f7-4d2f-9403-e84aaf928e35\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-csqmq"
Sep 17 16:11:03.638740 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:03.638661    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b256234b-64f7-4d2f-9403-e84aaf928e35-lib-modules\") pod \"perf-node-gather-daemonset-csqmq\" (UID: \"b256234b-64f7-4d2f-9403-e84aaf928e35\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-csqmq"
Sep 17 16:11:03.638740 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:03.638723    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b256234b-64f7-4d2f-9403-e84aaf928e35-proc\") pod \"perf-node-gather-daemonset-csqmq\" (UID: \"b256234b-64f7-4d2f-9403-e84aaf928e35\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-csqmq"
Sep 17 16:11:03.739221 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:03.739184    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b256234b-64f7-4d2f-9403-e84aaf928e35-proc\") pod \"perf-node-gather-daemonset-csqmq\" (UID: \"b256234b-64f7-4d2f-9403-e84aaf928e35\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-csqmq"
Sep 17 16:11:03.739424 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:03.739247    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gt58\" (UniqueName: \"kubernetes.io/projected/b256234b-64f7-4d2f-9403-e84aaf928e35-kube-api-access-4gt58\") pod \"perf-node-gather-daemonset-csqmq\" (UID: \"b256234b-64f7-4d2f-9403-e84aaf928e35\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-csqmq"
Sep 17 16:11:03.739424 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:03.739316    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b256234b-64f7-4d2f-9403-e84aaf928e35-proc\") pod \"perf-node-gather-daemonset-csqmq\" (UID: \"b256234b-64f7-4d2f-9403-e84aaf928e35\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-csqmq"
Sep 17 16:11:03.739424 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:03.739323    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b256234b-64f7-4d2f-9403-e84aaf928e35-sys\") pod \"perf-node-gather-daemonset-csqmq\" (UID: \"b256234b-64f7-4d2f-9403-e84aaf928e35\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-csqmq"
Sep 17 16:11:03.739424 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:03.739372    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b256234b-64f7-4d2f-9403-e84aaf928e35-sys\") pod \"perf-node-gather-daemonset-csqmq\" (UID: \"b256234b-64f7-4d2f-9403-e84aaf928e35\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-csqmq"
Sep 17 16:11:03.739424 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:03.739397    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b256234b-64f7-4d2f-9403-e84aaf928e35-podres\") pod \"perf-node-gather-daemonset-csqmq\" (UID: \"b256234b-64f7-4d2f-9403-e84aaf928e35\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-csqmq"
Sep 17 16:11:03.739424 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:03.739424    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b256234b-64f7-4d2f-9403-e84aaf928e35-lib-modules\") pod \"perf-node-gather-daemonset-csqmq\" (UID: \"b256234b-64f7-4d2f-9403-e84aaf928e35\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-csqmq"
Sep 17 16:11:03.739659 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:03.739552    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b256234b-64f7-4d2f-9403-e84aaf928e35-lib-modules\") pod \"perf-node-gather-daemonset-csqmq\" (UID: \"b256234b-64f7-4d2f-9403-e84aaf928e35\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-csqmq"
Sep 17 16:11:03.739659 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:03.739552    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b256234b-64f7-4d2f-9403-e84aaf928e35-podres\") pod \"perf-node-gather-daemonset-csqmq\" (UID: \"b256234b-64f7-4d2f-9403-e84aaf928e35\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-csqmq"
Sep 17 16:11:03.749257 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:03.749231    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gt58\" (UniqueName: \"kubernetes.io/projected/b256234b-64f7-4d2f-9403-e84aaf928e35-kube-api-access-4gt58\") pod \"perf-node-gather-daemonset-csqmq\" (UID: \"b256234b-64f7-4d2f-9403-e84aaf928e35\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-csqmq"
Sep 17 16:11:03.912460 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:03.912378    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-45zt7_2c069c9e-0856-46b0-ba10-67829c05ac94/console-operator/1.log"
Sep 17 16:11:03.916636 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:03.916597    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-45zt7_2c069c9e-0856-46b0-ba10-67829c05ac94/console-operator/2.log"
Sep 17 16:11:03.927890 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:03.927869    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-csqmq"
Sep 17 16:11:04.220449 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:04.220416    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-25lkm/perf-node-gather-daemonset-csqmq"]
Sep 17 16:11:04.222716 ip-10-0-134-104 kubenswrapper[2556]: W0917 16:11:04.222691    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb256234b_64f7_4d2f_9403_e84aaf928e35.slice/crio-5fc916e7fd241e62270619b384c68c404812d43014cfd1b31d39924339e00e05 WatchSource:0}: Error finding container 5fc916e7fd241e62270619b384c68c404812d43014cfd1b31d39924339e00e05: Status 404 returned error can't find the container with id 5fc916e7fd241e62270619b384c68c404812d43014cfd1b31d39924339e00e05
Sep 17 16:11:04.631076 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:04.631009    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-5rsgl_3baff753-aee2-4eb4-a85f-4d01a2deda3d/download-server/0.log"
Sep 17 16:11:04.716091 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:04.716045    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-csqmq" event={"ID":"b256234b-64f7-4d2f-9403-e84aaf928e35","Type":"ContainerStarted","Data":"ab3d51ff77ecff9ca867ab2083b8c386cab431866751efd393d9e50cfab2111a"}
Sep 17 16:11:04.716257 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:04.716107    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-csqmq" event={"ID":"b256234b-64f7-4d2f-9403-e84aaf928e35","Type":"ContainerStarted","Data":"5fc916e7fd241e62270619b384c68c404812d43014cfd1b31d39924339e00e05"}
Sep 17 16:11:04.716257 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:04.716136    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-csqmq"
Sep 17 16:11:04.735241 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:04.735193    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-csqmq" podStartSLOduration=1.73517641 podStartE2EDuration="1.73517641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:11:04.733914925 +0000 UTC m=+2487.113932675" watchObservedRunningTime="2026-09-17 16:11:04.73517641 +0000 UTC m=+2487.115194163"
Sep 17 16:11:05.492160 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:05.492113    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-8pwxw_c9d7e7c9-d70d-4fd3-86df-ada691b1d830/volume-data-source-validator/0.log"
Sep 17 16:11:06.667695 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:06.667666    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k7kbh_de6d8a3f-074e-4f18-9650-0ca1cb32100f/dns/0.log"
Sep 17 16:11:06.705479 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:06.705451    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k7kbh_de6d8a3f-074e-4f18-9650-0ca1cb32100f/kube-rbac-proxy/0.log"
Sep 17 16:11:06.910432 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:06.910390    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xfzrp_593418fc-096d-445c-9863-9ed9f57ee092/dns-node-resolver/0.log"
Sep 17 16:11:07.599256 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:07.599231    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s9svr_fca9ba31-c254-4a6e-8607-fb6a4af8efc2/node-ca/0.log"
Sep 17 16:11:09.314511 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:09.314480    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-btzhg_2ec252e5-1112-40e2-b579-966af538ac4f/payload-pre-processing/0.log"
Sep 17 16:11:09.350839 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:09.350801    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-2wlv5_5bd2083c-fee3-4da3-95dd-1be423dab576/payload-processing/0.log"
Sep 17 16:11:09.381732 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:09.381708    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85cf78879b-z2qnj_8b73fab2-e530-4b68-acf0-c71d1943cb7c/router/0.log"
Sep 17 16:11:10.732205 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:10.732175    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-csqmq"
Sep 17 16:11:10.783608 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:10.783577    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r975h_ac3c1f9c-dbfd-4097-b8c5-50d3fa2962c5/serve-healthcheck-canary/0.log"
Sep 17 16:11:11.306178 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:11.306148    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-xqb4f_95eb306e-4e4f-48ed-9b49-14471d4b4ddf/insights-operator/0.log"
Sep 17 16:11:11.306488 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:11.306468    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-xqb4f_95eb306e-4e4f-48ed-9b49-14471d4b4ddf/insights-operator/1.log"
Sep 17 16:11:11.525441 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:11.525415    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n7dnd_82788049-f54a-43a9-903a-f59be68b859a/kube-rbac-proxy/0.log"
Sep 17 16:11:11.551960 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:11.551934    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n7dnd_82788049-f54a-43a9-903a-f59be68b859a/exporter/0.log"
Sep 17 16:11:11.577735 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:11.577667    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n7dnd_82788049-f54a-43a9-903a-f59be68b859a/extractor/0.log"
Sep 17 16:11:14.445264 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:14.445213    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-btzhg_2ec252e5-1112-40e2-b579-966af538ac4f/payload-pre-processing/0.log"
Sep 17 16:11:14.474636 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:14.474606    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-2wlv5_5bd2083c-fee3-4da3-95dd-1be423dab576/payload-processing/0.log"
Sep 17 16:11:14.493002 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:14.492966    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85cf78879b-z2qnj_8b73fab2-e530-4b68-acf0-c71d1943cb7c/router/0.log"
Sep 17 16:11:14.575888 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:14.575859    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-lq9sm_89181401-93fb-4dac-8941-57ebe347292a/manager/0.log"
Sep 17 16:11:14.632439 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:14.632412    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b9f9c646d-w244f_f5028686-ff8b-44ba-9bdb-59de4c69aa12/manager/0.log"
Sep 17 16:11:14.656740 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:14.656712    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-4g2kw_693bf480-ee10-4366-9335-ef0c2651f980/server/0.log"
Sep 17 16:11:14.842662 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:14.842625    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5df9996869-86l8g_fa8daaea-084c-4cb6-9c92-2b5a61aac914/manager/0.log"
Sep 17 16:11:14.864824 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:14.864795    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5df9996869-86l8g_fa8daaea-084c-4cb6-9c92-2b5a61aac914/copy-manifests/0.log"
Sep 17 16:11:16.285385 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:16.285352    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6c7d57fb89-fc5dq_0ccf5b35-fb27-4838-8a3b-37a3ed4a9f39/manager/0.log"
Sep 17 16:11:16.541804 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:16.541716    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-btzhg_2ec252e5-1112-40e2-b579-966af538ac4f/payload-pre-processing/0.log"
Sep 17 16:11:16.567830 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:16.567805    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-2wlv5_5bd2083c-fee3-4da3-95dd-1be423dab576/payload-processing/0.log"
Sep 17 16:11:16.596139 ip-10-0-134-104 kubenswrapper[2556]: I0917 16:11:16.596110    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85cf78879b-z2qnj_8b73fab2-e530-4b68-acf0-c71d1943cb7c/router/0.log"