Sep 11 17:18:31.641957 ip-10-0-130-89 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 17:18:31.641971 ip-10-0-130-89 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 17:18:31.641997 ip-10-0-130-89 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 17:18:31.642212 ip-10-0-130-89 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 17:18:41.888630 ip-10-0-130-89 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 11 17:18:41.888651 ip-10-0-130-89 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 7f4f7fd41e814327a837a283645ec00c --
Sep 11 17:20:32.081184 ip-10-0-130-89 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 17:20:32.449882 ip-10-0-130-89 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 17:20:32.449882 ip-10-0-130-89 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 17:20:32.449882 ip-10-0-130-89 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 17:20:32.449882 ip-10-0-130-89 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 17:20:32.449882 ip-10-0-130-89 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 17:20:32.451782 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.451693    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 17:20:32.454627 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454611    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:20:32.454627 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454627    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:20:32.454692 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454631    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:20:32.454692 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454635    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:20:32.454692 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454638    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:20:32.454692 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454641    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:20:32.454692 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454644    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:20:32.454692 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454647    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:20:32.454692 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454650    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:20:32.454692 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454653    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:20:32.454692 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454656    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:20:32.454692 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454659    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:20:32.454692 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454662    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:20:32.454692 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454665    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:20:32.454692 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454668    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:20:32.454692 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454670    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:20:32.454692 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454673    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:20:32.454692 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454676    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:20:32.454692 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454678    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:20:32.454692 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454681    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:20:32.454692 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454684    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:20:32.454692 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454694    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:20:32.455192 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454697    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:20:32.455192 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454700    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:20:32.455192 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454703    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:20:32.455192 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454706    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:20:32.455192 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454710    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:20:32.455192 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454714    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:20:32.455192 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454717    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:20:32.455192 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454720    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:20:32.455192 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454722    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:20:32.455192 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454725    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:20:32.455192 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454728    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:20:32.455192 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454731    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:20:32.455192 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454733    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:20:32.455192 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454736    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:20:32.455192 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454738    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:20:32.455192 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454741    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:20:32.455192 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454744    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:20:32.455192 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454746    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:20:32.455192 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454748    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:20:32.455655 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454751    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:20:32.455655 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454753    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:20:32.455655 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454756    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:20:32.455655 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454758    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:20:32.455655 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454760    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:20:32.455655 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454763    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:20:32.455655 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454765    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:20:32.455655 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454768    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:20:32.455655 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454770    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:20:32.455655 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454773    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:20:32.455655 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454776    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:20:32.455655 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454779    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:20:32.455655 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454781    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:20:32.455655 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454785    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:20:32.455655 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454788    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:20:32.455655 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454790    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:20:32.455655 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454793    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:20:32.455655 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454796    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:20:32.455655 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454798    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:20:32.455655 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454801    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:20:32.456152 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454803    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:20:32.456152 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454806    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:20:32.456152 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454808    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:20:32.456152 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454812    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:20:32.456152 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454817    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:20:32.456152 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454819    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:20:32.456152 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454822    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:20:32.456152 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454824    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:20:32.456152 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454827    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:20:32.456152 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454829    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:20:32.456152 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454832    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:20:32.456152 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454834    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:20:32.456152 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454837    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:20:32.456152 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454839    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:20:32.456152 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454842    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:20:32.456152 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454846    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:20:32.456152 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454848    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:20:32.456152 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454851    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:20:32.456152 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454854    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:20:32.456152 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454856    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:20:32.456686 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454859    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:20:32.456686 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454861    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:20:32.456686 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454864    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:20:32.456686 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454866    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:20:32.456686 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454869    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:20:32.456686 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454872    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:20:32.456686 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454876    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:20:32.456686 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454879    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:20:32.456686 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454881    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:20:32.456686 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454884    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:20:32.456686 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454887    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:20:32.456686 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454889    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:20:32.456686 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454906    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:20:32.456686 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454911    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:20:32.456686 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454915    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:20:32.456686 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454921    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:20:32.456686 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454926    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:20:32.456686 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.454929    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:20:32.456686 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455676    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:20:32.456686 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455683    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:20:32.457197 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455686    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:20:32.457197 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455689    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:20:32.457197 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455692    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:20:32.457197 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455695    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:20:32.457197 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455698    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:20:32.457197 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455701    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:20:32.457197 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455703    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:20:32.457197 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455706    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:20:32.457197 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455708    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:20:32.457197 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455711    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:20:32.457197 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455715    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:20:32.457197 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455717    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:20:32.457197 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455720    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:20:32.457197 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455722    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:20:32.457197 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455725    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:20:32.457197 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455728    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:20:32.457197 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455730    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:20:32.457197 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455734    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:20:32.457197 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455737    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:20:32.457671 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455740    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:20:32.457671 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455744    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:20:32.457671 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455747    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:20:32.457671 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455749    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:20:32.457671 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455752    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:20:32.457671 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455754    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:20:32.457671 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455757    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:20:32.457671 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455760    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:20:32.457671 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455762    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:20:32.457671 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455765    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:20:32.457671 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455767    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:20:32.457671 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455770    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:20:32.457671 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455772    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:20:32.457671 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455777    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:20:32.457671 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455781    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:20:32.457671 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455784    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:20:32.457671 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455787    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:20:32.457671 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455790    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:20:32.457671 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455792    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:20:32.457671 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455795    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:20:32.458183 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455798    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:20:32.458183 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455802    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:20:32.458183 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455806    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:20:32.458183 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455809    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:20:32.458183 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455812    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:20:32.458183 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455815    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:20:32.458183 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455818    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:20:32.458183 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455822    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:20:32.458183 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455824    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:20:32.458183 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455827    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:20:32.458183 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455830    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:20:32.458183 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455834    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:20:32.458183 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455836    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:20:32.458183 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455840    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:20:32.458183 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455842    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:20:32.458183 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455845    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:20:32.458183 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455847    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:20:32.458183 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455850    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:20:32.458183 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455852    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:20:32.458642 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455855    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:20:32.458642 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455857    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:20:32.458642 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455860    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:20:32.458642 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455862    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:20:32.458642 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455865    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:20:32.458642 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455867    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:20:32.458642 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455870    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:20:32.458642 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455872    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:20:32.458642 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455876    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:20:32.458642 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455878    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:20:32.458642 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455881    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:20:32.458642 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455884    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:20:32.458642 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455886    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:20:32.458642 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455889    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:20:32.458642 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455891    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:20:32.458642 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455908    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:20:32.458642 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455912    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:20:32.458642 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455914    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:20:32.458642 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455917    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:20:32.458642 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455919    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:20:32.459137 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455922    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:20:32.459137 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455925    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:20:32.459137 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455927    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:20:32.459137 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455931    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:20:32.459137 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455934    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:20:32.459137 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455937    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:20:32.459137 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455940    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:20:32.459137 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455943    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:20:32.459137 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455946    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:20:32.459137 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455948    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:20:32.459137 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455951    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:20:32.459137 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455954    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:20:32.459137 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455956    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:20:32.459137 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455959    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:20:32.459137 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455961    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:20:32.459137 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455964    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:20:32.459137 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455966    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:20:32.459137 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455969    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:20:32.459137 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.455971    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:20:32.459137 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456354    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:20:32.459632 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456360    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:20:32.459632 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456363    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:20:32.459632 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456365    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:20:32.459632 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456368    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:20:32.459632 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456371    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:20:32.459632 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456374    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:20:32.459632 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456377    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:20:32.459632 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456380    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:20:32.459632 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456382    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:20:32.459632 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456385    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:20:32.459632 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456387    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:20:32.459632 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456391    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:20:32.459632 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456394    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:20:32.459632 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456398    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:20:32.459632 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456402    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:20:32.459632 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456406    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:20:32.459632 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456410    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:20:32.459632 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456414    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:20:32.459632 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456417    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:20:32.460118 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456420    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:20:32.460118 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456422    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:20:32.460118 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456425    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:20:32.460118 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456428    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:20:32.460118 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456430    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:20:32.460118 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456433    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:20:32.460118 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456435    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:20:32.460118 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456438    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:20:32.460118 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456440    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:20:32.460118 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456443    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:20:32.460118 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456446    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:20:32.460118 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456448    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:20:32.460118 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456451    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:20:32.460118 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456454    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:20:32.460118 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456456    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:20:32.460118 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456459    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:20:32.460118 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456461    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:20:32.460118 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456464    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:20:32.460118 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456467    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:20:32.460585 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456469    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:20:32.460585 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456472    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:20:32.460585 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456475    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:20:32.460585 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456477    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:20:32.460585 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456480    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:20:32.460585 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456482    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:20:32.460585 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456486    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:20:32.460585 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456489    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:20:32.460585 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456492    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:20:32.460585 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456496    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:20:32.460585 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456499    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:20:32.460585 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456502    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:20:32.460585 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456505    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:20:32.460585 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456509    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:20:32.460585 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456512    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:20:32.460585 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456515    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:20:32.460585 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456518    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:20:32.460585 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456520    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:20:32.460585 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456523    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:20:32.461142 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456526    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:20:32.461142 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456543    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:20:32.461142 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456547    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:20:32.461142 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456550    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:20:32.461142 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456553    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:20:32.461142 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456556    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:20:32.461142 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456559    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:20:32.461142 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456562    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:20:32.461142 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456565    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:20:32.461142 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456568    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:20:32.461142 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456570    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:20:32.461142 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456573    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:20:32.461142 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456576    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:20:32.461142 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456579    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:20:32.461142 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456581    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:20:32.461142 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456584    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:20:32.461142 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456586    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:20:32.461142 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456589    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:20:32.461142 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456591    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:20:32.461142 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456594    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:20:32.461672 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456597    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:20:32.461672 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456599    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:20:32.461672 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456603    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:20:32.461672 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456606    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:20:32.461672 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456608    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:20:32.461672 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456611    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:20:32.461672 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456613    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:20:32.461672 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456616    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:20:32.461672 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456619    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:20:32.461672 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456621    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:20:32.461672 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456624    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:20:32.461672 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456626    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:20:32.461672 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456629    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:20:32.461672 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456632    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:20:32.461672 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456634    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:20:32.461672 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456637    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:20:32.461672 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456639    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:20:32.461672 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456642    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:20:32.461672 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456644    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:20:32.461672 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456647    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:20:32.462175 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.456650    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:20:32.462175 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456720    2560 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 17:20:32.462175 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456756    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 17:20:32.462175 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456763    2560 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 17:20:32.462175 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456770    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 17:20:32.462175 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456774    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 17:20:32.462175 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456777    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 17:20:32.462175 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456781    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 17:20:32.462175 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456786    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 17:20:32.462175 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456791    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 17:20:32.462175 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456795    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 17:20:32.462175 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456798    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 17:20:32.462175 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456801    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 17:20:32.462175 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456805    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 17:20:32.462175 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456808    2560 flags.go:64] FLAG: --cgroup-root=""
Sep 11 17:20:32.462175 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456811    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 17:20:32.462175 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456814    2560 flags.go:64] FLAG: --client-ca-file=""
Sep 11 17:20:32.462175 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456817    2560 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 17:20:32.462175 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456820    2560 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 17:20:32.462175 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456826    2560 flags.go:64] FLAG: --cluster-domain=""
Sep 11 17:20:32.462175 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456829    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 17:20:32.462175 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456832    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 17:20:32.462175 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456836    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 17:20:32.462175 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456839    2560 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 17:20:32.462758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456843    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 17:20:32.462758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456846    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 17:20:32.462758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456849    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 17:20:32.462758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456853    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 17:20:32.462758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456856    2560 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 17:20:32.462758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456859    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 17:20:32.462758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456862    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 17:20:32.462758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456865    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 17:20:32.462758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456868    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 17:20:32.462758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456872    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 17:20:32.462758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456875    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 17:20:32.462758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456878    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 17:20:32.462758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456881    2560 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 17:20:32.462758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456884    2560 flags.go:64] FLAG: --enable-server="true"
Sep 11 17:20:32.462758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456887    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 17:20:32.462758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456905    2560 flags.go:64] FLAG: --event-burst="100"
Sep 11 17:20:32.462758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456908    2560 flags.go:64] FLAG: --event-qps="50"
Sep 11 17:20:32.462758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456913    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 17:20:32.462758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456916    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 17:20:32.462758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456919    2560 flags.go:64] FLAG: --eviction-hard=""
Sep 11 17:20:32.462758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456923    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 17:20:32.462758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456926    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 17:20:32.462758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456930    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 17:20:32.462758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456933    2560 flags.go:64] FLAG: --eviction-soft=""
Sep 11 17:20:32.462758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456936    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 17:20:32.463363 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456939    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 17:20:32.463363 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456942    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 17:20:32.463363 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456945    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 17:20:32.463363 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456948    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 17:20:32.463363 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456951    2560 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 17:20:32.463363 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456954    2560 flags.go:64] FLAG: --feature-gates=""
Sep 11 17:20:32.463363 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456958    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 17:20:32.463363 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456961    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 17:20:32.463363 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456964    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 17:20:32.463363 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456967    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 17:20:32.463363 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456970    2560 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 17:20:32.463363 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456974    2560 flags.go:64] FLAG: --help="false"
Sep 11 17:20:32.463363 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456977    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-130-89.ec2.internal"
Sep 11 17:20:32.463363 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456982    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 17:20:32.463363 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456985    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 17:20:32.463363 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456988    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 17:20:32.463363 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456991    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 17:20:32.463363 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456994    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 17:20:32.463363 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.456997    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 17:20:32.463363 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457000    2560 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 17:20:32.463363 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457004    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 17:20:32.463363 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457006    2560 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 17:20:32.463363 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457009    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 17:20:32.463960 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457013    2560 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 17:20:32.463960 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457018    2560 flags.go:64] FLAG: --kube-reserved=""
Sep 11 17:20:32.463960 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457021    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 17:20:32.463960 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457024    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 17:20:32.463960 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457027    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 17:20:32.463960 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457030    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 17:20:32.463960 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457033    2560 flags.go:64] FLAG: --lock-file=""
Sep 11 17:20:32.463960 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457036    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 17:20:32.463960 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457039    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 17:20:32.463960 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457042    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 17:20:32.463960 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457047    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 17:20:32.463960 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457050    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 17:20:32.463960 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457053    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 17:20:32.463960 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457056    2560 flags.go:64] FLAG: --logging-format="text"
Sep 11 17:20:32.463960 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457059    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 17:20:32.463960 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457063    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 17:20:32.463960 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457066    2560 flags.go:64] FLAG: --manifest-url=""
Sep 11 17:20:32.463960 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457069    2560 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 17:20:32.463960 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457074    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 17:20:32.463960 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457077    2560 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 17:20:32.463960 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457086    2560 flags.go:64] FLAG: --max-pods="110"
Sep 11 17:20:32.463960 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457089    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 17:20:32.463960 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457095    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 17:20:32.463960 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457098    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 17:20:32.463960 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457101    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 17:20:32.464850 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457104    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 17:20:32.464850 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457107    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 17:20:32.464850 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457110    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 17:20:32.464850 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457117    2560 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 17:20:32.464850 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457120    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 17:20:32.464850 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457123    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 17:20:32.464850 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457126    2560 flags.go:64] FLAG: --pod-cidr=""
Sep 11 17:20:32.464850 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457129    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 17:20:32.464850 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457134    2560 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 17:20:32.464850 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457138    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 17:20:32.464850 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457142    2560 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 17:20:32.464850 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457145    2560 flags.go:64] FLAG: --port="10250"
Sep 11 17:20:32.464850 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457148    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 17:20:32.464850 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457151    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0676ab60768b20ca7"
Sep 11 17:20:32.464850 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457154    2560 flags.go:64] FLAG: --qos-reserved=""
Sep 11 17:20:32.464850 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457157    2560 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 17:20:32.464850 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457160    2560 flags.go:64] FLAG: --register-node="true"
Sep 11 17:20:32.464850 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457163    2560 flags.go:64] FLAG: --register-with-taints=""
Sep 11 17:20:32.464850 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457167    2560 flags.go:64] FLAG: --registry-burst="10"
Sep 11 17:20:32.464850 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457170    2560 flags.go:64] FLAG: --registry-qps="5"
Sep 11 17:20:32.464850 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457173    2560 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 17:20:32.464850 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457176    2560 flags.go:64] FLAG: --reserved-memory=""
Sep 11 17:20:32.464850 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457179    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 17:20:32.464850 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457183    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 17:20:32.464850 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457186    2560 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 17:20:32.465550 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457189    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 17:20:32.465550 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457192    2560 flags.go:64] FLAG: --runonce="false"
Sep 11 17:20:32.465550 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457195    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 17:20:32.465550 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457198    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 17:20:32.465550 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457206    2560 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 17:20:32.465550 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457209    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 17:20:32.465550 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457212    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 17:20:32.465550 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457215    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 17:20:32.465550 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457219    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 17:20:32.465550 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457223    2560 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 17:20:32.465550 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457226    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 17:20:32.465550 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457229    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 17:20:32.465550 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457232    2560 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 17:20:32.465550 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457235    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 17:20:32.465550 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457238    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 17:20:32.465550 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457241    2560 flags.go:64] FLAG: --system-cgroups=""
Sep 11 17:20:32.465550 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457245    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 17:20:32.465550 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457250    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 17:20:32.465550 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457253    2560 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 17:20:32.465550 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457256    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 17:20:32.465550 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457260    2560 flags.go:64] FLAG: --tls-min-version=""
Sep 11 17:20:32.465550 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457263    2560 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 17:20:32.465550 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457266    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 17:20:32.465550 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457268    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 17:20:32.465550 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457271    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 17:20:32.466234 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457274    2560 flags.go:64] FLAG: --v="2"
Sep 11 17:20:32.466234 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457278    2560 flags.go:64] FLAG: --version="false"
Sep 11 17:20:32.466234 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457282    2560 flags.go:64] FLAG: --vmodule=""
Sep 11 17:20:32.466234 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457286    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 17:20:32.466234 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.457289    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 17:20:32.466234 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457405    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:20:32.466234 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457409    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:20:32.466234 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457413    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:20:32.466234 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457416    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:20:32.466234 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457419    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:20:32.466234 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457422    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:20:32.466234 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457427    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:20:32.466234 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457431    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:20:32.466234 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457435    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:20:32.466234 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457438    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:20:32.466234 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457441    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:20:32.466234 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457443    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:20:32.466234 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457446    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:20:32.466234 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457449    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:20:32.466234 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457452    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:20:32.466234 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457454    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:20:32.466768 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457457    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:20:32.466768 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457459    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:20:32.466768 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457462    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:20:32.466768 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457465    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:20:32.466768 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457468    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:20:32.466768 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457471    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:20:32.466768 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457473    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:20:32.466768 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457476    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:20:32.466768 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457479    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:20:32.466768 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457482    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:20:32.466768 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457484    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:20:32.466768 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457487    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:20:32.466768 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457489    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:20:32.466768 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457492    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:20:32.466768 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457495    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:20:32.466768 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457497    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:20:32.466768 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457500    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:20:32.466768 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457503    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:20:32.466768 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457506    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:20:32.467268 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457508    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:20:32.467268 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457511    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:20:32.467268 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457513    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:20:32.467268 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457518    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:20:32.467268 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457520    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:20:32.467268 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457523    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:20:32.467268 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457525    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:20:32.467268 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457528    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:20:32.467268 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457530    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:20:32.467268 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457533    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:20:32.467268 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457536    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:20:32.467268 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457538    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:20:32.467268 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457541    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:20:32.467268 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457543    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:20:32.467268 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457546    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:20:32.467268 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457549    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:20:32.467268 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457552    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:20:32.467268 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457556    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:20:32.467268 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457559    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:20:32.467749 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457562    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:20:32.467749 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457565    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:20:32.467749 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457568    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:20:32.467749 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457570    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:20:32.467749 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457573    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:20:32.467749 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457576    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:20:32.467749 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457579    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:20:32.467749 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457581    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:20:32.467749 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457585    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:20:32.467749 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457589    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:20:32.467749 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457592    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:20:32.467749 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457595    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:20:32.467749 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457597    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:20:32.467749 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457601    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:20:32.467749 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457603    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:20:32.467749 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457606    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:20:32.467749 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457610    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:20:32.467749 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457612    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:20:32.467749 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457615    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:20:32.467749 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457617    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:20:32.468258 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457620    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:20:32.468258 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457622    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:20:32.468258 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457625    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:20:32.468258 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457628    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:20:32.468258 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457630    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:20:32.468258 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457632    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:20:32.468258 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457635    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:20:32.468258 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457638    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:20:32.468258 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457640    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:20:32.468258 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457644    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:20:32.468258 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457646    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:20:32.468258 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457649    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:20:32.468258 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457652    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:20:32.468258 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457654    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:20:32.468258 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457657    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:20:32.468258 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457659    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:20:32.468258 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457662    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:20:32.468258 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457665    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:20:32.468258 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457667    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:20:32.468258 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457670    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:20:32.468759 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457673    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:20:32.468759 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457675    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:20:32.468759 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457678    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:20:32.468759 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457681    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:20:32.468759 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.457684    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:20:32.468759 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.458113    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 17:20:32.468925 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.468782    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 17:20:32.468925 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.468804    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 17:20:32.469578 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.469567    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 17:20:32.469607 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.469578    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 17:20:32.469710 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469698    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:20:32.469710 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469709    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:20:32.469766 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469713    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:20:32.469766 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469717    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:20:32.469766 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469720    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:20:32.469766 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469723    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:20:32.469766 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469726    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:20:32.469766 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469729    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:20:32.469766 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469732    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:20:32.469766 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469735    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:20:32.469766 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469738    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:20:32.469766 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469741    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:20:32.469766 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469744    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:20:32.469766 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469749    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:20:32.469766 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469752    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:20:32.469766 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469755    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:20:32.469766 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469758    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:20:32.469766 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469761    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:20:32.469766 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469764    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:20:32.469766 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469767    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:20:32.469766 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469770    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:20:32.469766 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469773    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:20:32.470338 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469776    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:20:32.470338 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469780    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:20:32.470338 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469784    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:20:32.470338 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469791    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:20:32.470338 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469794    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:20:32.470338 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469798    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:20:32.470338 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469807    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:20:32.470338 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469810    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:20:32.470338 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469813    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:20:32.470338 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469816    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:20:32.470338 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469818    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:20:32.470338 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469821    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:20:32.470338 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469824    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:20:32.470338 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469826    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:20:32.470338 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469829    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:20:32.470338 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469832    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:20:32.470338 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469838    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:20:32.470338 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469840    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:20:32.470338 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469843    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:20:32.470338 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469846    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:20:32.470822 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469848    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:20:32.470822 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469852    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:20:32.470822 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469855    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:20:32.470822 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469858    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:20:32.470822 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469861    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:20:32.470822 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469864    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:20:32.470822 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469866    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:20:32.470822 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469869    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:20:32.470822 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469874    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:20:32.470822 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469877    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:20:32.470822 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469880    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:20:32.470822 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469883    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:20:32.470822 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469886    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:20:32.470822 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469889    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:20:32.470822 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469910    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:20:32.470822 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469915    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:20:32.470822 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469919    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:20:32.470822 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469922    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:20:32.470822 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469924    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:20:32.470822 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469934    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:20:32.471332 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469937    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:20:32.471332 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469942    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:20:32.471332 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469945    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:20:32.471332 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469948    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:20:32.471332 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469950    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:20:32.471332 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469953    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:20:32.471332 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469956    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:20:32.471332 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469959    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:20:32.471332 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469963    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:20:32.471332 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469966    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:20:32.471332 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469968    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:20:32.471332 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469971    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:20:32.471332 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469975    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:20:32.471332 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469981    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:20:32.471332 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469984    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:20:32.471332 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469987    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:20:32.471332 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469990    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:20:32.471332 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469993    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:20:32.471332 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469996    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:20:32.471332 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.469999    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:20:32.471816 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470001    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:20:32.471816 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470004    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:20:32.471816 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470008    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:20:32.471816 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470011    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:20:32.471816 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470014    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:20:32.471816 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470017    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:20:32.471816 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470022    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:20:32.471816 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470025    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:20:32.471816 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470027    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:20:32.471816 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470030    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:20:32.471816 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470033    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:20:32.471816 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470036    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:20:32.471816 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470044    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:20:32.471816 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470047    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:20:32.471816 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470050    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:20:32.471816 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470053    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:20:32.471816 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470055    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:20:32.472247 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.470061    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 17:20:32.472247 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470313    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:20:32.472247 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470320    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:20:32.472247 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470323    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:20:32.472247 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470327    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:20:32.472247 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470330    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:20:32.472247 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470333    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:20:32.472247 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470336    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:20:32.472247 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470338    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:20:32.472247 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470341    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:20:32.472247 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470347    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:20:32.472247 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470350    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:20:32.472247 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470353    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:20:32.472247 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470356    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:20:32.472605 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470358    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:20:32.472605 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470361    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:20:32.472605 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470364    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:20:32.472605 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470366    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:20:32.472605 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470370    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:20:32.472605 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470372    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:20:32.472605 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470375    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:20:32.472605 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470378    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:20:32.472605 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470381    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:20:32.472605 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470386    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:20:32.472605 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470389    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:20:32.472605 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470392    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:20:32.472605 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470395    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:20:32.472605 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470398    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:20:32.472605 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470406    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:20:32.472605 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470409    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:20:32.472605 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470412    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:20:32.472605 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470414    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:20:32.472605 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470417    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:20:32.472605 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470420    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:20:32.473093 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470423    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:20:32.473093 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470428    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:20:32.473093 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470431    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:20:32.473093 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470433    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:20:32.473093 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470436    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:20:32.473093 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470438    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:20:32.473093 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470441    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:20:32.473093 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470444    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:20:32.473093 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470447    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:20:32.473093 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470449    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:20:32.473093 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470452    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:20:32.473093 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470455    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:20:32.473093 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470457    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:20:32.473093 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470463    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:20:32.473093 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470466    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:20:32.473093 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470469    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:20:32.473093 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470472    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:20:32.473093 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470475    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:20:32.473093 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470478    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:20:32.473093 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470481    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:20:32.473569 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470484    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:20:32.473569 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470486    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:20:32.473569 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470489    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:20:32.473569 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470492    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:20:32.473569 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470495    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:20:32.473569 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470497    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:20:32.473569 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470502    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:20:32.473569 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470512    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:20:32.473569 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470516    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:20:32.473569 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470519    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:20:32.473569 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470523    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:20:32.473569 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470526    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:20:32.473569 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470529    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:20:32.473569 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470532    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:20:32.473569 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470535    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:20:32.473569 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470538    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:20:32.473569 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470540    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:20:32.473569 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470543    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:20:32.473569 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470548    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:20:32.474121 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470551    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:20:32.474121 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470554    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:20:32.474121 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470557    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:20:32.474121 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470559    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:20:32.474121 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470562    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:20:32.474121 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470565    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:20:32.474121 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470567    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:20:32.474121 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470570    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:20:32.474121 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470573    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:20:32.474121 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470576    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:20:32.474121 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470579    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:20:32.474121 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470587    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:20:32.474121 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470592    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:20:32.474121 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470596    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:20:32.474121 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470600    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:20:32.474121 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470603    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:20:32.474121 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470606    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:20:32.474121 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470608    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:20:32.474121 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470611    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:20:32.474572 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470614    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:20:32.474572 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470617    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:20:32.474572 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470626    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:20:32.474572 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470629    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:20:32.474572 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470631    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:20:32.474572 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470637    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:20:32.474572 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470639    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:20:32.474572 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:32.470642    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:20:32.474572 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.470648    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 17:20:32.474572 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.470835    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 17:20:32.474572 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.473041    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 17:20:32.474572 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.474102    2560 server.go:1019] "Starting client certificate rotation"
Sep 11 17:20:32.474572 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.474208    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 17:20:32.474929 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.474916    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 17:20:32.476507 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.476493    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 17:20:32.477836 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.477819    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 17:20:32.486762 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.486745    2560 log.go:25] "Validated CRI v1 runtime API"
Sep 11 17:20:32.491261 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.491243    2560 log.go:25] "Validated CRI v1 image API"
Sep 11 17:20:32.493514 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.493497    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 17:20:32.496888 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.496863    2560 fs.go:135] Filesystem UUIDs: map[082a2ef6-772d-4307-adbe-bec6356d783d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fcac9163-b4fc-4639-9b00-2766564cb57c:/dev/nvme0n1p3]
Sep 11 17:20:32.496956 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.496889    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 17:20:32.501273 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.501257    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 17:20:32.502750 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.502631    2560 manager.go:216] Machine: {Timestamp:2026-09-11 17:20:32.500933144 +0000 UTC m=+0.305105555 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099213 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2279da10fc6cd5f2bbc4e788f885cb SystemUUID:ec2279da-10fc-6cd5-f2bb-c4e788f885cb BootID:7f4f7fd4-1e81-4327-a837-a283645ec00c Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:12:f8:f5:95:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:12:f8:f5:95:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:0f:17:db:c2:29 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 17:20:32.502750 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.502745    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 17:20:32.502942 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.502838    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 17:20:32.505938 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.505891    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 17:20:32.506099 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.505951    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-89.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 17:20:32.506554 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.506545    2560 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 17:20:32.506587 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.506557    2560 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 17:20:32.506587 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.506565    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 17:20:32.507649 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.507637    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 17:20:32.508374 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.508364    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 17:20:32.511187 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.511174    2560 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 17:20:32.511334 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.511324    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 17:20:32.511380 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.511356    2560 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 17:20:32.511380 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.511370    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 17:20:32.511437 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.511406    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 17:20:32.511437 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.511418    2560 kubelet.go:398] "Adding apiserver pod source"
Sep 11 17:20:32.511437 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.511430    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 17:20:32.511910 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.511877    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 17:20:32.511910 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.511909    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 17:20:32.514173 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.514152    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 17:20:32.516143 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.516129    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 17:20:32.516226 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.516160    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 17:20:32.518126 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.518111    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 17:20:32.518126 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.518130    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 17:20:32.518232 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.518137    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 17:20:32.518232 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.518143    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 17:20:32.518232 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.518152    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 17:20:32.518232 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.518172    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 17:20:32.518232 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.518186    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 17:20:32.518232 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.518192    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 17:20:32.518232 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.518211    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 17:20:32.518232 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.518217    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 17:20:32.518232 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.518226    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 17:20:32.518232 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.518232    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 17:20:32.519616 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.519599    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 17:20:32.519652 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.519627    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 17:20:32.523346 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.523328    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-89.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 17:20:32.523433 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:32.523362    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 17:20:32.523433 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:32.523391    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-89.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 17:20:32.524887 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.524872    2560 server.go:1268] "Started kubelet"
Sep 11 17:20:32.524970 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.524938    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 17:20:32.525042 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.524994    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 17:20:32.525075 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.525066    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 17:20:32.525780 ip-10-0-130-89 systemd[1]: Started Kubernetes Kubelet.
Sep 11 17:20:32.526427 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.526409    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 17:20:32.527711 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.527571    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z8frs"
Sep 11 17:20:32.528402 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.528249    2560 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 17:20:32.529778 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:32.529756    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 17:20:32.531769 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.531753    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 17:20:32.532289 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.532276    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 17:20:32.532421 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:32.532397    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-89.ec2.internal\" not found"
Sep 11 17:20:32.532784 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.532770    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 17:20:32.533222 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.533095    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 17:20:32.533315 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.533225    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 17:20:32.533373 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.533327    2560 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 17:20:32.533373 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.533335    2560 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 17:20:32.533471 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.533392    2560 factory.go:55] Registering systemd factory
Sep 11 17:20:32.533471 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.533406    2560 factory.go:223] Registration of the systemd container factory successfully
Sep 11 17:20:32.533695 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.533675    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z8frs"
Sep 11 17:20:32.533771 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.533713    2560 factory.go:153] Registering CRI-O factory
Sep 11 17:20:32.533771 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.533730    2560 factory.go:223] Registration of the crio container factory successfully
Sep 11 17:20:32.533868 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.533795    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 17:20:32.533868 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.533815    2560 factory.go:103] Registering Raw factory
Sep 11 17:20:32.533868 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.533830    2560 manager.go:1196] Started watching for new ooms in manager
Sep 11 17:20:32.534030 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:32.533037    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-89.ec2.internal.18d453edbd8438f9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-89.ec2.internal,UID:ip-10-0-130-89.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-89.ec2.internal,},FirstTimestamp:2026-09-11 17:20:32.524843257 +0000 UTC m=+0.329015667,LastTimestamp:2026-09-11 17:20:32.524843257 +0000 UTC m=+0.329015667,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-89.ec2.internal,}"
Sep 11 17:20:32.534289 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.534268    2560 manager.go:349] Starting recovery of all containers
Sep 11 17:20:32.535743 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:32.535595    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 17:20:32.535835 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:32.535809    2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-89.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 17:20:32.548032 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.548011    2560 manager.go:354] Recovery completed
Sep 11 17:20:32.552281 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.552267    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:20:32.554159 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.554139    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-89.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:20:32.554249 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.554169    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-89.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:20:32.554249 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.554181    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-89.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:20:32.554795 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.554779    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 17:20:32.554795 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.554791    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 17:20:32.554944 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.554808    2560 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 17:20:32.554944 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.554823    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:20:32.555586 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.555566    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-89.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:20:32.555650 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.555603    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-89.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:20:32.555650 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.555618    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-89.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:20:32.556025 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.556013    2560 policy_none.go:49] "None policy: Start"
Sep 11 17:20:32.556083 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.556028    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 17:20:32.556083 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.556038    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 17:20:32.556994 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.556983    2560 policy_none.go:47] "Start"
Sep 11 17:20:32.580533 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.580515    2560 manager.go:337] "Starting Device Plugin manager"
Sep 11 17:20:32.580628 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:32.580558    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 17:20:32.580628 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.580571    2560 server.go:87] "Starting device plugin registration server"
Sep 11 17:20:32.581009 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.580995    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 17:20:32.581066 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.581012    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 17:20:32.581216 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.581127    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 17:20:32.581216 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.581213    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 17:20:32.581357 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.581222    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 17:20:32.581791 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:32.581747    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 17:20:32.581866 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:32.581806    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-89.ec2.internal\" not found"
Sep 11 17:20:32.705857 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.682007    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:20:32.705857 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.683070    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-89.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:20:32.705857 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.683098    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-89.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:20:32.705857 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.683108    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-89.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:20:32.705857 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.683132    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-89.ec2.internal"
Sep 11 17:20:32.705857 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.691162    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-89.ec2.internal"
Sep 11 17:20:32.705857 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:32.691183    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-89.ec2.internal\": node \"ip-10-0-130-89.ec2.internal\" not found"
Sep 11 17:20:32.707369 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.707348    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 17:20:32.708713 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.708694    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 17:20:32.708816 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.708738    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 17:20:32.708816 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.708756    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 17:20:32.708816 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:32.708794    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 17:20:32.710551 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:32.710528    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-89.ec2.internal\" not found"
Sep 11 17:20:32.711291 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.711276    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:20:32.809560 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.809516    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-89.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-89.ec2.internal"]
Sep 11 17:20:32.809742 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.809613    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:20:32.810967 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:32.810954    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-89.ec2.internal\" not found"
Sep 11 17:20:32.811363 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.811349    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-89.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:20:32.811422 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.811378    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-89.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:20:32.811422 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.811388    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-89.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:20:32.812522 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.812510    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:20:32.812643 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.812628    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-89.ec2.internal"
Sep 11 17:20:32.812679 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.812668    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:20:32.814313 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.814297    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-89.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:20:32.814382 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.814321    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-89.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:20:32.814382 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.814331    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-89.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:20:32.814382 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.814366    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-89.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:20:32.814476 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.814389    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-89.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:20:32.814476 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.814401    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-89.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:20:32.815605 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.815589    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-89.ec2.internal"
Sep 11 17:20:32.815646 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.815622    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:20:32.816430 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.816416    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-89.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:20:32.816485 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.816440    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-89.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:20:32.816956 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.816450    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-89.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:20:32.834990 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.834971    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29212e0b773c1bb0ac3a9172435b9a0f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-89.ec2.internal\" (UID: \"29212e0b773c1bb0ac3a9172435b9a0f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-89.ec2.internal"
Sep 11 17:20:32.835063 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.834993    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29212e0b773c1bb0ac3a9172435b9a0f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-89.ec2.internal\" (UID: \"29212e0b773c1bb0ac3a9172435b9a0f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-89.ec2.internal"
Sep 11 17:20:32.835063 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.835010    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/537070a60bc30f06ce6f629121ed76a8-config\") pod \"kube-apiserver-proxy-ip-10-0-130-89.ec2.internal\" (UID: \"537070a60bc30f06ce6f629121ed76a8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-89.ec2.internal"
Sep 11 17:20:32.841613 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:32.841590    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-89.ec2.internal\" not found" node="ip-10-0-130-89.ec2.internal"
Sep 11 17:20:32.844815 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:32.844799    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-89.ec2.internal\" not found" node="ip-10-0-130-89.ec2.internal"
Sep 11 17:20:32.911981 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:32.911947    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-89.ec2.internal\" not found"
Sep 11 17:20:32.935857 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.935829    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/537070a60bc30f06ce6f629121ed76a8-config\") pod \"kube-apiserver-proxy-ip-10-0-130-89.ec2.internal\" (UID: \"537070a60bc30f06ce6f629121ed76a8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-89.ec2.internal"
Sep 11 17:20:32.935946 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.935861    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29212e0b773c1bb0ac3a9172435b9a0f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-89.ec2.internal\" (UID: \"29212e0b773c1bb0ac3a9172435b9a0f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-89.ec2.internal"
Sep 11 17:20:32.935946 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.935880    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29212e0b773c1bb0ac3a9172435b9a0f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-89.ec2.internal\" (UID: \"29212e0b773c1bb0ac3a9172435b9a0f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-89.ec2.internal"
Sep 11 17:20:32.936012 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.935958    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/537070a60bc30f06ce6f629121ed76a8-config\") pod \"kube-apiserver-proxy-ip-10-0-130-89.ec2.internal\" (UID: \"537070a60bc30f06ce6f629121ed76a8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-89.ec2.internal"
Sep 11 17:20:32.936012 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.935973    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29212e0b773c1bb0ac3a9172435b9a0f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-89.ec2.internal\" (UID: \"29212e0b773c1bb0ac3a9172435b9a0f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-89.ec2.internal"
Sep 11 17:20:32.936012 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:32.935973    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29212e0b773c1bb0ac3a9172435b9a0f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-89.ec2.internal\" (UID: \"29212e0b773c1bb0ac3a9172435b9a0f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-89.ec2.internal"
Sep 11 17:20:33.012144 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:33.012067    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-89.ec2.internal\" not found"
Sep 11 17:20:33.112647 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:33.112605    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-89.ec2.internal\" not found"
Sep 11 17:20:33.144182 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:33.144158    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-89.ec2.internal"
Sep 11 17:20:33.148003 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:33.147743    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-89.ec2.internal"
Sep 11 17:20:33.213331 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:33.213291    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-89.ec2.internal\" not found"
Sep 11 17:20:33.314171 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:33.314149    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-89.ec2.internal\" not found"
Sep 11 17:20:33.355823 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:33.355797    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:20:33.416672 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:33.416635    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-89.ec2.internal\" not found"
Sep 11 17:20:33.474938 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:33.474908    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 17:20:33.475426 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:33.475043    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 17:20:33.475426 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:33.475104    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 17:20:33.516840 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:33.516809    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-89.ec2.internal\" not found"
Sep 11 17:20:33.524439 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:33.524418    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:20:33.537422 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:33.537375    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 17:15:33 +0000 UTC" deadline="2028-06-03 06:55:49.04783595 +0000 UTC"
Sep 11 17:20:33.537422 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:33.537415    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15133h35m15.510423667s"
Sep 11 17:20:33.619076 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:33.619042    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-89.ec2.internal\" not found"
Sep 11 17:20:33.719960 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:33.719923    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-89.ec2.internal\" not found"
Sep 11 17:20:33.775325 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:33.775282    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod537070a60bc30f06ce6f629121ed76a8.slice/crio-ecbbbf5be0081aa5d38caeaaac288bee45b7d10682f19b9fef6d8935a9486c60 WatchSource:0}: Error finding container ecbbbf5be0081aa5d38caeaaac288bee45b7d10682f19b9fef6d8935a9486c60: Status 404 returned error can't find the container with id ecbbbf5be0081aa5d38caeaaac288bee45b7d10682f19b9fef6d8935a9486c60
Sep 11 17:20:33.775733 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:33.775700    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29212e0b773c1bb0ac3a9172435b9a0f.slice/crio-4aecc2bd0c3d1cb17a96962286b06a602b8218f04a7323e5d905f3940a8834e9 WatchSource:0}: Error finding container 4aecc2bd0c3d1cb17a96962286b06a602b8218f04a7323e5d905f3940a8834e9: Status 404 returned error can't find the container with id 4aecc2bd0c3d1cb17a96962286b06a602b8218f04a7323e5d905f3940a8834e9
Sep 11 17:20:33.780160 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:33.780139    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:20:33.821623 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:33.821589    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-89.ec2.internal\" not found"
Sep 11 17:20:33.927806 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:33.927733    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-89.ec2.internal\" not found"
Sep 11 17:20:33.947528 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:33.947496    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:20:34.032592 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.032559    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-89.ec2.internal"
Sep 11 17:20:34.044811 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.044789    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 17:20:34.045729 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.045715    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-89.ec2.internal"
Sep 11 17:20:34.053047 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.053029    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 17:20:34.469726 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.469677    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:20:34.513197 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.513165    2560 apiserver.go:52] "Watching apiserver"
Sep 11 17:20:34.521793 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.520768    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 17:20:34.521793 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.521357    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-82kzs","kube-system/konnectivity-agent-pm66d","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf","openshift-cluster-node-tuning-operator/tuned-bcdn9","openshift-image-registry/node-ca-q9dr2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-89.ec2.internal","openshift-multus/network-metrics-daemon-sn2g7","openshift-network-diagnostics/network-check-target-xjqcw","kube-system/kube-apiserver-proxy-ip-10-0-130-89.ec2.internal","openshift-dns/node-resolver-4mxdh","openshift-multus/multus-additional-cni-plugins-6665r","openshift-multus/multus-c9pzn","openshift-network-operator/iptables-alerter-cjjws","openshift-ovn-kubernetes/ovnkube-node-8dnzt"]
Sep 11 17:20:34.524070 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.523686    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:20:34.524070 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:34.523769    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn2g7" podUID="1203d22c-52e3-4681-9707-66ca35b9e72a"
Sep 11 17:20:34.525569 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.525480    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cjjws"
Sep 11 17:20:34.528903 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.527097    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf"
Sep 11 17:20:34.528903 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.527950    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:20:34.528903 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.528154    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vb66d\""
Sep 11 17:20:34.528903 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.528415    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 17:20:34.528903 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.528606    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 17:20:34.533529 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.533509    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 17:20:34.533779 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.533765    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 17:20:34.534380 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.534023    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qwnzq\""
Sep 11 17:20:34.534380 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.534218    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 17:20:34.534629 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.534600    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 17:20:34.535999 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.534750    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.536840 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.536571    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q9dr2"
Sep 11 17:20:34.538695 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.538672    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.540255 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.540028    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 17:20:34.540255 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.540047    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 17:20:34.540255 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.540142    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:20:34.540255 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.540201    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ppmm6\""
Sep 11 17:20:34.540255 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.540206    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xg2nl\""
Sep 11 17:20:34.540568 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.540362    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 17:20:34.540568 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.540433    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:20:34.540568 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.540435    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 17:20:34.540568 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:34.540494    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjqcw" podUID="a8a65b90-c7a6-4272-a1e8-d60096120332"
Sep 11 17:20:34.544523 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.541855    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 17:20:34.544523 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.542107    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-z7hq6\""
Sep 11 17:20:34.544523 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.542300    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 17:20:34.544523 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.542803    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 17:20:34.544523 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.543045    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 17:20:34.544523 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.543160    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 17:20:34.544523 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.543468    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 17:20:34.544523 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.543480    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:20:34.544523 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:34.543550    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82kzs" podUID="97afce2d-e357-4c76-b513-2cb05322bbde"
Sep 11 17:20:34.544523 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.543678    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4206b2fb-d03d-4827-9658-6dc11f47b6dc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8zstf\" (UID: \"4206b2fb-d03d-4827-9658-6dc11f47b6dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf"
Sep 11 17:20:34.544523 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.543707    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-tmp\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.544523 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.543736    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z78b\" (UniqueName: \"kubernetes.io/projected/dc8ad2ff-cb6b-46a5-8f17-caf033796331-kube-api-access-2z78b\") pod \"iptables-alerter-cjjws\" (UID: \"dc8ad2ff-cb6b-46a5-8f17-caf033796331\") " pod="openshift-network-operator/iptables-alerter-cjjws"
Sep 11 17:20:34.544523 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.543760    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4206b2fb-d03d-4827-9658-6dc11f47b6dc-etc-selinux\") pod \"aws-ebs-csi-driver-node-8zstf\" (UID: \"4206b2fb-d03d-4827-9658-6dc11f47b6dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf"
Sep 11 17:20:34.544523 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.543785    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsb7j\" (UniqueName: \"kubernetes.io/projected/4206b2fb-d03d-4827-9658-6dc11f47b6dc-kube-api-access-qsb7j\") pod \"aws-ebs-csi-driver-node-8zstf\" (UID: \"4206b2fb-d03d-4827-9658-6dc11f47b6dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf"
Sep 11 17:20:34.544523 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.543684    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pm66d"
Sep 11 17:20:34.544523 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.543809    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-etc-modprobe-d\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.544523 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.543834    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-etc-kubernetes\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.544523 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.543857    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-etc-sysctl-d\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.544523 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.543880    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-lib-modules\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.544523 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.543903    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-var-lib-kubelet\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.544523 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.543927    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsnv2\" (UniqueName: \"kubernetes.io/projected/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-kube-api-access-xsnv2\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.545642 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.543986    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g55t\" (UniqueName: \"kubernetes.io/projected/1203d22c-52e3-4681-9707-66ca35b9e72a-kube-api-access-2g55t\") pod \"network-metrics-daemon-sn2g7\" (UID: \"1203d22c-52e3-4681-9707-66ca35b9e72a\") " pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:20:34.545642 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.544040    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4206b2fb-d03d-4827-9658-6dc11f47b6dc-sys-fs\") pod \"aws-ebs-csi-driver-node-8zstf\" (UID: \"4206b2fb-d03d-4827-9658-6dc11f47b6dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf"
Sep 11 17:20:34.545642 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.544067    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc8ad2ff-cb6b-46a5-8f17-caf033796331-iptables-alerter-script\") pod \"iptables-alerter-cjjws\" (UID: \"dc8ad2ff-cb6b-46a5-8f17-caf033796331\") " pod="openshift-network-operator/iptables-alerter-cjjws"
Sep 11 17:20:34.545642 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.544101    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc8ad2ff-cb6b-46a5-8f17-caf033796331-host-slash\") pod \"iptables-alerter-cjjws\" (UID: \"dc8ad2ff-cb6b-46a5-8f17-caf033796331\") " pod="openshift-network-operator/iptables-alerter-cjjws"
Sep 11 17:20:34.545642 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.544125    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4206b2fb-d03d-4827-9658-6dc11f47b6dc-socket-dir\") pod \"aws-ebs-csi-driver-node-8zstf\" (UID: \"4206b2fb-d03d-4827-9658-6dc11f47b6dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf"
Sep 11 17:20:34.545642 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.544150    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4206b2fb-d03d-4827-9658-6dc11f47b6dc-registration-dir\") pod \"aws-ebs-csi-driver-node-8zstf\" (UID: \"4206b2fb-d03d-4827-9658-6dc11f47b6dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf"
Sep 11 17:20:34.545642 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.544174    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1203d22c-52e3-4681-9707-66ca35b9e72a-metrics-certs\") pod \"network-metrics-daemon-sn2g7\" (UID: \"1203d22c-52e3-4681-9707-66ca35b9e72a\") " pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:20:34.545642 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.544219    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-etc-sysctl-conf\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.545642 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.544243    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-etc-systemd\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.545642 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.544278    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-sys\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.545642 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.544312    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-etc-sysconfig\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.545642 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.544336    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-run\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.545642 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.544359    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4206b2fb-d03d-4827-9658-6dc11f47b6dc-device-dir\") pod \"aws-ebs-csi-driver-node-8zstf\" (UID: \"4206b2fb-d03d-4827-9658-6dc11f47b6dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf"
Sep 11 17:20:34.545642 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.544381    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-host\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.545642 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.544416    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-etc-tuned\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.547277 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.547257    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vmr96\""
Sep 11 17:20:34.547440 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.547401    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 17:20:34.547497 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.547466    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4mxdh"
Sep 11 17:20:34.547650 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.547633    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 17:20:34.548049 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.548030    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6665r"
Sep 11 17:20:34.549547 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.549184    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.551990 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.550056    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 17:20:34.551990 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.550541    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2fb2w\""
Sep 11 17:20:34.551990 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.550725    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 17:20:34.551990 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.551006    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 17:20:34.551990 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.551015    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 17:20:34.551990 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.551172    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 17:20:34.551990 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.551298    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 17:20:34.551990 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.551572    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 17:20:34.551990 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.551645    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6msk6\""
Sep 11 17:20:34.551990 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.551783    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 17:20:34.551990 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.551795    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 17:20:34.551990 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.551850    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nb4pr\""
Sep 11 17:20:34.570159 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.570136    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cwlkm"
Sep 11 17:20:34.578642 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.578617    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cwlkm"
Sep 11 17:20:34.635058 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.634985    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 17:20:34.644696 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.644664    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk9mr\" (UniqueName: \"kubernetes.io/projected/a8a65b90-c7a6-4272-a1e8-d60096120332-kube-api-access-tk9mr\") pod \"network-check-target-xjqcw\" (UID: \"a8a65b90-c7a6-4272-a1e8-d60096120332\") " pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:20:34.644840 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.644710    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.644840 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.644740    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-ovnkube-config\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.644840 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.644767    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-cnibin\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.644840 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.644790    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-host-run-multus-certs\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.644840 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.644824    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f5mg\" (UniqueName: \"kubernetes.io/projected/ae0ddecb-8d11-4989-8531-3028a8397e79-kube-api-access-8f5mg\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.645107 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.644855    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-etc-kubernetes\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.645107 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.644903    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-run-systemd\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.645107 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.644940    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-env-overrides\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.645107 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.644987    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-ovn-node-metrics-cert\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.645107 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645016    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-multus-cni-dir\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.645107 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645022    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-etc-kubernetes\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.645107 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645047    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc8ad2ff-cb6b-46a5-8f17-caf033796331-iptables-alerter-script\") pod \"iptables-alerter-cjjws\" (UID: \"dc8ad2ff-cb6b-46a5-8f17-caf033796331\") " pod="openshift-network-operator/iptables-alerter-cjjws"
Sep 11 17:20:34.645107 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645083    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4206b2fb-d03d-4827-9658-6dc11f47b6dc-socket-dir\") pod \"aws-ebs-csi-driver-node-8zstf\" (UID: \"4206b2fb-d03d-4827-9658-6dc11f47b6dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf"
Sep 11 17:20:34.645384 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645111    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97afce2d-e357-4c76-b513-2cb05322bbde-original-pull-secret\") pod \"global-pull-secret-syncer-82kzs\" (UID: \"97afce2d-e357-4c76-b513-2cb05322bbde\") " pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:20:34.645384 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645138    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-run-openvswitch\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.645384 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645165    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c49e0b27-b621-4879-b168-3962a9a32c8f-tmp-dir\") pod \"node-resolver-4mxdh\" (UID: \"c49e0b27-b621-4879-b168-3962a9a32c8f\") " pod="openshift-dns/node-resolver-4mxdh"
Sep 11 17:20:34.645384 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645191    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgms7\" (UniqueName: \"kubernetes.io/projected/efa4af6c-75c3-4ab4-ba6b-92d5a86b9025-kube-api-access-cgms7\") pod \"multus-additional-cni-plugins-6665r\" (UID: \"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025\") " pod="openshift-multus/multus-additional-cni-plugins-6665r"
Sep 11 17:20:34.645384 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645217    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-multus-socket-dir-parent\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.645384 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645241    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsm9n\" (UniqueName: \"kubernetes.io/projected/c11a0ba2-8654-4e79-a883-80d867a24d80-kube-api-access-vsm9n\") pod \"node-ca-q9dr2\" (UID: \"c11a0ba2-8654-4e79-a883-80d867a24d80\") " pod="openshift-image-registry/node-ca-q9dr2"
Sep 11 17:20:34.645384 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645265    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-systemd-units\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.645384 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645289    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-host-slash\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.645384 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645313    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c49e0b27-b621-4879-b168-3962a9a32c8f-hosts-file\") pod \"node-resolver-4mxdh\" (UID: \"c49e0b27-b621-4879-b168-3962a9a32c8f\") " pod="openshift-dns/node-resolver-4mxdh"
Sep 11 17:20:34.645384 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645339    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-host-run-k8s-cni-cncf-io\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.645384 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645364    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-etc-kubernetes\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.645739 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645397    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4206b2fb-d03d-4827-9658-6dc11f47b6dc-device-dir\") pod \"aws-ebs-csi-driver-node-8zstf\" (UID: \"4206b2fb-d03d-4827-9658-6dc11f47b6dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf"
Sep 11 17:20:34.645739 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645421    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-etc-tuned\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.645739 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645446    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-var-lib-openvswitch\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.645739 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645470    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-host-cni-netd\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.645739 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645495    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4206b2fb-d03d-4827-9658-6dc11f47b6dc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8zstf\" (UID: \"4206b2fb-d03d-4827-9658-6dc11f47b6dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf"
Sep 11 17:20:34.645739 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645519    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-tmp\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.645739 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645544    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-etc-openvswitch\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.645739 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645568    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-ovnkube-script-lib\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.645739 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645596    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/efa4af6c-75c3-4ab4-ba6b-92d5a86b9025-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6665r\" (UID: \"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025\") " pod="openshift-multus/multus-additional-cni-plugins-6665r"
Sep 11 17:20:34.645739 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645620    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-hostroot\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.645739 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645644    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z78b\" (UniqueName: \"kubernetes.io/projected/dc8ad2ff-cb6b-46a5-8f17-caf033796331-kube-api-access-2z78b\") pod \"iptables-alerter-cjjws\" (UID: \"dc8ad2ff-cb6b-46a5-8f17-caf033796331\") " pod="openshift-network-operator/iptables-alerter-cjjws"
Sep 11 17:20:34.645739 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645668    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4206b2fb-d03d-4827-9658-6dc11f47b6dc-etc-selinux\") pod \"aws-ebs-csi-driver-node-8zstf\" (UID: \"4206b2fb-d03d-4827-9658-6dc11f47b6dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf"
Sep 11 17:20:34.645739 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645687    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc8ad2ff-cb6b-46a5-8f17-caf033796331-iptables-alerter-script\") pod \"iptables-alerter-cjjws\" (UID: \"dc8ad2ff-cb6b-46a5-8f17-caf033796331\") " pod="openshift-network-operator/iptables-alerter-cjjws"
Sep 11 17:20:34.645739 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645701    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsb7j\" (UniqueName: \"kubernetes.io/projected/4206b2fb-d03d-4827-9658-6dc11f47b6dc-kube-api-access-qsb7j\") pod \"aws-ebs-csi-driver-node-8zstf\" (UID: \"4206b2fb-d03d-4827-9658-6dc11f47b6dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf"
Sep 11 17:20:34.645739 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645727    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-etc-modprobe-d\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.646290 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645761    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-etc-sysctl-d\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.646290 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645830    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-lib-modules\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.646290 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645856    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-var-lib-kubelet\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.646290 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645885    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsnv2\" (UniqueName: \"kubernetes.io/projected/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-kube-api-access-xsnv2\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.646290 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645889    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-etc-modprobe-d\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.646290 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.645922    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88h4q\" (UniqueName: \"kubernetes.io/projected/c49e0b27-b621-4879-b168-3962a9a32c8f-kube-api-access-88h4q\") pod \"node-resolver-4mxdh\" (UID: \"c49e0b27-b621-4879-b168-3962a9a32c8f\") " pod="openshift-dns/node-resolver-4mxdh"
Sep 11 17:20:34.646290 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.646010    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4206b2fb-d03d-4827-9658-6dc11f47b6dc-device-dir\") pod \"aws-ebs-csi-driver-node-8zstf\" (UID: \"4206b2fb-d03d-4827-9658-6dc11f47b6dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf"
Sep 11 17:20:34.646290 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.646068    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-etc-sysctl-d\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.646290 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.646290    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4206b2fb-d03d-4827-9658-6dc11f47b6dc-socket-dir\") pod \"aws-ebs-csi-driver-node-8zstf\" (UID: \"4206b2fb-d03d-4827-9658-6dc11f47b6dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf"
Sep 11 17:20:34.646582 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.646335    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/efa4af6c-75c3-4ab4-ba6b-92d5a86b9025-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6665r\" (UID: \"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025\") " pod="openshift-multus/multus-additional-cni-plugins-6665r"
Sep 11 17:20:34.646582 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.646362    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-host-var-lib-cni-multus\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.646582 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.646391    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g55t\" (UniqueName: \"kubernetes.io/projected/1203d22c-52e3-4681-9707-66ca35b9e72a-kube-api-access-2g55t\") pod \"network-metrics-daemon-sn2g7\" (UID: \"1203d22c-52e3-4681-9707-66ca35b9e72a\") " pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:20:34.646582 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.646417    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4206b2fb-d03d-4827-9658-6dc11f47b6dc-sys-fs\") pod \"aws-ebs-csi-driver-node-8zstf\" (UID: \"4206b2fb-d03d-4827-9658-6dc11f47b6dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf"
Sep 11 17:20:34.646582 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.646463    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97afce2d-e357-4c76-b513-2cb05322bbde-kubelet-config\") pod \"global-pull-secret-syncer-82kzs\" (UID: \"97afce2d-e357-4c76-b513-2cb05322bbde\") " pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:20:34.646843 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.646620    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4206b2fb-d03d-4827-9658-6dc11f47b6dc-etc-selinux\") pod \"aws-ebs-csi-driver-node-8zstf\" (UID: \"4206b2fb-d03d-4827-9658-6dc11f47b6dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf"
Sep 11 17:20:34.646843 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.646642    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-run-ovn\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.646843 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.646679    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 17:20:34.646843 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.646697    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4206b2fb-d03d-4827-9658-6dc11f47b6dc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8zstf\" (UID: \"4206b2fb-d03d-4827-9658-6dc11f47b6dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf"
Sep 11 17:20:34.647076 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647051    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-var-lib-kubelet\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.647076 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647055    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-lib-modules\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.647170 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647104    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4206b2fb-d03d-4827-9658-6dc11f47b6dc-sys-fs\") pod \"aws-ebs-csi-driver-node-8zstf\" (UID: \"4206b2fb-d03d-4827-9658-6dc11f47b6dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf"
Sep 11 17:20:34.647170 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647109    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efa4af6c-75c3-4ab4-ba6b-92d5a86b9025-system-cni-dir\") pod \"multus-additional-cni-plugins-6665r\" (UID: \"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025\") " pod="openshift-multus/multus-additional-cni-plugins-6665r"
Sep 11 17:20:34.647170 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647153    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ae0ddecb-8d11-4989-8531-3028a8397e79-cni-binary-copy\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.647300 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647198    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ae0ddecb-8d11-4989-8531-3028a8397e79-multus-daemon-config\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.647300 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647239    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc8ad2ff-cb6b-46a5-8f17-caf033796331-host-slash\") pod \"iptables-alerter-cjjws\" (UID: \"dc8ad2ff-cb6b-46a5-8f17-caf033796331\") " pod="openshift-network-operator/iptables-alerter-cjjws"
Sep 11 17:20:34.647300 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647287    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc8ad2ff-cb6b-46a5-8f17-caf033796331-host-slash\") pod \"iptables-alerter-cjjws\" (UID: \"dc8ad2ff-cb6b-46a5-8f17-caf033796331\") " pod="openshift-network-operator/iptables-alerter-cjjws"
Sep 11 17:20:34.647435 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647318    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4206b2fb-d03d-4827-9658-6dc11f47b6dc-registration-dir\") pod \"aws-ebs-csi-driver-node-8zstf\" (UID: \"4206b2fb-d03d-4827-9658-6dc11f47b6dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf"
Sep 11 17:20:34.647435 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647356    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-log-socket\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.647435 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647389    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/efa4af6c-75c3-4ab4-ba6b-92d5a86b9025-cni-binary-copy\") pod \"multus-additional-cni-plugins-6665r\" (UID: \"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025\") " pod="openshift-multus/multus-additional-cni-plugins-6665r"
Sep 11 17:20:34.647565 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647430    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4206b2fb-d03d-4827-9658-6dc11f47b6dc-registration-dir\") pod \"aws-ebs-csi-driver-node-8zstf\" (UID: \"4206b2fb-d03d-4827-9658-6dc11f47b6dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf"
Sep 11 17:20:34.647565 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647476    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-host-var-lib-cni-bin\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.647565 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647508    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-multus-conf-dir\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.647565 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647539    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1203d22c-52e3-4681-9707-66ca35b9e72a-metrics-certs\") pod \"network-metrics-daemon-sn2g7\" (UID: \"1203d22c-52e3-4681-9707-66ca35b9e72a\") " pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:20:34.647734 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647578    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-etc-sysctl-conf\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.647734 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647614    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-etc-systemd\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.647734 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:34.647634    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:20:34.647734 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647640    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-sys\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.647734 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647668    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97afce2d-e357-4c76-b513-2cb05322bbde-dbus\") pod \"global-pull-secret-syncer-82kzs\" (UID: \"97afce2d-e357-4c76-b513-2cb05322bbde\") " pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:20:34.647734 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:34.647713    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1203d22c-52e3-4681-9707-66ca35b9e72a-metrics-certs podName:1203d22c-52e3-4681-9707-66ca35b9e72a nodeName:}" failed. No retries permitted until 2026-09-11 17:20:35.147684323 +0000 UTC m=+2.951856723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1203d22c-52e3-4681-9707-66ca35b9e72a-metrics-certs") pod "network-metrics-daemon-sn2g7" (UID: "1203d22c-52e3-4681-9707-66ca35b9e72a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:20:34.647734 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647733    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-host-cni-bin\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.648460 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647750    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-sys\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.648460 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647760    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/efa4af6c-75c3-4ab4-ba6b-92d5a86b9025-cnibin\") pod \"multus-additional-cni-plugins-6665r\" (UID: \"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025\") " pod="openshift-multus/multus-additional-cni-plugins-6665r"
Sep 11 17:20:34.648460 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647736    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-etc-systemd\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.648460 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647733    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-etc-sysctl-conf\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.648460 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647795    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-host-run-netns\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.648460 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647827    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-etc-sysconfig\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.648460 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647852    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-run\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.648460 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647875    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-node-log\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.648460 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647902    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-226nx\" (UniqueName: \"kubernetes.io/projected/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-kube-api-access-226nx\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.648460 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647931    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-etc-sysconfig\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.648460 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647945    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-run\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.648460 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647943    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-system-cni-dir\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.648460 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.647986    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-host-var-lib-kubelet\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.648460 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.648033    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-host\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.648460 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.648065    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c11a0ba2-8654-4e79-a883-80d867a24d80-host\") pod \"node-ca-q9dr2\" (UID: \"c11a0ba2-8654-4e79-a883-80d867a24d80\") " pod="openshift-image-registry/node-ca-q9dr2"
Sep 11 17:20:34.648460 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.648071    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-host\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.648460 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.648089    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/efb29080-6e59-4db3-8a34-bf363f39eb9c-agent-certs\") pod \"konnectivity-agent-pm66d\" (UID: \"efb29080-6e59-4db3-8a34-bf363f39eb9c\") " pod="kube-system/konnectivity-agent-pm66d"
Sep 11 17:20:34.648460 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.648113    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-host-run-netns\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.649342 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.648134    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c11a0ba2-8654-4e79-a883-80d867a24d80-serviceca\") pod \"node-ca-q9dr2\" (UID: \"c11a0ba2-8654-4e79-a883-80d867a24d80\") " pod="openshift-image-registry/node-ca-q9dr2"
Sep 11 17:20:34.649342 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.648159    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/efb29080-6e59-4db3-8a34-bf363f39eb9c-konnectivity-ca\") pod \"konnectivity-agent-pm66d\" (UID: \"efb29080-6e59-4db3-8a34-bf363f39eb9c\") " pod="kube-system/konnectivity-agent-pm66d"
Sep 11 17:20:34.649342 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.648180    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-host-kubelet\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.649342 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.648200    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-host-run-ovn-kubernetes\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.649342 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.648222    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/efa4af6c-75c3-4ab4-ba6b-92d5a86b9025-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6665r\" (UID: \"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025\") " pod="openshift-multus/multus-additional-cni-plugins-6665r"
Sep 11 17:20:34.653043 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.652434    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-tmp\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.653372 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.653349    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-etc-tuned\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.655449 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.655426    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g55t\" (UniqueName: \"kubernetes.io/projected/1203d22c-52e3-4681-9707-66ca35b9e72a-kube-api-access-2g55t\") pod \"network-metrics-daemon-sn2g7\" (UID: \"1203d22c-52e3-4681-9707-66ca35b9e72a\") " pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:20:34.655957 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.655914    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsnv2\" (UniqueName: \"kubernetes.io/projected/2db2c929-0e48-4ceb-83c3-2cf8a2c087e0-kube-api-access-xsnv2\") pod \"tuned-bcdn9\" (UID: \"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0\") " pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.660992 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.660945    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z78b\" (UniqueName: \"kubernetes.io/projected/dc8ad2ff-cb6b-46a5-8f17-caf033796331-kube-api-access-2z78b\") pod \"iptables-alerter-cjjws\" (UID: \"dc8ad2ff-cb6b-46a5-8f17-caf033796331\") " pod="openshift-network-operator/iptables-alerter-cjjws"
Sep 11 17:20:34.661408 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.661391    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsb7j\" (UniqueName: \"kubernetes.io/projected/4206b2fb-d03d-4827-9658-6dc11f47b6dc-kube-api-access-qsb7j\") pod \"aws-ebs-csi-driver-node-8zstf\" (UID: \"4206b2fb-d03d-4827-9658-6dc11f47b6dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf"
Sep 11 17:20:34.715530 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.715466    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-89.ec2.internal" event={"ID":"537070a60bc30f06ce6f629121ed76a8","Type":"ContainerStarted","Data":"ecbbbf5be0081aa5d38caeaaac288bee45b7d10682f19b9fef6d8935a9486c60"}
Sep 11 17:20:34.717425 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.717392    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-89.ec2.internal" event={"ID":"29212e0b773c1bb0ac3a9172435b9a0f","Type":"ContainerStarted","Data":"4aecc2bd0c3d1cb17a96962286b06a602b8218f04a7323e5d905f3940a8834e9"}
Sep 11 17:20:34.749382 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.749296    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ae0ddecb-8d11-4989-8531-3028a8397e79-multus-daemon-config\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.749382 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.749342    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-log-socket\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.749382 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.749369    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/efa4af6c-75c3-4ab4-ba6b-92d5a86b9025-cni-binary-copy\") pod \"multus-additional-cni-plugins-6665r\" (UID: \"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025\") " pod="openshift-multus/multus-additional-cni-plugins-6665r"
Sep 11 17:20:34.749664 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.749398    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-host-var-lib-cni-bin\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.749664 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.749432    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-multus-conf-dir\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.749664 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.749472    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97afce2d-e357-4c76-b513-2cb05322bbde-dbus\") pod \"global-pull-secret-syncer-82kzs\" (UID: \"97afce2d-e357-4c76-b513-2cb05322bbde\") " pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:20:34.749664 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.749496    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-host-cni-bin\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.749664 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.749520    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/efa4af6c-75c3-4ab4-ba6b-92d5a86b9025-cnibin\") pod \"multus-additional-cni-plugins-6665r\" (UID: \"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025\") " pod="openshift-multus/multus-additional-cni-plugins-6665r"
Sep 11 17:20:34.749664 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.749546    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-host-run-netns\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.749664 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.749578    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-node-log\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.749664 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.749602    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-226nx\" (UniqueName: \"kubernetes.io/projected/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-kube-api-access-226nx\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.749664 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.749627    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-system-cni-dir\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.749664 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.749650    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-host-var-lib-kubelet\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.750186 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.749676    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c11a0ba2-8654-4e79-a883-80d867a24d80-host\") pod \"node-ca-q9dr2\" (UID: \"c11a0ba2-8654-4e79-a883-80d867a24d80\") " pod="openshift-image-registry/node-ca-q9dr2"
Sep 11 17:20:34.750186 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.749703    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/efb29080-6e59-4db3-8a34-bf363f39eb9c-agent-certs\") pod \"konnectivity-agent-pm66d\" (UID: \"efb29080-6e59-4db3-8a34-bf363f39eb9c\") " pod="kube-system/konnectivity-agent-pm66d"
Sep 11 17:20:34.750186 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.749729    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-host-run-netns\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.750186 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.749753    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c11a0ba2-8654-4e79-a883-80d867a24d80-serviceca\") pod \"node-ca-q9dr2\" (UID: \"c11a0ba2-8654-4e79-a883-80d867a24d80\") " pod="openshift-image-registry/node-ca-q9dr2"
Sep 11 17:20:34.750186 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.749778    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/efb29080-6e59-4db3-8a34-bf363f39eb9c-konnectivity-ca\") pod \"konnectivity-agent-pm66d\" (UID: \"efb29080-6e59-4db3-8a34-bf363f39eb9c\") " pod="kube-system/konnectivity-agent-pm66d"
Sep 11 17:20:34.750186 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.749803    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-host-kubelet\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.750186 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.749840    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-host-run-ovn-kubernetes\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.750186 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.749868    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/efa4af6c-75c3-4ab4-ba6b-92d5a86b9025-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6665r\" (UID: \"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025\") " pod="openshift-multus/multus-additional-cni-plugins-6665r"
Sep 11 17:20:34.750186 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.749913    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk9mr\" (UniqueName: \"kubernetes.io/projected/a8a65b90-c7a6-4272-a1e8-d60096120332-kube-api-access-tk9mr\") pod \"network-check-target-xjqcw\" (UID: \"a8a65b90-c7a6-4272-a1e8-d60096120332\") " pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:20:34.750186 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.749954    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.750186 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.749982    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-ovnkube-config\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.750186 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750008    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-cnibin\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.750186 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750014    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ae0ddecb-8d11-4989-8531-3028a8397e79-multus-daemon-config\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.750186 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750033    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-host-run-multus-certs\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.750186 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750082    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f5mg\" (UniqueName: \"kubernetes.io/projected/ae0ddecb-8d11-4989-8531-3028a8397e79-kube-api-access-8f5mg\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.750186 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750090    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-host-run-multus-certs\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.750186 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750117    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-run-systemd\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.751091 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750145    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c11a0ba2-8654-4e79-a883-80d867a24d80-host\") pod \"node-ca-q9dr2\" (UID: \"c11a0ba2-8654-4e79-a883-80d867a24d80\") " pod="openshift-image-registry/node-ca-q9dr2"
Sep 11 17:20:34.751091 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750147    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-env-overrides\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.751091 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750155    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-log-socket\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.751091 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750175    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-ovn-node-metrics-cert\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.751091 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750184    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-host-var-lib-kubelet\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.751091 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750202    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-multus-cni-dir\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.751091 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750234    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97afce2d-e357-4c76-b513-2cb05322bbde-original-pull-secret\") pod \"global-pull-secret-syncer-82kzs\" (UID: \"97afce2d-e357-4c76-b513-2cb05322bbde\") " pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:20:34.751091 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750263    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-run-openvswitch\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.751091 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750289    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c49e0b27-b621-4879-b168-3962a9a32c8f-tmp-dir\") pod \"node-resolver-4mxdh\" (UID: \"c49e0b27-b621-4879-b168-3962a9a32c8f\") " pod="openshift-dns/node-resolver-4mxdh"
Sep 11 17:20:34.751091 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750316    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgms7\" (UniqueName: \"kubernetes.io/projected/efa4af6c-75c3-4ab4-ba6b-92d5a86b9025-kube-api-access-cgms7\") pod \"multus-additional-cni-plugins-6665r\" (UID: \"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025\") " pod="openshift-multus/multus-additional-cni-plugins-6665r"
Sep 11 17:20:34.751091 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750344    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-multus-socket-dir-parent\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.751091 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750394    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsm9n\" (UniqueName: \"kubernetes.io/projected/c11a0ba2-8654-4e79-a883-80d867a24d80-kube-api-access-vsm9n\") pod \"node-ca-q9dr2\" (UID: \"c11a0ba2-8654-4e79-a883-80d867a24d80\") " pod="openshift-image-registry/node-ca-q9dr2"
Sep 11 17:20:34.751091 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750422    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-systemd-units\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.751091 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750449    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-host-slash\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.751091 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750475    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c49e0b27-b621-4879-b168-3962a9a32c8f-hosts-file\") pod \"node-resolver-4mxdh\" (UID: \"c49e0b27-b621-4879-b168-3962a9a32c8f\") " pod="openshift-dns/node-resolver-4mxdh"
Sep 11 17:20:34.751091 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750500    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-host-run-k8s-cni-cncf-io\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.751091 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750528    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-etc-kubernetes\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.751091 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750558    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-var-lib-openvswitch\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.751996 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750570    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/efa4af6c-75c3-4ab4-ba6b-92d5a86b9025-cni-binary-copy\") pod \"multus-additional-cni-plugins-6665r\" (UID: \"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025\") " pod="openshift-multus/multus-additional-cni-plugins-6665r"
Sep 11 17:20:34.751996 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750585    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-host-cni-netd\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.751996 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750612    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-host-var-lib-cni-bin\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.751996 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750617    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-etc-openvswitch\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.751996 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750637    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-multus-conf-dir\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.751996 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750645    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-ovnkube-script-lib\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.751996 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750670    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/efa4af6c-75c3-4ab4-ba6b-92d5a86b9025-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6665r\" (UID: \"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025\") " pod="openshift-multus/multus-additional-cni-plugins-6665r"
Sep 11 17:20:34.751996 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750695    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-hostroot\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.751996 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750738    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97afce2d-e357-4c76-b513-2cb05322bbde-dbus\") pod \"global-pull-secret-syncer-82kzs\" (UID: \"97afce2d-e357-4c76-b513-2cb05322bbde\") " pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:20:34.751996 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750737    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88h4q\" (UniqueName: \"kubernetes.io/projected/c49e0b27-b621-4879-b168-3962a9a32c8f-kube-api-access-88h4q\") pod \"node-resolver-4mxdh\" (UID: \"c49e0b27-b621-4879-b168-3962a9a32c8f\") " pod="openshift-dns/node-resolver-4mxdh"
Sep 11 17:20:34.751996 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750769    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/efa4af6c-75c3-4ab4-ba6b-92d5a86b9025-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6665r\" (UID: \"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025\") " pod="openshift-multus/multus-additional-cni-plugins-6665r"
Sep 11 17:20:34.751996 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750799    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-host-var-lib-cni-multus\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.751996 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750826    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c49e0b27-b621-4879-b168-3962a9a32c8f-tmp-dir\") pod \"node-resolver-4mxdh\" (UID: \"c49e0b27-b621-4879-b168-3962a9a32c8f\") " pod="openshift-dns/node-resolver-4mxdh"
Sep 11 17:20:34.751996 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750832    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97afce2d-e357-4c76-b513-2cb05322bbde-kubelet-config\") pod \"global-pull-secret-syncer-82kzs\" (UID: \"97afce2d-e357-4c76-b513-2cb05322bbde\") " pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:20:34.751996 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750869    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97afce2d-e357-4c76-b513-2cb05322bbde-kubelet-config\") pod \"global-pull-secret-syncer-82kzs\" (UID: \"97afce2d-e357-4c76-b513-2cb05322bbde\") " pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:20:34.751996 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750888    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-run-ovn\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.751996 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750909    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-host-cni-bin\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.752852 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750943    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efa4af6c-75c3-4ab4-ba6b-92d5a86b9025-system-cni-dir\") pod \"multus-additional-cni-plugins-6665r\" (UID: \"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025\") " pod="openshift-multus/multus-additional-cni-plugins-6665r"
Sep 11 17:20:34.752852 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750971    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ae0ddecb-8d11-4989-8531-3028a8397e79-cni-binary-copy\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.752852 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.750996    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/efa4af6c-75c3-4ab4-ba6b-92d5a86b9025-cnibin\") pod \"multus-additional-cni-plugins-6665r\" (UID: \"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025\") " pod="openshift-multus/multus-additional-cni-plugins-6665r"
Sep 11 17:20:34.752852 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.751059    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-multus-socket-dir-parent\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.752852 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.751126    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-host-cni-netd\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.752852 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.751143    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-multus-cni-dir\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.752852 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.751144    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-hostroot\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.752852 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.751170    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-run-systemd\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.752852 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.751246    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/efa4af6c-75c3-4ab4-ba6b-92d5a86b9025-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6665r\" (UID: \"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025\") " pod="openshift-multus/multus-additional-cni-plugins-6665r"
Sep 11 17:20:34.752852 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.751302    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-host-slash\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.752852 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.751343    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-etc-openvswitch\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.752852 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.751378    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-var-lib-openvswitch\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.752852 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.751415    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efa4af6c-75c3-4ab4-ba6b-92d5a86b9025-system-cni-dir\") pod \"multus-additional-cni-plugins-6665r\" (UID: \"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025\") " pod="openshift-multus/multus-additional-cni-plugins-6665r"
Sep 11 17:20:34.752852 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.751424    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-host-run-ovn-kubernetes\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.752852 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.751470    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.752852 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.751487    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-host-run-netns\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.752852 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.751760    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ae0ddecb-8d11-4989-8531-3028a8397e79-cni-binary-copy\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.752852 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.751818    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-host-run-netns\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.753616 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.751839    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-ovnkube-script-lib\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.753616 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.751856    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-node-log\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.753616 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.751902    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c49e0b27-b621-4879-b168-3962a9a32c8f-hosts-file\") pod \"node-resolver-4mxdh\" (UID: \"c49e0b27-b621-4879-b168-3962a9a32c8f\") " pod="openshift-dns/node-resolver-4mxdh"
Sep 11 17:20:34.753616 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.751921    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c11a0ba2-8654-4e79-a883-80d867a24d80-serviceca\") pod \"node-ca-q9dr2\" (UID: \"c11a0ba2-8654-4e79-a883-80d867a24d80\") " pod="openshift-image-registry/node-ca-q9dr2"
Sep 11 17:20:34.753616 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.751957    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-host-kubelet\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.753616 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.751992    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/efa4af6c-75c3-4ab4-ba6b-92d5a86b9025-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6665r\" (UID: \"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025\") " pod="openshift-multus/multus-additional-cni-plugins-6665r"
Sep 11 17:20:34.753616 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.751998    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-run-openvswitch\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.753616 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:34.752214    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:20:34.753616 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.752218    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-system-cni-dir\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.753616 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:34.752286    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97afce2d-e357-4c76-b513-2cb05322bbde-original-pull-secret podName:97afce2d-e357-4c76-b513-2cb05322bbde nodeName:}" failed. No retries permitted until 2026-09-11 17:20:35.252267551 +0000 UTC m=+3.056439964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97afce2d-e357-4c76-b513-2cb05322bbde-original-pull-secret") pod "global-pull-secret-syncer-82kzs" (UID: "97afce2d-e357-4c76-b513-2cb05322bbde") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:20:34.753616 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.752522    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-run-ovn\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.753616 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.752535    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-ovnkube-config\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.753616 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.752629    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-cnibin\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.753616 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.752641    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-etc-kubernetes\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.753616 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.752696    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-systemd-units\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.753616 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.752745    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-host-run-k8s-cni-cncf-io\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.753616 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.752795    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ae0ddecb-8d11-4989-8531-3028a8397e79-host-var-lib-cni-multus\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.754457 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.752822    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/efa4af6c-75c3-4ab4-ba6b-92d5a86b9025-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6665r\" (UID: \"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025\") " pod="openshift-multus/multus-additional-cni-plugins-6665r"
Sep 11 17:20:34.754457 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.752870    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/efb29080-6e59-4db3-8a34-bf363f39eb9c-konnectivity-ca\") pod \"konnectivity-agent-pm66d\" (UID: \"efb29080-6e59-4db3-8a34-bf363f39eb9c\") " pod="kube-system/konnectivity-agent-pm66d"
Sep 11 17:20:34.754457 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.753038    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-env-overrides\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.754457 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.753210    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/efb29080-6e59-4db3-8a34-bf363f39eb9c-agent-certs\") pod \"konnectivity-agent-pm66d\" (UID: \"efb29080-6e59-4db3-8a34-bf363f39eb9c\") " pod="kube-system/konnectivity-agent-pm66d"
Sep 11 17:20:34.754457 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.754063    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-ovn-node-metrics-cert\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.761686 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.761641    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88h4q\" (UniqueName: \"kubernetes.io/projected/c49e0b27-b621-4879-b168-3962a9a32c8f-kube-api-access-88h4q\") pod \"node-resolver-4mxdh\" (UID: \"c49e0b27-b621-4879-b168-3962a9a32c8f\") " pod="openshift-dns/node-resolver-4mxdh"
Sep 11 17:20:34.762718 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:34.761880    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:20:34.762718 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:34.761903    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:20:34.762718 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:34.761920    2560 projected.go:196] Error preparing data for projected volume kube-api-access-tk9mr for pod openshift-network-diagnostics/network-check-target-xjqcw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:20:34.762718 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:34.762139    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8a65b90-c7a6-4272-a1e8-d60096120332-kube-api-access-tk9mr podName:a8a65b90-c7a6-4272-a1e8-d60096120332 nodeName:}" failed. No retries permitted until 2026-09-11 17:20:35.262118069 +0000 UTC m=+3.066290483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tk9mr" (UniqueName: "kubernetes.io/projected/a8a65b90-c7a6-4272-a1e8-d60096120332-kube-api-access-tk9mr") pod "network-check-target-xjqcw" (UID: "a8a65b90-c7a6-4272-a1e8-d60096120332") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:20:34.765877 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.765614    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f5mg\" (UniqueName: \"kubernetes.io/projected/ae0ddecb-8d11-4989-8531-3028a8397e79-kube-api-access-8f5mg\") pod \"multus-c9pzn\" (UID: \"ae0ddecb-8d11-4989-8531-3028a8397e79\") " pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:34.766724 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.766696    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsm9n\" (UniqueName: \"kubernetes.io/projected/c11a0ba2-8654-4e79-a883-80d867a24d80-kube-api-access-vsm9n\") pod \"node-ca-q9dr2\" (UID: \"c11a0ba2-8654-4e79-a883-80d867a24d80\") " pod="openshift-image-registry/node-ca-q9dr2"
Sep 11 17:20:34.767901 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.767847    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-226nx\" (UniqueName: \"kubernetes.io/projected/1a9b26d3-c512-4ea8-a59a-a1a718b2026f-kube-api-access-226nx\") pod \"ovnkube-node-8dnzt\" (UID: \"1a9b26d3-c512-4ea8-a59a-a1a718b2026f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.768758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.768735    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgms7\" (UniqueName: \"kubernetes.io/projected/efa4af6c-75c3-4ab4-ba6b-92d5a86b9025-kube-api-access-cgms7\") pod \"multus-additional-cni-plugins-6665r\" (UID: \"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025\") " pod="openshift-multus/multus-additional-cni-plugins-6665r"
Sep 11 17:20:34.849179 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.849145    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cjjws"
Sep 11 17:20:34.857376 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.857352    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf"
Sep 11 17:20:34.869312 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.869286    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bcdn9"
Sep 11 17:20:34.875302 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.875270    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q9dr2"
Sep 11 17:20:34.885471 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.885450    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:34.894702 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.894682    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pm66d"
Sep 11 17:20:34.904597 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.904550    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4mxdh"
Sep 11 17:20:34.915586 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.915565    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6665r"
Sep 11 17:20:34.923562 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:34.923538    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c9pzn"
Sep 11 17:20:35.154814 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:35.154780    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1203d22c-52e3-4681-9707-66ca35b9e72a-metrics-certs\") pod \"network-metrics-daemon-sn2g7\" (UID: \"1203d22c-52e3-4681-9707-66ca35b9e72a\") " pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:20:35.154999 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:35.154978    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:20:35.155070 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:35.155058    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1203d22c-52e3-4681-9707-66ca35b9e72a-metrics-certs podName:1203d22c-52e3-4681-9707-66ca35b9e72a nodeName:}" failed. No retries permitted until 2026-09-11 17:20:36.155035834 +0000 UTC m=+3.959208254 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1203d22c-52e3-4681-9707-66ca35b9e72a-metrics-certs") pod "network-metrics-daemon-sn2g7" (UID: "1203d22c-52e3-4681-9707-66ca35b9e72a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:20:35.255191 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:35.255154    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97afce2d-e357-4c76-b513-2cb05322bbde-original-pull-secret\") pod \"global-pull-secret-syncer-82kzs\" (UID: \"97afce2d-e357-4c76-b513-2cb05322bbde\") " pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:20:35.255359 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:35.255311    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:20:35.255434 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:35.255382    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97afce2d-e357-4c76-b513-2cb05322bbde-original-pull-secret podName:97afce2d-e357-4c76-b513-2cb05322bbde nodeName:}" failed. No retries permitted until 2026-09-11 17:20:36.255361118 +0000 UTC m=+4.059533533 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97afce2d-e357-4c76-b513-2cb05322bbde-original-pull-secret") pod "global-pull-secret-syncer-82kzs" (UID: "97afce2d-e357-4c76-b513-2cb05322bbde") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:20:35.356249 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:35.356117    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:20:35.356249 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:35.356156    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk9mr\" (UniqueName: \"kubernetes.io/projected/a8a65b90-c7a6-4272-a1e8-d60096120332-kube-api-access-tk9mr\") pod \"network-check-target-xjqcw\" (UID: \"a8a65b90-c7a6-4272-a1e8-d60096120332\") " pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:20:35.356491 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:35.356400    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:20:35.356491 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:35.356423    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:20:35.356491 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:35.356440    2560 projected.go:196] Error preparing data for projected volume kube-api-access-tk9mr for pod openshift-network-diagnostics/network-check-target-xjqcw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:20:35.356597 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:35.356510    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8a65b90-c7a6-4272-a1e8-d60096120332-kube-api-access-tk9mr podName:a8a65b90-c7a6-4272-a1e8-d60096120332 nodeName:}" failed. No retries permitted until 2026-09-11 17:20:36.356491747 +0000 UTC m=+4.160664157 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tk9mr" (UniqueName: "kubernetes.io/projected/a8a65b90-c7a6-4272-a1e8-d60096120332-kube-api-access-tk9mr") pod "network-check-target-xjqcw" (UID: "a8a65b90-c7a6-4272-a1e8-d60096120332") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:20:35.581185 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:35.581114    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 17:15:35 +0000 UTC" deadline="2028-03-03 09:45:09.277750138 +0000 UTC"
Sep 11 17:20:35.581185 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:35.581180    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12928h24m33.696574857s"
Sep 11 17:20:35.709778 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:35.709739    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:20:35.709964 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:35.709757    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:20:35.709964 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:35.709877    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82kzs" podUID="97afce2d-e357-4c76-b513-2cb05322bbde"
Sep 11 17:20:35.710104 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:35.709964    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn2g7" podUID="1203d22c-52e3-4681-9707-66ca35b9e72a"
Sep 11 17:20:35.844428 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:35.844398    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4206b2fb_d03d_4827_9658_6dc11f47b6dc.slice/crio-702022b801a92c8bd5dd1f1f7eeae8f85b0c4426029fd922b8131e1ade69430e WatchSource:0}: Error finding container 702022b801a92c8bd5dd1f1f7eeae8f85b0c4426029fd922b8131e1ade69430e: Status 404 returned error can't find the container with id 702022b801a92c8bd5dd1f1f7eeae8f85b0c4426029fd922b8131e1ade69430e
Sep 11 17:20:35.846452 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:35.846401    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a9b26d3_c512_4ea8_a59a_a1a718b2026f.slice/crio-720ac5238f205db7cfde6769d20a2b6d75f75f1363d60680fa9dca62377ceb18 WatchSource:0}: Error finding container 720ac5238f205db7cfde6769d20a2b6d75f75f1363d60680fa9dca62377ceb18: Status 404 returned error can't find the container with id 720ac5238f205db7cfde6769d20a2b6d75f75f1363d60680fa9dca62377ceb18
Sep 11 17:20:35.847752 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:35.847621    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc49e0b27_b621_4879_b168_3962a9a32c8f.slice/crio-e5a38ed3a348b95cf67c914ac393552bd043586f0d1fb04cf4beff83775b226a WatchSource:0}: Error finding container e5a38ed3a348b95cf67c914ac393552bd043586f0d1fb04cf4beff83775b226a: Status 404 returned error can't find the container with id e5a38ed3a348b95cf67c914ac393552bd043586f0d1fb04cf4beff83775b226a
Sep 11 17:20:35.850868 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:35.850838    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db2c929_0e48_4ceb_83c3_2cf8a2c087e0.slice/crio-6cf9b41afc4246a2fbe76ae8a68ce20720f45c221c5ff0ec513d521e36250d32 WatchSource:0}: Error finding container 6cf9b41afc4246a2fbe76ae8a68ce20720f45c221c5ff0ec513d521e36250d32: Status 404 returned error can't find the container with id 6cf9b41afc4246a2fbe76ae8a68ce20720f45c221c5ff0ec513d521e36250d32
Sep 11 17:20:35.851928 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:35.851902    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa4af6c_75c3_4ab4_ba6b_92d5a86b9025.slice/crio-c077a5a132e760e256fd110141dbaa88e350b0d5e59576bd8dddda96a7fb3871 WatchSource:0}: Error finding container c077a5a132e760e256fd110141dbaa88e350b0d5e59576bd8dddda96a7fb3871: Status 404 returned error can't find the container with id c077a5a132e760e256fd110141dbaa88e350b0d5e59576bd8dddda96a7fb3871
Sep 11 17:20:35.853321 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:35.853287    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc11a0ba2_8654_4e79_a883_80d867a24d80.slice/crio-3b69a05116f5174fa86bdfdc8e50e6661bf379185c8d09caa736b85b01cd1e64 WatchSource:0}: Error finding container 3b69a05116f5174fa86bdfdc8e50e6661bf379185c8d09caa736b85b01cd1e64: Status 404 returned error can't find the container with id 3b69a05116f5174fa86bdfdc8e50e6661bf379185c8d09caa736b85b01cd1e64
Sep 11 17:20:35.854351 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:35.854318    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc8ad2ff_cb6b_46a5_8f17_caf033796331.slice/crio-fadc51a16956fbd2dc84e38e332346d167aa7a7e33cfd2cabcd09a33c3e3952a WatchSource:0}: Error finding container fadc51a16956fbd2dc84e38e332346d167aa7a7e33cfd2cabcd09a33c3e3952a: Status 404 returned error can't find the container with id fadc51a16956fbd2dc84e38e332346d167aa7a7e33cfd2cabcd09a33c3e3952a
Sep 11 17:20:35.855471 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:35.855427    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae0ddecb_8d11_4989_8531_3028a8397e79.slice/crio-fa5c226bcc3cdeae1941ffb5613b2e43452b28db361f0ab6ff87c479446eb445 WatchSource:0}: Error finding container fa5c226bcc3cdeae1941ffb5613b2e43452b28db361f0ab6ff87c479446eb445: Status 404 returned error can't find the container with id fa5c226bcc3cdeae1941ffb5613b2e43452b28db361f0ab6ff87c479446eb445
Sep 11 17:20:35.857890 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:20:35.857844    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb29080_6e59_4db3_8a34_bf363f39eb9c.slice/crio-c72ce41bba945fb02ec4802c37115c7cfc42eef7352cb488c3474112ea447ca6 WatchSource:0}: Error finding container c72ce41bba945fb02ec4802c37115c7cfc42eef7352cb488c3474112ea447ca6: Status 404 returned error can't find the container with id c72ce41bba945fb02ec4802c37115c7cfc42eef7352cb488c3474112ea447ca6
Sep 11 17:20:36.162416 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:36.162386    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1203d22c-52e3-4681-9707-66ca35b9e72a-metrics-certs\") pod \"network-metrics-daemon-sn2g7\" (UID: \"1203d22c-52e3-4681-9707-66ca35b9e72a\") " pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:20:36.162540 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:36.162513    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:20:36.162598 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:36.162572    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1203d22c-52e3-4681-9707-66ca35b9e72a-metrics-certs podName:1203d22c-52e3-4681-9707-66ca35b9e72a nodeName:}" failed. No retries permitted until 2026-09-11 17:20:38.16255934 +0000 UTC m=+5.966731738 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1203d22c-52e3-4681-9707-66ca35b9e72a-metrics-certs") pod "network-metrics-daemon-sn2g7" (UID: "1203d22c-52e3-4681-9707-66ca35b9e72a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:20:36.263188 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:36.263145    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97afce2d-e357-4c76-b513-2cb05322bbde-original-pull-secret\") pod \"global-pull-secret-syncer-82kzs\" (UID: \"97afce2d-e357-4c76-b513-2cb05322bbde\") " pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:20:36.263360 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:36.263303    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:20:36.263405 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:36.263373    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97afce2d-e357-4c76-b513-2cb05322bbde-original-pull-secret podName:97afce2d-e357-4c76-b513-2cb05322bbde nodeName:}" failed. No retries permitted until 2026-09-11 17:20:38.263355883 +0000 UTC m=+6.067528282 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97afce2d-e357-4c76-b513-2cb05322bbde-original-pull-secret") pod "global-pull-secret-syncer-82kzs" (UID: "97afce2d-e357-4c76-b513-2cb05322bbde") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:20:36.364687 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:36.364589    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk9mr\" (UniqueName: \"kubernetes.io/projected/a8a65b90-c7a6-4272-a1e8-d60096120332-kube-api-access-tk9mr\") pod \"network-check-target-xjqcw\" (UID: \"a8a65b90-c7a6-4272-a1e8-d60096120332\") " pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:20:36.364858 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:36.364792    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:20:36.364858 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:36.364816    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:20:36.364858 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:36.364830    2560 projected.go:196] Error preparing data for projected volume kube-api-access-tk9mr for pod openshift-network-diagnostics/network-check-target-xjqcw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:20:36.365103 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:36.364894    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8a65b90-c7a6-4272-a1e8-d60096120332-kube-api-access-tk9mr podName:a8a65b90-c7a6-4272-a1e8-d60096120332 nodeName:}" failed. No retries permitted until 2026-09-11 17:20:38.364873146 +0000 UTC m=+6.169045585 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tk9mr" (UniqueName: "kubernetes.io/projected/a8a65b90-c7a6-4272-a1e8-d60096120332-kube-api-access-tk9mr") pod "network-check-target-xjqcw" (UID: "a8a65b90-c7a6-4272-a1e8-d60096120332") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:20:36.581399 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:36.581355    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 17:15:35 +0000 UTC" deadline="2028-02-06 07:07:59.103779603 +0000 UTC"
Sep 11 17:20:36.581399 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:36.581397    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12301h47m22.522388824s"
Sep 11 17:20:36.710206 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:36.710122    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:20:36.710375 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:36.710278    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjqcw" podUID="a8a65b90-c7a6-4272-a1e8-d60096120332"
Sep 11 17:20:36.727218 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:36.727180    2560 generic.go:356] "Generic (PLEG): container finished" podID="29212e0b773c1bb0ac3a9172435b9a0f" containerID="79786b05f00d8909272a85ab0da9012e5d4cc9b8db49b186bea3bb412a1ab8e7" exitCode=0
Sep 11 17:20:36.727447 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:36.727316    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-89.ec2.internal" event={"ID":"29212e0b773c1bb0ac3a9172435b9a0f","Type":"ContainerDied","Data":"79786b05f00d8909272a85ab0da9012e5d4cc9b8db49b186bea3bb412a1ab8e7"}
Sep 11 17:20:36.730794 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:36.730763    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pm66d" event={"ID":"efb29080-6e59-4db3-8a34-bf363f39eb9c","Type":"ContainerStarted","Data":"c72ce41bba945fb02ec4802c37115c7cfc42eef7352cb488c3474112ea447ca6"}
Sep 11 17:20:36.751795 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:36.751762    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q9dr2" event={"ID":"c11a0ba2-8654-4e79-a883-80d867a24d80","Type":"ContainerStarted","Data":"3b69a05116f5174fa86bdfdc8e50e6661bf379185c8d09caa736b85b01cd1e64"}
Sep 11 17:20:36.757867 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:36.757807    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cjjws" event={"ID":"dc8ad2ff-cb6b-46a5-8f17-caf033796331","Type":"ContainerStarted","Data":"fadc51a16956fbd2dc84e38e332346d167aa7a7e33cfd2cabcd09a33c3e3952a"}
Sep 11 17:20:36.772199 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:36.772131    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6665r" event={"ID":"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025","Type":"ContainerStarted","Data":"c077a5a132e760e256fd110141dbaa88e350b0d5e59576bd8dddda96a7fb3871"}
Sep 11 17:20:36.784500 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:36.784413    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bcdn9" event={"ID":"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0","Type":"ContainerStarted","Data":"6cf9b41afc4246a2fbe76ae8a68ce20720f45c221c5ff0ec513d521e36250d32"}
Sep 11 17:20:36.812127 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:36.812029    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4mxdh" event={"ID":"c49e0b27-b621-4879-b168-3962a9a32c8f","Type":"ContainerStarted","Data":"e5a38ed3a348b95cf67c914ac393552bd043586f0d1fb04cf4beff83775b226a"}
Sep 11 17:20:36.832966 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:36.832923    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-89.ec2.internal" event={"ID":"537070a60bc30f06ce6f629121ed76a8","Type":"ContainerStarted","Data":"4d46bdac9f650e0685274f6a2cad6f08320e583e0afb0b60ba16b79e117187be"}
Sep 11 17:20:36.857354 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:36.857319    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c9pzn" event={"ID":"ae0ddecb-8d11-4989-8531-3028a8397e79","Type":"ContainerStarted","Data":"fa5c226bcc3cdeae1941ffb5613b2e43452b28db361f0ab6ff87c479446eb445"}
Sep 11 17:20:36.869897 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:36.869774    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt" event={"ID":"1a9b26d3-c512-4ea8-a59a-a1a718b2026f","Type":"ContainerStarted","Data":"720ac5238f205db7cfde6769d20a2b6d75f75f1363d60680fa9dca62377ceb18"}
Sep 11 17:20:36.877493 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:36.877403    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf" event={"ID":"4206b2fb-d03d-4827-9658-6dc11f47b6dc","Type":"ContainerStarted","Data":"702022b801a92c8bd5dd1f1f7eeae8f85b0c4426029fd922b8131e1ade69430e"}
Sep 11 17:20:37.711228 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:37.709696    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:20:37.711228 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:37.709865    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn2g7" podUID="1203d22c-52e3-4681-9707-66ca35b9e72a"
Sep 11 17:20:37.711228 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:37.710315    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:20:37.711228 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:37.710414    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82kzs" podUID="97afce2d-e357-4c76-b513-2cb05322bbde"
Sep 11 17:20:37.898083 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:37.898043    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-89.ec2.internal" event={"ID":"29212e0b773c1bb0ac3a9172435b9a0f","Type":"ContainerStarted","Data":"43cfbe52f8c911da6de9eb8485808a540162383b7aa64033401d48b1c655c4a8"}
Sep 11 17:20:37.917866 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:37.917803    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-89.ec2.internal" podStartSLOduration=3.917776463 podStartE2EDuration="3.917776463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:20:37.917397799 +0000 UTC m=+5.721570224" watchObservedRunningTime="2026-09-11 17:20:37.917776463 +0000 UTC m=+5.721948885"
Sep 11 17:20:37.918082 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:37.918037    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-89.ec2.internal" podStartSLOduration=3.918027609 podStartE2EDuration="3.918027609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:20:36.852916201 +0000 UTC m=+4.657088627" watchObservedRunningTime="2026-09-11 17:20:37.918027609 +0000 UTC m=+5.722200034"
Sep 11 17:20:38.184291 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:38.184198    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1203d22c-52e3-4681-9707-66ca35b9e72a-metrics-certs\") pod \"network-metrics-daemon-sn2g7\" (UID: \"1203d22c-52e3-4681-9707-66ca35b9e72a\") " pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:20:38.184462 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:38.184437    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:20:38.184533 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:38.184506    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1203d22c-52e3-4681-9707-66ca35b9e72a-metrics-certs podName:1203d22c-52e3-4681-9707-66ca35b9e72a nodeName:}" failed. No retries permitted until 2026-09-11 17:20:42.184485124 +0000 UTC m=+9.988657542 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1203d22c-52e3-4681-9707-66ca35b9e72a-metrics-certs") pod "network-metrics-daemon-sn2g7" (UID: "1203d22c-52e3-4681-9707-66ca35b9e72a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:20:38.285612 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:38.285570    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97afce2d-e357-4c76-b513-2cb05322bbde-original-pull-secret\") pod \"global-pull-secret-syncer-82kzs\" (UID: \"97afce2d-e357-4c76-b513-2cb05322bbde\") " pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:20:38.285813 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:38.285777    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:20:38.285875 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:38.285851    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97afce2d-e357-4c76-b513-2cb05322bbde-original-pull-secret podName:97afce2d-e357-4c76-b513-2cb05322bbde nodeName:}" failed. No retries permitted until 2026-09-11 17:20:42.285830848 +0000 UTC m=+10.090003265 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97afce2d-e357-4c76-b513-2cb05322bbde-original-pull-secret") pod "global-pull-secret-syncer-82kzs" (UID: "97afce2d-e357-4c76-b513-2cb05322bbde") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:20:38.387075 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:38.386367    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk9mr\" (UniqueName: \"kubernetes.io/projected/a8a65b90-c7a6-4272-a1e8-d60096120332-kube-api-access-tk9mr\") pod \"network-check-target-xjqcw\" (UID: \"a8a65b90-c7a6-4272-a1e8-d60096120332\") " pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:20:38.387075 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:38.386566    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:20:38.387075 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:38.386594    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:20:38.387075 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:38.386608    2560 projected.go:196] Error preparing data for projected volume kube-api-access-tk9mr for pod openshift-network-diagnostics/network-check-target-xjqcw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:20:38.387075 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:38.386669    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8a65b90-c7a6-4272-a1e8-d60096120332-kube-api-access-tk9mr podName:a8a65b90-c7a6-4272-a1e8-d60096120332 nodeName:}" failed. No retries permitted until 2026-09-11 17:20:42.386649183 +0000 UTC m=+10.190821584 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tk9mr" (UniqueName: "kubernetes.io/projected/a8a65b90-c7a6-4272-a1e8-d60096120332-kube-api-access-tk9mr") pod "network-check-target-xjqcw" (UID: "a8a65b90-c7a6-4272-a1e8-d60096120332") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:20:38.710610 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:38.710089    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:20:38.710610 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:38.710235    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjqcw" podUID="a8a65b90-c7a6-4272-a1e8-d60096120332"
Sep 11 17:20:39.709492 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:39.709457    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:20:39.709968 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:39.709603    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82kzs" podUID="97afce2d-e357-4c76-b513-2cb05322bbde"
Sep 11 17:20:39.710069 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:39.710029    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:20:39.710183 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:39.710160    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn2g7" podUID="1203d22c-52e3-4681-9707-66ca35b9e72a"
Sep 11 17:20:40.710358 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:40.710314    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:20:40.710876 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:40.710454    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjqcw" podUID="a8a65b90-c7a6-4272-a1e8-d60096120332"
Sep 11 17:20:41.709849 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:41.709583    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:20:41.710046 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:41.709612    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:20:41.710046 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:41.709994    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn2g7" podUID="1203d22c-52e3-4681-9707-66ca35b9e72a"
Sep 11 17:20:41.710167 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:41.710040    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82kzs" podUID="97afce2d-e357-4c76-b513-2cb05322bbde"
Sep 11 17:20:42.217679 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:42.217643    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1203d22c-52e3-4681-9707-66ca35b9e72a-metrics-certs\") pod \"network-metrics-daemon-sn2g7\" (UID: \"1203d22c-52e3-4681-9707-66ca35b9e72a\") " pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:20:42.218160 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:42.217808    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:20:42.218160 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:42.217868    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1203d22c-52e3-4681-9707-66ca35b9e72a-metrics-certs podName:1203d22c-52e3-4681-9707-66ca35b9e72a nodeName:}" failed. No retries permitted until 2026-09-11 17:20:50.217851086 +0000 UTC m=+18.022023488 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1203d22c-52e3-4681-9707-66ca35b9e72a-metrics-certs") pod "network-metrics-daemon-sn2g7" (UID: "1203d22c-52e3-4681-9707-66ca35b9e72a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:20:42.318615 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:42.318549    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97afce2d-e357-4c76-b513-2cb05322bbde-original-pull-secret\") pod \"global-pull-secret-syncer-82kzs\" (UID: \"97afce2d-e357-4c76-b513-2cb05322bbde\") " pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:20:42.318778 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:42.318750    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:20:42.318845 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:42.318806    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97afce2d-e357-4c76-b513-2cb05322bbde-original-pull-secret podName:97afce2d-e357-4c76-b513-2cb05322bbde nodeName:}" failed. No retries permitted until 2026-09-11 17:20:50.318788485 +0000 UTC m=+18.122960889 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97afce2d-e357-4c76-b513-2cb05322bbde-original-pull-secret") pod "global-pull-secret-syncer-82kzs" (UID: "97afce2d-e357-4c76-b513-2cb05322bbde") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:20:42.419203 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:42.419163    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk9mr\" (UniqueName: \"kubernetes.io/projected/a8a65b90-c7a6-4272-a1e8-d60096120332-kube-api-access-tk9mr\") pod \"network-check-target-xjqcw\" (UID: \"a8a65b90-c7a6-4272-a1e8-d60096120332\") " pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:20:42.419380 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:42.419324    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:20:42.419380 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:42.419341    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:20:42.419380 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:42.419353    2560 projected.go:196] Error preparing data for projected volume kube-api-access-tk9mr for pod openshift-network-diagnostics/network-check-target-xjqcw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:20:42.419497 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:42.419408    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8a65b90-c7a6-4272-a1e8-d60096120332-kube-api-access-tk9mr podName:a8a65b90-c7a6-4272-a1e8-d60096120332 nodeName:}" failed. No retries permitted until 2026-09-11 17:20:50.419390525 +0000 UTC m=+18.223562940 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tk9mr" (UniqueName: "kubernetes.io/projected/a8a65b90-c7a6-4272-a1e8-d60096120332-kube-api-access-tk9mr") pod "network-check-target-xjqcw" (UID: "a8a65b90-c7a6-4272-a1e8-d60096120332") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:20:42.709655 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:42.709601    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:20:42.710659 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:42.710622    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjqcw" podUID="a8a65b90-c7a6-4272-a1e8-d60096120332"
Sep 11 17:20:43.709195 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:43.709161    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:20:43.709643 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:43.709206    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:20:43.709643 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:43.709307    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn2g7" podUID="1203d22c-52e3-4681-9707-66ca35b9e72a"
Sep 11 17:20:43.709816 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:43.709767    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82kzs" podUID="97afce2d-e357-4c76-b513-2cb05322bbde"
Sep 11 17:20:44.709795 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:44.709761    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:20:44.710199 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:44.709917    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjqcw" podUID="a8a65b90-c7a6-4272-a1e8-d60096120332"
Sep 11 17:20:45.709107 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:45.709079    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:20:45.709245 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:45.709180    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn2g7" podUID="1203d22c-52e3-4681-9707-66ca35b9e72a"
Sep 11 17:20:45.709296 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:45.709236    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:20:45.709355 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:45.709330    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82kzs" podUID="97afce2d-e357-4c76-b513-2cb05322bbde"
Sep 11 17:20:46.709276 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:46.709244    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:20:46.709766 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:46.709338    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjqcw" podUID="a8a65b90-c7a6-4272-a1e8-d60096120332"
Sep 11 17:20:47.709815 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:47.709784    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:20:47.710230 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:47.709908    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn2g7" podUID="1203d22c-52e3-4681-9707-66ca35b9e72a"
Sep 11 17:20:47.710230 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:47.709934    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:20:47.710230 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:47.710050    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82kzs" podUID="97afce2d-e357-4c76-b513-2cb05322bbde"
Sep 11 17:20:48.709103 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:48.709067    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:20:48.709333 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:48.709208    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjqcw" podUID="a8a65b90-c7a6-4272-a1e8-d60096120332"
Sep 11 17:20:49.710001 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:49.709956    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:20:49.710505 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:49.709961    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:20:49.710505 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:49.710081    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82kzs" podUID="97afce2d-e357-4c76-b513-2cb05322bbde"
Sep 11 17:20:49.710505 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:49.710179    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn2g7" podUID="1203d22c-52e3-4681-9707-66ca35b9e72a"
Sep 11 17:20:50.283220 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:50.283176    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1203d22c-52e3-4681-9707-66ca35b9e72a-metrics-certs\") pod \"network-metrics-daemon-sn2g7\" (UID: \"1203d22c-52e3-4681-9707-66ca35b9e72a\") " pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:20:50.283468 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:50.283291    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:20:50.283468 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:50.283350    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1203d22c-52e3-4681-9707-66ca35b9e72a-metrics-certs podName:1203d22c-52e3-4681-9707-66ca35b9e72a nodeName:}" failed. No retries permitted until 2026-09-11 17:21:06.283331669 +0000 UTC m=+34.087504070 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1203d22c-52e3-4681-9707-66ca35b9e72a-metrics-certs") pod "network-metrics-daemon-sn2g7" (UID: "1203d22c-52e3-4681-9707-66ca35b9e72a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:20:50.383618 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:50.383571    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97afce2d-e357-4c76-b513-2cb05322bbde-original-pull-secret\") pod \"global-pull-secret-syncer-82kzs\" (UID: \"97afce2d-e357-4c76-b513-2cb05322bbde\") " pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:20:50.383814 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:50.383753    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:20:50.383880 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:50.383838    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97afce2d-e357-4c76-b513-2cb05322bbde-original-pull-secret podName:97afce2d-e357-4c76-b513-2cb05322bbde nodeName:}" failed. No retries permitted until 2026-09-11 17:21:06.383815778 +0000 UTC m=+34.187988181 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97afce2d-e357-4c76-b513-2cb05322bbde-original-pull-secret") pod "global-pull-secret-syncer-82kzs" (UID: "97afce2d-e357-4c76-b513-2cb05322bbde") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:20:50.484167 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:50.484134    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk9mr\" (UniqueName: \"kubernetes.io/projected/a8a65b90-c7a6-4272-a1e8-d60096120332-kube-api-access-tk9mr\") pod \"network-check-target-xjqcw\" (UID: \"a8a65b90-c7a6-4272-a1e8-d60096120332\") " pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:20:50.484334 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:50.484279    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:20:50.484334 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:50.484296    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:20:50.484334 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:50.484309    2560 projected.go:196] Error preparing data for projected volume kube-api-access-tk9mr for pod openshift-network-diagnostics/network-check-target-xjqcw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:20:50.484466 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:50.484360    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8a65b90-c7a6-4272-a1e8-d60096120332-kube-api-access-tk9mr podName:a8a65b90-c7a6-4272-a1e8-d60096120332 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:06.484347862 +0000 UTC m=+34.288520259 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tk9mr" (UniqueName: "kubernetes.io/projected/a8a65b90-c7a6-4272-a1e8-d60096120332-kube-api-access-tk9mr") pod "network-check-target-xjqcw" (UID: "a8a65b90-c7a6-4272-a1e8-d60096120332") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:20:50.709582 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:50.709551    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:20:50.709776 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:50.709658    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjqcw" podUID="a8a65b90-c7a6-4272-a1e8-d60096120332"
Sep 11 17:20:51.709793 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:51.709754    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:20:51.710283 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:51.709905    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn2g7" podUID="1203d22c-52e3-4681-9707-66ca35b9e72a"
Sep 11 17:20:51.710283 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:51.709961    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:20:51.710283 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:51.710082    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82kzs" podUID="97afce2d-e357-4c76-b513-2cb05322bbde"
Sep 11 17:20:52.710230 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:52.710196    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:20:52.710678 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:52.710301    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjqcw" podUID="a8a65b90-c7a6-4272-a1e8-d60096120332"
Sep 11 17:20:53.709496 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:53.709464    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:20:53.709682 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:53.709585    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82kzs" podUID="97afce2d-e357-4c76-b513-2cb05322bbde"
Sep 11 17:20:53.709682 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:53.709616    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:20:53.709798 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:53.709680    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn2g7" podUID="1203d22c-52e3-4681-9707-66ca35b9e72a"
Sep 11 17:20:54.709736 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:54.709501    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:20:54.710488 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:54.709827    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjqcw" podUID="a8a65b90-c7a6-4272-a1e8-d60096120332"
Sep 11 17:20:54.935503 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:54.935474    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf" event={"ID":"4206b2fb-d03d-4827-9658-6dc11f47b6dc","Type":"ContainerStarted","Data":"53e0782f9741fa4af228a2905c6919896d5cb6acd65316caf250f6135df2a546"}
Sep 11 17:20:54.936942 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:54.936914    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pm66d" event={"ID":"efb29080-6e59-4db3-8a34-bf363f39eb9c","Type":"ContainerStarted","Data":"45ba5fb6e50e0b97c52e45275b0367ce0499b7c59b8ff4df384b927386889cbb"}
Sep 11 17:20:54.938317 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:54.938295    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q9dr2" event={"ID":"c11a0ba2-8654-4e79-a883-80d867a24d80","Type":"ContainerStarted","Data":"10e20b49ee68cd9b3976cad8de34529e9814b09182f38a9210dcbdc4cf706026"}
Sep 11 17:20:54.939954 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:54.939930    2560 generic.go:356] "Generic (PLEG): container finished" podID="efa4af6c-75c3-4ab4-ba6b-92d5a86b9025" containerID="ce4604b11c3a465da74d5d8bcdc8dbf21b853519e531a0158195afdee5c31a81" exitCode=0
Sep 11 17:20:54.940044 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:54.940019    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6665r" event={"ID":"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025","Type":"ContainerDied","Data":"ce4604b11c3a465da74d5d8bcdc8dbf21b853519e531a0158195afdee5c31a81"}
Sep 11 17:20:54.941778 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:54.941730    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bcdn9" event={"ID":"2db2c929-0e48-4ceb-83c3-2cf8a2c087e0","Type":"ContainerStarted","Data":"c0ce9dcd56f5cbf63b9c3997f2357a39b3df3589bd740e5acd1ab0d60fdc6957"}
Sep 11 17:20:54.944172 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:54.943949    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4mxdh" event={"ID":"c49e0b27-b621-4879-b168-3962a9a32c8f","Type":"ContainerStarted","Data":"893b82ed8a3ef414e8d953ca3b0cfaf5ddf7915bd4118e2595cb8dde7df172d4"}
Sep 11 17:20:54.946299 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:54.946273    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c9pzn" event={"ID":"ae0ddecb-8d11-4989-8531-3028a8397e79","Type":"ContainerStarted","Data":"d1748109f1d5d9236e3673e3e12e40193324a2bbb29a844cbb1e18f81caa5f6c"}
Sep 11 17:20:54.950275 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:54.950257    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt" event={"ID":"1a9b26d3-c512-4ea8-a59a-a1a718b2026f","Type":"ContainerStarted","Data":"62ce3bb57308f6e9bcd1a1f7a7bb762f8c5625dc0092dc824463d9b2c5533a07"}
Sep 11 17:20:54.950348 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:54.950308    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt" event={"ID":"1a9b26d3-c512-4ea8-a59a-a1a718b2026f","Type":"ContainerStarted","Data":"88900a14b2472385976f9961fc7a20fe6f9d869f3127f28ba4b43d63448fc545"}
Sep 11 17:20:54.950348 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:54.950337    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt" event={"ID":"1a9b26d3-c512-4ea8-a59a-a1a718b2026f","Type":"ContainerStarted","Data":"e40ff69a304f2dc8638e960da9b394f7936210f8f5f76b0870a8245bc9dc210e"}
Sep 11 17:20:54.950416 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:54.950363    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt" event={"ID":"1a9b26d3-c512-4ea8-a59a-a1a718b2026f","Type":"ContainerStarted","Data":"f4289ce49293acb9c36c9058550e25c7b96aa5015a934f864824889c6e2b610a"}
Sep 11 17:20:54.951654 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:54.951618    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pm66d" podStartSLOduration=7.08329741 podStartE2EDuration="21.951607148s" totalImagesPullingTime="14.868309738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:20:54.951227709 +0000 UTC m=+22.755400128" watchObservedRunningTime="2026-09-11 17:20:54.951607148 +0000 UTC m=+22.755779566"
Sep 11 17:20:54.966233 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:54.966086    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4mxdh" podStartSLOduration=3.559235496 podStartE2EDuration="21.966068593s" totalImagesPullingTime="18.406833097s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:20:54.965909766 +0000 UTC m=+22.770082183" watchObservedRunningTime="2026-09-11 17:20:54.966068593 +0000 UTC m=+22.770241014"
Sep 11 17:20:54.980985 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:54.980886    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-c9pzn" podStartSLOduration=3.325793703 podStartE2EDuration="21.980872092s" totalImagesPullingTime="18.655078389s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:20:54.980851875 +0000 UTC m=+22.785024294" watchObservedRunningTime="2026-09-11 17:20:54.980872092 +0000 UTC m=+22.785044492"
Sep 11 17:20:54.995994 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:54.995946    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bcdn9" podStartSLOduration=3.401781055 podStartE2EDuration="21.995929654s" totalImagesPullingTime="18.594148599s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:20:54.995924742 +0000 UTC m=+22.800097163" watchObservedRunningTime="2026-09-11 17:20:54.995929654 +0000 UTC m=+22.800102050"
Sep 11 17:20:55.291422 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:55.291053    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pm66d"
Sep 11 17:20:55.291859 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:55.291837    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pm66d"
Sep 11 17:20:55.308383 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:55.308340    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q9dr2" podStartSLOduration=3.90701405 podStartE2EDuration="22.308324047s" totalImagesPullingTime="18.401309997s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:20:55.031409225 +0000 UTC m=+22.835581643" watchObservedRunningTime="2026-09-11 17:20:55.308324047 +0000 UTC m=+23.112496465"
Sep 11 17:20:55.709846 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:55.709822    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:20:55.710245 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:55.709943    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82kzs" podUID="97afce2d-e357-4c76-b513-2cb05322bbde"
Sep 11 17:20:55.710245 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:55.709993    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:20:55.710245 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:55.710118    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn2g7" podUID="1203d22c-52e3-4681-9707-66ca35b9e72a"
Sep 11 17:20:55.869316 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:55.869281    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 17:20:55.956873 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:55.956836    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt" event={"ID":"1a9b26d3-c512-4ea8-a59a-a1a718b2026f","Type":"ContainerStarted","Data":"bc147f6a161016903172528a7e482f19b7e0b1cdeedf197475c70cc5ccacb874"}
Sep 11 17:20:55.957073 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:55.956937    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt" event={"ID":"1a9b26d3-c512-4ea8-a59a-a1a718b2026f","Type":"ContainerStarted","Data":"dd94d485153385b37330b7fe1bcd808bdd436c11ac0d3d71b54f8a270e103bea"}
Sep 11 17:20:55.959086 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:55.959056    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf" event={"ID":"4206b2fb-d03d-4827-9658-6dc11f47b6dc","Type":"ContainerStarted","Data":"0509e8bdfdf577b13f15cb49f2f422efa9bbc6acf62849a8bfc7148d8ba4b796"}
Sep 11 17:20:55.960784 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:55.960747    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cjjws" event={"ID":"dc8ad2ff-cb6b-46a5-8f17-caf033796331","Type":"ContainerStarted","Data":"a7d64787a4096480bc070c8ffb8f849752c4d9ecf08e98dc7289140de578227b"}
Sep 11 17:20:55.961818 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:55.961785    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pm66d"
Sep 11 17:20:55.962272 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:55.962245    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pm66d"
Sep 11 17:20:55.975272 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:55.975211    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cjjws" podStartSLOduration=4.575508242 podStartE2EDuration="22.97519021s" totalImagesPullingTime="18.399681968s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:20:55.974938521 +0000 UTC m=+23.779110940" watchObservedRunningTime="2026-09-11 17:20:55.97519021 +0000 UTC m=+23.779362641"
Sep 11 17:20:56.594327 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:56.594210    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T17:20:55.869307469Z","UUID":"4a5c2cb3-5f42-4a97-badf-8b8e0237d197","Handler":null,"Name":"","Endpoint":""}
Sep 11 17:20:56.596225 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:56.596202    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 17:20:56.596350 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:56.596238    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 17:20:56.710146 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:56.710110    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:20:56.710503 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:56.710251    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjqcw" podUID="a8a65b90-c7a6-4272-a1e8-d60096120332"
Sep 11 17:20:57.709530 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:57.709496    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:20:57.709713 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:57.709548    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:20:57.709713 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:57.709647    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn2g7" podUID="1203d22c-52e3-4681-9707-66ca35b9e72a"
Sep 11 17:20:57.709839 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:57.709784    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82kzs" podUID="97afce2d-e357-4c76-b513-2cb05322bbde"
Sep 11 17:20:57.969627 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:57.969541    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt" event={"ID":"1a9b26d3-c512-4ea8-a59a-a1a718b2026f","Type":"ContainerStarted","Data":"fe55cc4023a752f566dc8fbdbc9861601c267d2ebf9d61b8bb7a16d2a44d870a"}
Sep 11 17:20:57.971799 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:57.971771    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf" event={"ID":"4206b2fb-d03d-4827-9658-6dc11f47b6dc","Type":"ContainerStarted","Data":"60fdb408dd5011ea77fe8376cbf1d028ee1fc82024a90e48e6e695968ea730ec"}
Sep 11 17:20:57.991333 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:57.991286    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zstf" podStartSLOduration=4.291199012 podStartE2EDuration="24.991269981s" totalImagesPullingTime="20.700070969s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:20:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:20:57.988881905 +0000 UTC m=+25.793054326" watchObservedRunningTime="2026-09-11 17:20:57.991269981 +0000 UTC m=+25.795442399"
Sep 11 17:20:58.709412 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:58.709380    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:20:58.709598 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:58.709513    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjqcw" podUID="a8a65b90-c7a6-4272-a1e8-d60096120332"
Sep 11 17:20:59.709144 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:59.708948    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:20:59.709793 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:59.708976    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:20:59.709793 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:59.709230    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82kzs" podUID="97afce2d-e357-4c76-b513-2cb05322bbde"
Sep 11 17:20:59.709793 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:20:59.709311    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn2g7" podUID="1203d22c-52e3-4681-9707-66ca35b9e72a"
Sep 11 17:20:59.978527 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:59.978496    2560 generic.go:356] "Generic (PLEG): container finished" podID="efa4af6c-75c3-4ab4-ba6b-92d5a86b9025" containerID="70dd35469cd6fbb2922c5652ebc33d2b70787e35e226db9a37a1ca87992ed7a3" exitCode=0
Sep 11 17:20:59.978712 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:59.978565    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6665r" event={"ID":"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025","Type":"ContainerDied","Data":"70dd35469cd6fbb2922c5652ebc33d2b70787e35e226db9a37a1ca87992ed7a3"}
Sep 11 17:20:59.982559 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:59.982533    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt" event={"ID":"1a9b26d3-c512-4ea8-a59a-a1a718b2026f","Type":"ContainerStarted","Data":"fa99b88d47e3d2c025bb7fcdfc6815b41ab668a4a88a97abd07f681f27551f98"}
Sep 11 17:20:59.982755 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:59.982739    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:59.982828 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:59.982807    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:20:59.997687 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:20:59.997664    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:21:00.026607 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:00.026561    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt" podStartSLOduration=8.362652512 podStartE2EDuration="27.026545145s" totalImagesPullingTime="18.663892633s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:21:00.025392824 +0000 UTC m=+27.829565226" watchObservedRunningTime="2026-09-11 17:21:00.026545145 +0000 UTC m=+27.830717589"
Sep 11 17:21:00.709578 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:00.709542    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:21:00.710272 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:00.709646    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjqcw" podUID="a8a65b90-c7a6-4272-a1e8-d60096120332"
Sep 11 17:21:00.985823 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:00.985734    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:21:01.004036 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:01.004006    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:21:01.632836 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:01.632632    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-82kzs"]
Sep 11 17:21:01.633032 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:01.632937    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:21:01.633032 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:01.633020    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82kzs" podUID="97afce2d-e357-4c76-b513-2cb05322bbde"
Sep 11 17:21:01.635822 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:01.635799    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sn2g7"]
Sep 11 17:21:01.635962 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:01.635905    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:21:01.636024 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:01.635996    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn2g7" podUID="1203d22c-52e3-4681-9707-66ca35b9e72a"
Sep 11 17:21:01.636640 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:01.636609    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xjqcw"]
Sep 11 17:21:01.636721 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:01.636696    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:21:01.636803 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:01.636779    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjqcw" podUID="a8a65b90-c7a6-4272-a1e8-d60096120332"
Sep 11 17:21:01.989811 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:01.989722    2560 generic.go:356] "Generic (PLEG): container finished" podID="efa4af6c-75c3-4ab4-ba6b-92d5a86b9025" containerID="028fc81922658bead8ffca495074872c6e20db05abb0b64741191a7e2a0390a9" exitCode=0
Sep 11 17:21:01.990247 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:01.989807    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6665r" event={"ID":"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025","Type":"ContainerDied","Data":"028fc81922658bead8ffca495074872c6e20db05abb0b64741191a7e2a0390a9"}
Sep 11 17:21:02.710049 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:02.710015    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:21:02.710218 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:02.710134    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82kzs" podUID="97afce2d-e357-4c76-b513-2cb05322bbde"
Sep 11 17:21:02.994618 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:02.994537    2560 generic.go:356] "Generic (PLEG): container finished" podID="efa4af6c-75c3-4ab4-ba6b-92d5a86b9025" containerID="409beb87a7a331ae6febcc98b3c9635d23edbc2631fc8b627a1ab259ecd586a3" exitCode=0
Sep 11 17:21:02.994618 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:02.994586    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6665r" event={"ID":"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025","Type":"ContainerDied","Data":"409beb87a7a331ae6febcc98b3c9635d23edbc2631fc8b627a1ab259ecd586a3"}
Sep 11 17:21:03.709952 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:03.709870    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:21:03.709952 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:03.709916    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:21:03.710117 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:03.710002    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjqcw" podUID="a8a65b90-c7a6-4272-a1e8-d60096120332"
Sep 11 17:21:03.710117 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:03.710095    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn2g7" podUID="1203d22c-52e3-4681-9707-66ca35b9e72a"
Sep 11 17:21:04.709041 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:04.709005    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:21:04.709448 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:04.709153    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82kzs" podUID="97afce2d-e357-4c76-b513-2cb05322bbde"
Sep 11 17:21:05.709865 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:05.709829    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:21:05.710258 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:05.709944    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjqcw" podUID="a8a65b90-c7a6-4272-a1e8-d60096120332"
Sep 11 17:21:05.710258 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:05.709992    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:21:05.710258 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:05.710114    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sn2g7" podUID="1203d22c-52e3-4681-9707-66ca35b9e72a"
Sep 11 17:21:06.297462 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:06.297416    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1203d22c-52e3-4681-9707-66ca35b9e72a-metrics-certs\") pod \"network-metrics-daemon-sn2g7\" (UID: \"1203d22c-52e3-4681-9707-66ca35b9e72a\") " pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:21:06.297669 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:06.297550    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:21:06.297669 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:06.297612    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1203d22c-52e3-4681-9707-66ca35b9e72a-metrics-certs podName:1203d22c-52e3-4681-9707-66ca35b9e72a nodeName:}" failed. No retries permitted until 2026-09-11 17:21:38.297592108 +0000 UTC m=+66.101764505 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1203d22c-52e3-4681-9707-66ca35b9e72a-metrics-certs") pod "network-metrics-daemon-sn2g7" (UID: "1203d22c-52e3-4681-9707-66ca35b9e72a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:21:06.397824 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:06.397777    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97afce2d-e357-4c76-b513-2cb05322bbde-original-pull-secret\") pod \"global-pull-secret-syncer-82kzs\" (UID: \"97afce2d-e357-4c76-b513-2cb05322bbde\") " pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:21:06.398024 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:06.397948    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:21:06.398081 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:06.398035    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97afce2d-e357-4c76-b513-2cb05322bbde-original-pull-secret podName:97afce2d-e357-4c76-b513-2cb05322bbde nodeName:}" failed. No retries permitted until 2026-09-11 17:21:38.398013848 +0000 UTC m=+66.202186249 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97afce2d-e357-4c76-b513-2cb05322bbde-original-pull-secret") pod "global-pull-secret-syncer-82kzs" (UID: "97afce2d-e357-4c76-b513-2cb05322bbde") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:21:06.498347 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:06.498315    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk9mr\" (UniqueName: \"kubernetes.io/projected/a8a65b90-c7a6-4272-a1e8-d60096120332-kube-api-access-tk9mr\") pod \"network-check-target-xjqcw\" (UID: \"a8a65b90-c7a6-4272-a1e8-d60096120332\") " pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:21:06.498534 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:06.498465    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:21:06.498534 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:06.498490    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:21:06.498534 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:06.498502    2560 projected.go:196] Error preparing data for projected volume kube-api-access-tk9mr for pod openshift-network-diagnostics/network-check-target-xjqcw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:21:06.498672 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:06.498557    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8a65b90-c7a6-4272-a1e8-d60096120332-kube-api-access-tk9mr podName:a8a65b90-c7a6-4272-a1e8-d60096120332 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:38.498541979 +0000 UTC m=+66.302714384 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tk9mr" (UniqueName: "kubernetes.io/projected/a8a65b90-c7a6-4272-a1e8-d60096120332-kube-api-access-tk9mr") pod "network-check-target-xjqcw" (UID: "a8a65b90-c7a6-4272-a1e8-d60096120332") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:21:06.709549 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:06.709515    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:21:06.709847 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:06.709652    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82kzs" podUID="97afce2d-e357-4c76-b513-2cb05322bbde"
Sep 11 17:21:07.525822 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.525791    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-89.ec2.internal" event="NodeReady"
Sep 11 17:21:07.526182 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.525941    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 17:21:07.576440 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.576404    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mp4vh"]
Sep 11 17:21:07.589666 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.589641    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xz968"]
Sep 11 17:21:07.589827 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.589773    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mp4vh"
Sep 11 17:21:07.592430 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.592400    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 17:21:07.592769 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.592750    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 17:21:07.593022 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.593007    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n6xk4\""
Sep 11 17:21:07.605594 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.605573    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mp4vh"]
Sep 11 17:21:07.605680 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.605606    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xz968"]
Sep 11 17:21:07.605725 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.605680    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xz968"
Sep 11 17:21:07.608496 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.608479    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 17:21:07.608568 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.608499    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 17:21:07.608568 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.608518    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 17:21:07.608650 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.608502    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-k28ks\""
Sep 11 17:21:07.706399 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.706379    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1dbb8c42-1654-4f42-8237-cf79927d51db-metrics-tls\") pod \"dns-default-mp4vh\" (UID: \"1dbb8c42-1654-4f42-8237-cf79927d51db\") " pod="openshift-dns/dns-default-mp4vh"
Sep 11 17:21:07.706475 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.706406    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nb66\" (UniqueName: \"kubernetes.io/projected/1dbb8c42-1654-4f42-8237-cf79927d51db-kube-api-access-7nb66\") pod \"dns-default-mp4vh\" (UID: \"1dbb8c42-1654-4f42-8237-cf79927d51db\") " pod="openshift-dns/dns-default-mp4vh"
Sep 11 17:21:07.706475 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.706426    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1dbb8c42-1654-4f42-8237-cf79927d51db-config-volume\") pod \"dns-default-mp4vh\" (UID: \"1dbb8c42-1654-4f42-8237-cf79927d51db\") " pod="openshift-dns/dns-default-mp4vh"
Sep 11 17:21:07.706543 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.706477    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fcf52a64-2196-4fb8-9b91-a8f0c4956cdd-cert\") pod \"ingress-canary-xz968\" (UID: \"fcf52a64-2196-4fb8-9b91-a8f0c4956cdd\") " pod="openshift-ingress-canary/ingress-canary-xz968"
Sep 11 17:21:07.706543 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.706502    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz4fw\" (UniqueName: \"kubernetes.io/projected/fcf52a64-2196-4fb8-9b91-a8f0c4956cdd-kube-api-access-sz4fw\") pod \"ingress-canary-xz968\" (UID: \"fcf52a64-2196-4fb8-9b91-a8f0c4956cdd\") " pod="openshift-ingress-canary/ingress-canary-xz968"
Sep 11 17:21:07.706543 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.706526    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dbb8c42-1654-4f42-8237-cf79927d51db-tmp-dir\") pod \"dns-default-mp4vh\" (UID: \"1dbb8c42-1654-4f42-8237-cf79927d51db\") " pod="openshift-dns/dns-default-mp4vh"
Sep 11 17:21:07.709476 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.709458    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:21:07.709623 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.709608    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:21:07.712112 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.712088    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 17:21:07.712224 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.712125    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 17:21:07.712224 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.712169    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-47wvr\""
Sep 11 17:21:07.712386 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.712374    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 17:21:07.712497 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.712485    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f9zbq\""
Sep 11 17:21:07.807236 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.807196    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1dbb8c42-1654-4f42-8237-cf79927d51db-metrics-tls\") pod \"dns-default-mp4vh\" (UID: \"1dbb8c42-1654-4f42-8237-cf79927d51db\") " pod="openshift-dns/dns-default-mp4vh"
Sep 11 17:21:07.807236 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.807231    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nb66\" (UniqueName: \"kubernetes.io/projected/1dbb8c42-1654-4f42-8237-cf79927d51db-kube-api-access-7nb66\") pod \"dns-default-mp4vh\" (UID: \"1dbb8c42-1654-4f42-8237-cf79927d51db\") " pod="openshift-dns/dns-default-mp4vh"
Sep 11 17:21:07.807475 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.807251    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1dbb8c42-1654-4f42-8237-cf79927d51db-config-volume\") pod \"dns-default-mp4vh\" (UID: \"1dbb8c42-1654-4f42-8237-cf79927d51db\") " pod="openshift-dns/dns-default-mp4vh"
Sep 11 17:21:07.807475 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.807288    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fcf52a64-2196-4fb8-9b91-a8f0c4956cdd-cert\") pod \"ingress-canary-xz968\" (UID: \"fcf52a64-2196-4fb8-9b91-a8f0c4956cdd\") " pod="openshift-ingress-canary/ingress-canary-xz968"
Sep 11 17:21:07.807475 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.807306    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz4fw\" (UniqueName: \"kubernetes.io/projected/fcf52a64-2196-4fb8-9b91-a8f0c4956cdd-kube-api-access-sz4fw\") pod \"ingress-canary-xz968\" (UID: \"fcf52a64-2196-4fb8-9b91-a8f0c4956cdd\") " pod="openshift-ingress-canary/ingress-canary-xz968"
Sep 11 17:21:07.807475 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:07.807353    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:21:07.807475 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:07.807415    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dbb8c42-1654-4f42-8237-cf79927d51db-metrics-tls podName:1dbb8c42-1654-4f42-8237-cf79927d51db nodeName:}" failed. No retries permitted until 2026-09-11 17:21:08.307401483 +0000 UTC m=+36.111573881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1dbb8c42-1654-4f42-8237-cf79927d51db-metrics-tls") pod "dns-default-mp4vh" (UID: "1dbb8c42-1654-4f42-8237-cf79927d51db") : secret "dns-default-metrics-tls" not found
Sep 11 17:21:07.807770 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.807502    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dbb8c42-1654-4f42-8237-cf79927d51db-tmp-dir\") pod \"dns-default-mp4vh\" (UID: \"1dbb8c42-1654-4f42-8237-cf79927d51db\") " pod="openshift-dns/dns-default-mp4vh"
Sep 11 17:21:07.807810 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.807789    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dbb8c42-1654-4f42-8237-cf79927d51db-tmp-dir\") pod \"dns-default-mp4vh\" (UID: \"1dbb8c42-1654-4f42-8237-cf79927d51db\") " pod="openshift-dns/dns-default-mp4vh"
Sep 11 17:21:07.807840 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.807828    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1dbb8c42-1654-4f42-8237-cf79927d51db-config-volume\") pod \"dns-default-mp4vh\" (UID: \"1dbb8c42-1654-4f42-8237-cf79927d51db\") " pod="openshift-dns/dns-default-mp4vh"
Sep 11 17:21:07.811315 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.811285    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fcf52a64-2196-4fb8-9b91-a8f0c4956cdd-cert\") pod \"ingress-canary-xz968\" (UID: \"fcf52a64-2196-4fb8-9b91-a8f0c4956cdd\") " pod="openshift-ingress-canary/ingress-canary-xz968"
Sep 11 17:21:07.818227 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.818195    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nb66\" (UniqueName: \"kubernetes.io/projected/1dbb8c42-1654-4f42-8237-cf79927d51db-kube-api-access-7nb66\") pod \"dns-default-mp4vh\" (UID: \"1dbb8c42-1654-4f42-8237-cf79927d51db\") " pod="openshift-dns/dns-default-mp4vh"
Sep 11 17:21:07.828484 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.828459    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz4fw\" (UniqueName: \"kubernetes.io/projected/fcf52a64-2196-4fb8-9b91-a8f0c4956cdd-kube-api-access-sz4fw\") pod \"ingress-canary-xz968\" (UID: \"fcf52a64-2196-4fb8-9b91-a8f0c4956cdd\") " pod="openshift-ingress-canary/ingress-canary-xz968"
Sep 11 17:21:07.913769 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:07.913744    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xz968"
Sep 11 17:21:08.013877 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:08.013847    2560 generic.go:356] "Generic (PLEG): container finished" podID="efa4af6c-75c3-4ab4-ba6b-92d5a86b9025" containerID="c13fde57af147a7c5aaa3b4d46efc5966749dfbef9a465fec4326b8b820a4b7b" exitCode=0
Sep 11 17:21:08.013877 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:08.013888    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6665r" event={"ID":"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025","Type":"ContainerDied","Data":"c13fde57af147a7c5aaa3b4d46efc5966749dfbef9a465fec4326b8b820a4b7b"}
Sep 11 17:21:08.214515 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:08.214325    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xz968"]
Sep 11 17:21:08.217455 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:21:08.217429    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcf52a64_2196_4fb8_9b91_a8f0c4956cdd.slice/crio-a66ae7769814c9c43421bf0104f80c0846481168779133c8f3ad49fb3fe3ee4f WatchSource:0}: Error finding container a66ae7769814c9c43421bf0104f80c0846481168779133c8f3ad49fb3fe3ee4f: Status 404 returned error can't find the container with id a66ae7769814c9c43421bf0104f80c0846481168779133c8f3ad49fb3fe3ee4f
Sep 11 17:21:08.310521 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:08.310483    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1dbb8c42-1654-4f42-8237-cf79927d51db-metrics-tls\") pod \"dns-default-mp4vh\" (UID: \"1dbb8c42-1654-4f42-8237-cf79927d51db\") " pod="openshift-dns/dns-default-mp4vh"
Sep 11 17:21:08.310655 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:08.310610    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:21:08.310710 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:08.310671    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dbb8c42-1654-4f42-8237-cf79927d51db-metrics-tls podName:1dbb8c42-1654-4f42-8237-cf79927d51db nodeName:}" failed. No retries permitted until 2026-09-11 17:21:09.310653298 +0000 UTC m=+37.114825715 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1dbb8c42-1654-4f42-8237-cf79927d51db-metrics-tls") pod "dns-default-mp4vh" (UID: "1dbb8c42-1654-4f42-8237-cf79927d51db") : secret "dns-default-metrics-tls" not found
Sep 11 17:21:08.709489 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:08.709449    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:21:08.712265 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:08.712244    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 17:21:09.018034 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:09.017604    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xz968" event={"ID":"fcf52a64-2196-4fb8-9b91-a8f0c4956cdd","Type":"ContainerStarted","Data":"a66ae7769814c9c43421bf0104f80c0846481168779133c8f3ad49fb3fe3ee4f"}
Sep 11 17:21:09.021076 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:09.021047    2560 generic.go:356] "Generic (PLEG): container finished" podID="efa4af6c-75c3-4ab4-ba6b-92d5a86b9025" containerID="88f1059fb2a062d6c5ebed0ef6cb7d3b6faed293e4d78dbe4947ad0850769df9" exitCode=0
Sep 11 17:21:09.021205 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:09.021104    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6665r" event={"ID":"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025","Type":"ContainerDied","Data":"88f1059fb2a062d6c5ebed0ef6cb7d3b6faed293e4d78dbe4947ad0850769df9"}
Sep 11 17:21:09.317013 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:09.316883    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1dbb8c42-1654-4f42-8237-cf79927d51db-metrics-tls\") pod \"dns-default-mp4vh\" (UID: \"1dbb8c42-1654-4f42-8237-cf79927d51db\") " pod="openshift-dns/dns-default-mp4vh"
Sep 11 17:21:09.317162 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:09.317067    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:21:09.317162 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:09.317152    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dbb8c42-1654-4f42-8237-cf79927d51db-metrics-tls podName:1dbb8c42-1654-4f42-8237-cf79927d51db nodeName:}" failed. No retries permitted until 2026-09-11 17:21:11.317129108 +0000 UTC m=+39.121301509 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1dbb8c42-1654-4f42-8237-cf79927d51db-metrics-tls") pod "dns-default-mp4vh" (UID: "1dbb8c42-1654-4f42-8237-cf79927d51db") : secret "dns-default-metrics-tls" not found
Sep 11 17:21:10.028077 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:10.028042    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6665r" event={"ID":"efa4af6c-75c3-4ab4-ba6b-92d5a86b9025","Type":"ContainerStarted","Data":"21a420c1ea57d38ce9d82440adfe7efb139bcd9971c959d3c931627ab7260480"}
Sep 11 17:21:10.051241 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:10.051176    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6665r" podStartSLOduration=7.314785089 podStartE2EDuration="37.051158479s" totalImagesPullingTime="29.73637339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:20:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:21:10.050073248 +0000 UTC m=+37.854245669" watchObservedRunningTime="2026-09-11 17:21:10.051158479 +0000 UTC m=+37.855330898"
Sep 11 17:21:11.032681 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:11.032595    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xz968" event={"ID":"fcf52a64-2196-4fb8-9b91-a8f0c4956cdd","Type":"ContainerStarted","Data":"9e143d180c348c3d713b205443c19fa0463288a563877fc70c32b7a81dd33d29"}
Sep 11 17:21:11.048736 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:11.048682    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xz968" podStartSLOduration=1.5225849550000001 podStartE2EDuration="4.048667546s" totalImagesPullingTime="2.526082591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:21:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:21:11.047967849 +0000 UTC m=+38.852140269" watchObservedRunningTime="2026-09-11 17:21:11.048667546 +0000 UTC m=+38.852839959"
Sep 11 17:21:11.332059 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:11.332018    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1dbb8c42-1654-4f42-8237-cf79927d51db-metrics-tls\") pod \"dns-default-mp4vh\" (UID: \"1dbb8c42-1654-4f42-8237-cf79927d51db\") " pod="openshift-dns/dns-default-mp4vh"
Sep 11 17:21:11.332249 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:11.332165    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:21:11.332249 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:11.332232    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dbb8c42-1654-4f42-8237-cf79927d51db-metrics-tls podName:1dbb8c42-1654-4f42-8237-cf79927d51db nodeName:}" failed. No retries permitted until 2026-09-11 17:21:15.332213678 +0000 UTC m=+43.136386093 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1dbb8c42-1654-4f42-8237-cf79927d51db-metrics-tls") pod "dns-default-mp4vh" (UID: "1dbb8c42-1654-4f42-8237-cf79927d51db") : secret "dns-default-metrics-tls" not found
Sep 11 17:21:15.361028 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:15.360983    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1dbb8c42-1654-4f42-8237-cf79927d51db-metrics-tls\") pod \"dns-default-mp4vh\" (UID: \"1dbb8c42-1654-4f42-8237-cf79927d51db\") " pod="openshift-dns/dns-default-mp4vh"
Sep 11 17:21:15.361515 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:15.361172    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:21:15.361515 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:15.361254    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dbb8c42-1654-4f42-8237-cf79927d51db-metrics-tls podName:1dbb8c42-1654-4f42-8237-cf79927d51db nodeName:}" failed. No retries permitted until 2026-09-11 17:21:23.361234924 +0000 UTC m=+51.165407334 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1dbb8c42-1654-4f42-8237-cf79927d51db-metrics-tls") pod "dns-default-mp4vh" (UID: "1dbb8c42-1654-4f42-8237-cf79927d51db") : secret "dns-default-metrics-tls" not found
Sep 11 17:21:23.415911 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:23.415856    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1dbb8c42-1654-4f42-8237-cf79927d51db-metrics-tls\") pod \"dns-default-mp4vh\" (UID: \"1dbb8c42-1654-4f42-8237-cf79927d51db\") " pod="openshift-dns/dns-default-mp4vh"
Sep 11 17:21:23.416351 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:23.416012    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:21:23.416351 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:23.416082    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dbb8c42-1654-4f42-8237-cf79927d51db-metrics-tls podName:1dbb8c42-1654-4f42-8237-cf79927d51db nodeName:}" failed. No retries permitted until 2026-09-11 17:21:39.416064078 +0000 UTC m=+67.220236493 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1dbb8c42-1654-4f42-8237-cf79927d51db-metrics-tls") pod "dns-default-mp4vh" (UID: "1dbb8c42-1654-4f42-8237-cf79927d51db") : secret "dns-default-metrics-tls" not found
Sep 11 17:21:27.677867 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.677830    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-25988"]
Sep 11 17:21:27.679568 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.679552    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-25988"
Sep 11 17:21:27.686183 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.686159    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 11 17:21:27.686296 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.686178    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 11 17:21:27.687142 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.687121    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-8ljkv\""
Sep 11 17:21:27.687240 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.687153    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:21:27.687240 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.687156    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 11 17:21:27.704604 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.704581    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-25988"]
Sep 11 17:21:27.779366 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.779329    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jd6vw"]
Sep 11 17:21:27.783812 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.783785    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-747d6b5cfd-8p2b2"]
Sep 11 17:21:27.783988 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.783938    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jd6vw"
Sep 11 17:21:27.785519 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.785500    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-6fcjs"]
Sep 11 17:21:27.785640 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.785609    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:27.786308 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.786285    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 11 17:21:27.786420 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.786316    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-xv4m5\""
Sep 11 17:21:27.786420 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.786348    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 11 17:21:27.786420 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.786374    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 11 17:21:27.786666 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.786650    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:21:27.787082 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.787066    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-6fcjs"
Sep 11 17:21:27.788477 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.788456    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 17:21:27.788881 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.788865    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 17:21:27.788977 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.788925    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nbpzt\""
Sep 11 17:21:27.789525 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.789508    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 11 17:21:27.789525 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.789516    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-xq6bw\""
Sep 11 17:21:27.789670 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.789561    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:21:27.789913 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.789881    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 11 17:21:27.791133 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.791114    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 11 17:21:27.793066 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.793049    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 17:21:27.797251 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.797227    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-6fcjs"]
Sep 11 17:21:27.800395 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.799156    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 17:21:27.800395 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.799178    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 11 17:21:27.805934 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.805890    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-747d6b5cfd-8p2b2"]
Sep 11 17:21:27.806890 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.806872    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jd6vw"]
Sep 11 17:21:27.850543 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.850505    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1581956-5e0a-4ed1-9258-985cf6ca7373-config\") pod \"service-ca-operator-cf7c776b7-25988\" (UID: \"c1581956-5e0a-4ed1-9258-985cf6ca7373\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-25988"
Sep 11 17:21:27.850543 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.850548    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1581956-5e0a-4ed1-9258-985cf6ca7373-serving-cert\") pod \"service-ca-operator-cf7c776b7-25988\" (UID: \"c1581956-5e0a-4ed1-9258-985cf6ca7373\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-25988"
Sep 11 17:21:27.850761 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.850567    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdv8s\" (UniqueName: \"kubernetes.io/projected/c1581956-5e0a-4ed1-9258-985cf6ca7373-kube-api-access-hdv8s\") pod \"service-ca-operator-cf7c776b7-25988\" (UID: \"c1581956-5e0a-4ed1-9258-985cf6ca7373\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-25988"
Sep 11 17:21:27.951995 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.951888    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/477cac01-ab9e-4441-b07d-b7e53401c616-serving-cert\") pod \"console-operator-696884fb9c-6fcjs\" (UID: \"477cac01-ab9e-4441-b07d-b7e53401c616\") " pod="openshift-console-operator/console-operator-696884fb9c-6fcjs"
Sep 11 17:21:27.951995 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.951953    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9abdd1bd-edc8-41c3-91e6-da724784b425-trusted-ca\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:27.951995 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.951987    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-bound-sa-token\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:27.952212 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.952013    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1581956-5e0a-4ed1-9258-985cf6ca7373-config\") pod \"service-ca-operator-cf7c776b7-25988\" (UID: \"c1581956-5e0a-4ed1-9258-985cf6ca7373\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-25988"
Sep 11 17:21:27.952212 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.952032    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9abdd1bd-edc8-41c3-91e6-da724784b425-ca-trust-extracted\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:27.952212 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.952079    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fksz\" (UniqueName: \"kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-kube-api-access-8fksz\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:27.952212 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.952109    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8rgp\" (UniqueName: \"kubernetes.io/projected/477cac01-ab9e-4441-b07d-b7e53401c616-kube-api-access-j8rgp\") pod \"console-operator-696884fb9c-6fcjs\" (UID: \"477cac01-ab9e-4441-b07d-b7e53401c616\") " pod="openshift-console-operator/console-operator-696884fb9c-6fcjs"
Sep 11 17:21:27.952212 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.952143    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1581956-5e0a-4ed1-9258-985cf6ca7373-serving-cert\") pod \"service-ca-operator-cf7c776b7-25988\" (UID: \"c1581956-5e0a-4ed1-9258-985cf6ca7373\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-25988"
Sep 11 17:21:27.952212 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.952164    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdv8s\" (UniqueName: \"kubernetes.io/projected/c1581956-5e0a-4ed1-9258-985cf6ca7373-kube-api-access-hdv8s\") pod \"service-ca-operator-cf7c776b7-25988\" (UID: \"c1581956-5e0a-4ed1-9258-985cf6ca7373\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-25988"
Sep 11 17:21:27.952212 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.952207    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b996965e-b19e-4959-b58c-a3cb4039d465-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-jd6vw\" (UID: \"b996965e-b19e-4959-b58c-a3cb4039d465\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jd6vw"
Sep 11 17:21:27.952512 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.952222    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kd7b\" (UniqueName: \"kubernetes.io/projected/b996965e-b19e-4959-b58c-a3cb4039d465-kube-api-access-4kd7b\") pod \"kube-storage-version-migrator-operator-6c8f84f897-jd6vw\" (UID: \"b996965e-b19e-4959-b58c-a3cb4039d465\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jd6vw"
Sep 11 17:21:27.952512 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.952237    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-registry-tls\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:27.952512 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.952344    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b996965e-b19e-4959-b58c-a3cb4039d465-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-jd6vw\" (UID: \"b996965e-b19e-4959-b58c-a3cb4039d465\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jd6vw"
Sep 11 17:21:27.952512 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.952368    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9abdd1bd-edc8-41c3-91e6-da724784b425-installation-pull-secrets\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:27.952512 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.952399    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/477cac01-ab9e-4441-b07d-b7e53401c616-config\") pod \"console-operator-696884fb9c-6fcjs\" (UID: \"477cac01-ab9e-4441-b07d-b7e53401c616\") " pod="openshift-console-operator/console-operator-696884fb9c-6fcjs"
Sep 11 17:21:27.952512 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.952415    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/477cac01-ab9e-4441-b07d-b7e53401c616-trusted-ca\") pod \"console-operator-696884fb9c-6fcjs\" (UID: \"477cac01-ab9e-4441-b07d-b7e53401c616\") " pod="openshift-console-operator/console-operator-696884fb9c-6fcjs"
Sep 11 17:21:27.952512 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.952445    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9abdd1bd-edc8-41c3-91e6-da724784b425-image-registry-private-configuration\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:27.952512 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.952469    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9abdd1bd-edc8-41c3-91e6-da724784b425-registry-certificates\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:27.952789 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.952626    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1581956-5e0a-4ed1-9258-985cf6ca7373-config\") pod \"service-ca-operator-cf7c776b7-25988\" (UID: \"c1581956-5e0a-4ed1-9258-985cf6ca7373\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-25988"
Sep 11 17:21:27.955601 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.955580    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1581956-5e0a-4ed1-9258-985cf6ca7373-serving-cert\") pod \"service-ca-operator-cf7c776b7-25988\" (UID: \"c1581956-5e0a-4ed1-9258-985cf6ca7373\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-25988"
Sep 11 17:21:27.963033 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.963006    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdv8s\" (UniqueName: \"kubernetes.io/projected/c1581956-5e0a-4ed1-9258-985cf6ca7373-kube-api-access-hdv8s\") pod \"service-ca-operator-cf7c776b7-25988\" (UID: \"c1581956-5e0a-4ed1-9258-985cf6ca7373\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-25988"
Sep 11 17:21:27.987741 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:27.987708    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-25988"
Sep 11 17:21:28.053423 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.053393    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b996965e-b19e-4959-b58c-a3cb4039d465-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-jd6vw\" (UID: \"b996965e-b19e-4959-b58c-a3cb4039d465\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jd6vw"
Sep 11 17:21:28.053574 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.053428    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kd7b\" (UniqueName: \"kubernetes.io/projected/b996965e-b19e-4959-b58c-a3cb4039d465-kube-api-access-4kd7b\") pod \"kube-storage-version-migrator-operator-6c8f84f897-jd6vw\" (UID: \"b996965e-b19e-4959-b58c-a3cb4039d465\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jd6vw"
Sep 11 17:21:28.053574 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.053446    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-registry-tls\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:28.053574 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.053467    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b996965e-b19e-4959-b58c-a3cb4039d465-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-jd6vw\" (UID: \"b996965e-b19e-4959-b58c-a3cb4039d465\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jd6vw"
Sep 11 17:21:28.053574 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:28.053561    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 17:21:28.053574 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:28.053578    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-747d6b5cfd-8p2b2: secret "image-registry-tls" not found
Sep 11 17:21:28.053822 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:28.053627    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-registry-tls podName:9abdd1bd-edc8-41c3-91e6-da724784b425 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:28.553611418 +0000 UTC m=+56.357783815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-registry-tls") pod "image-registry-747d6b5cfd-8p2b2" (UID: "9abdd1bd-edc8-41c3-91e6-da724784b425") : secret "image-registry-tls" not found
Sep 11 17:21:28.053822 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.053737    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9abdd1bd-edc8-41c3-91e6-da724784b425-installation-pull-secrets\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:28.053822 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.053772    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/477cac01-ab9e-4441-b07d-b7e53401c616-config\") pod \"console-operator-696884fb9c-6fcjs\" (UID: \"477cac01-ab9e-4441-b07d-b7e53401c616\") " pod="openshift-console-operator/console-operator-696884fb9c-6fcjs"
Sep 11 17:21:28.053822 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.053793    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/477cac01-ab9e-4441-b07d-b7e53401c616-trusted-ca\") pod \"console-operator-696884fb9c-6fcjs\" (UID: \"477cac01-ab9e-4441-b07d-b7e53401c616\") " pod="openshift-console-operator/console-operator-696884fb9c-6fcjs"
Sep 11 17:21:28.054148 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.053831    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9abdd1bd-edc8-41c3-91e6-da724784b425-image-registry-private-configuration\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:28.054148 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.053860    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9abdd1bd-edc8-41c3-91e6-da724784b425-registry-certificates\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:28.054148 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.053885    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/477cac01-ab9e-4441-b07d-b7e53401c616-serving-cert\") pod \"console-operator-696884fb9c-6fcjs\" (UID: \"477cac01-ab9e-4441-b07d-b7e53401c616\") " pod="openshift-console-operator/console-operator-696884fb9c-6fcjs"
Sep 11 17:21:28.054148 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.053924    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9abdd1bd-edc8-41c3-91e6-da724784b425-trusted-ca\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:28.054148 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.053957    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-bound-sa-token\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:28.054148 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.053994    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9abdd1bd-edc8-41c3-91e6-da724784b425-ca-trust-extracted\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:28.054148 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.054017    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fksz\" (UniqueName: \"kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-kube-api-access-8fksz\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:28.054148 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.054041    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8rgp\" (UniqueName: \"kubernetes.io/projected/477cac01-ab9e-4441-b07d-b7e53401c616-kube-api-access-j8rgp\") pod \"console-operator-696884fb9c-6fcjs\" (UID: \"477cac01-ab9e-4441-b07d-b7e53401c616\") " pod="openshift-console-operator/console-operator-696884fb9c-6fcjs"
Sep 11 17:21:28.054148 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.054063    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b996965e-b19e-4959-b58c-a3cb4039d465-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-jd6vw\" (UID: \"b996965e-b19e-4959-b58c-a3cb4039d465\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jd6vw"
Sep 11 17:21:28.054582 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.054557    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9abdd1bd-edc8-41c3-91e6-da724784b425-registry-certificates\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:28.054652 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.054629    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/477cac01-ab9e-4441-b07d-b7e53401c616-config\") pod \"console-operator-696884fb9c-6fcjs\" (UID: \"477cac01-ab9e-4441-b07d-b7e53401c616\") " pod="openshift-console-operator/console-operator-696884fb9c-6fcjs"
Sep 11 17:21:28.055011 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.054972    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9abdd1bd-edc8-41c3-91e6-da724784b425-ca-trust-extracted\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:28.055011 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.055003    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9abdd1bd-edc8-41c3-91e6-da724784b425-trusted-ca\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:28.055277 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.055243    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/477cac01-ab9e-4441-b07d-b7e53401c616-trusted-ca\") pod \"console-operator-696884fb9c-6fcjs\" (UID: \"477cac01-ab9e-4441-b07d-b7e53401c616\") " pod="openshift-console-operator/console-operator-696884fb9c-6fcjs"
Sep 11 17:21:28.056557 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.056537    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b996965e-b19e-4959-b58c-a3cb4039d465-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-jd6vw\" (UID: \"b996965e-b19e-4959-b58c-a3cb4039d465\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jd6vw"
Sep 11 17:21:28.056714 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.056696    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9abdd1bd-edc8-41c3-91e6-da724784b425-installation-pull-secrets\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:28.057360 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.057339    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9abdd1bd-edc8-41c3-91e6-da724784b425-image-registry-private-configuration\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:28.057787 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.057771    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/477cac01-ab9e-4441-b07d-b7e53401c616-serving-cert\") pod \"console-operator-696884fb9c-6fcjs\" (UID: \"477cac01-ab9e-4441-b07d-b7e53401c616\") " pod="openshift-console-operator/console-operator-696884fb9c-6fcjs"
Sep 11 17:21:28.066409 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.066372    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kd7b\" (UniqueName: \"kubernetes.io/projected/b996965e-b19e-4959-b58c-a3cb4039d465-kube-api-access-4kd7b\") pod \"kube-storage-version-migrator-operator-6c8f84f897-jd6vw\" (UID: \"b996965e-b19e-4959-b58c-a3cb4039d465\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jd6vw"
Sep 11 17:21:28.066553 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.066445    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fksz\" (UniqueName: \"kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-kube-api-access-8fksz\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:28.066699 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.066681    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-bound-sa-token\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:28.066822 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.066806    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8rgp\" (UniqueName: \"kubernetes.io/projected/477cac01-ab9e-4441-b07d-b7e53401c616-kube-api-access-j8rgp\") pod \"console-operator-696884fb9c-6fcjs\" (UID: \"477cac01-ab9e-4441-b07d-b7e53401c616\") " pod="openshift-console-operator/console-operator-696884fb9c-6fcjs"
Sep 11 17:21:28.094462 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.094422    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jd6vw"
Sep 11 17:21:28.108677 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.108644    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-6fcjs"
Sep 11 17:21:28.254587 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.254507    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-25988"]
Sep 11 17:21:28.257359 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:21:28.257324    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1581956_5e0a_4ed1_9258_985cf6ca7373.slice/crio-041f2d513b6821406c3659555ce150a5a3a8aef42972a95457344474e197b598 WatchSource:0}: Error finding container 041f2d513b6821406c3659555ce150a5a3a8aef42972a95457344474e197b598: Status 404 returned error can't find the container with id 041f2d513b6821406c3659555ce150a5a3a8aef42972a95457344474e197b598
Sep 11 17:21:28.366385 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.366276    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jd6vw"]
Sep 11 17:21:28.368914 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:21:28.368858    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb996965e_b19e_4959_b58c_a3cb4039d465.slice/crio-0f455213ae415af137fdad771366ebf93607f58021efd86d08191a3595990d2b WatchSource:0}: Error finding container 0f455213ae415af137fdad771366ebf93607f58021efd86d08191a3595990d2b: Status 404 returned error can't find the container with id 0f455213ae415af137fdad771366ebf93607f58021efd86d08191a3595990d2b
Sep 11 17:21:28.386271 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.386240    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-6fcjs"]
Sep 11 17:21:28.390212 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:21:28.390185    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod477cac01_ab9e_4441_b07d_b7e53401c616.slice/crio-cb18231279941aa234d7ba702dfa748d7e4277b6aad8d35e96c5987c953bc399 WatchSource:0}: Error finding container cb18231279941aa234d7ba702dfa748d7e4277b6aad8d35e96c5987c953bc399: Status 404 returned error can't find the container with id cb18231279941aa234d7ba702dfa748d7e4277b6aad8d35e96c5987c953bc399
Sep 11 17:21:28.558115 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:28.558017    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-registry-tls\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:28.558296 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:28.558170    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 17:21:28.558296 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:28.558189    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-747d6b5cfd-8p2b2: secret "image-registry-tls" not found
Sep 11 17:21:28.558296 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:28.558255    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-registry-tls podName:9abdd1bd-edc8-41c3-91e6-da724784b425 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:29.558235164 +0000 UTC m=+57.362407580 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-registry-tls") pod "image-registry-747d6b5cfd-8p2b2" (UID: "9abdd1bd-edc8-41c3-91e6-da724784b425") : secret "image-registry-tls" not found
Sep 11 17:21:29.075134 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:29.075059    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jd6vw" event={"ID":"b996965e-b19e-4959-b58c-a3cb4039d465","Type":"ContainerStarted","Data":"0f455213ae415af137fdad771366ebf93607f58021efd86d08191a3595990d2b"}
Sep 11 17:21:29.076516 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:29.076484    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-6fcjs" event={"ID":"477cac01-ab9e-4441-b07d-b7e53401c616","Type":"ContainerStarted","Data":"cb18231279941aa234d7ba702dfa748d7e4277b6aad8d35e96c5987c953bc399"}
Sep 11 17:21:29.077829 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:29.077777    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-25988" event={"ID":"c1581956-5e0a-4ed1-9258-985cf6ca7373","Type":"ContainerStarted","Data":"041f2d513b6821406c3659555ce150a5a3a8aef42972a95457344474e197b598"}
Sep 11 17:21:29.565147 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:29.565043    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-registry-tls\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:29.565316 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:29.565213    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 17:21:29.565316 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:29.565233    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-747d6b5cfd-8p2b2: secret "image-registry-tls" not found
Sep 11 17:21:29.565316 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:29.565296    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-registry-tls podName:9abdd1bd-edc8-41c3-91e6-da724784b425 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:31.565277171 +0000 UTC m=+59.369449583 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-registry-tls") pod "image-registry-747d6b5cfd-8p2b2" (UID: "9abdd1bd-edc8-41c3-91e6-da724784b425") : secret "image-registry-tls" not found
Sep 11 17:21:31.583110 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:31.583062    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-registry-tls\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:31.583616 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:31.583239    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 17:21:31.583616 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:31.583260    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-747d6b5cfd-8p2b2: secret "image-registry-tls" not found
Sep 11 17:21:31.583616 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:31.583318    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-registry-tls podName:9abdd1bd-edc8-41c3-91e6-da724784b425 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:35.583302218 +0000 UTC m=+63.387474616 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-registry-tls") pod "image-registry-747d6b5cfd-8p2b2" (UID: "9abdd1bd-edc8-41c3-91e6-da724784b425") : secret "image-registry-tls" not found
Sep 11 17:21:31.749528 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:31.749491    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8hpnj"]
Sep 11 17:21:31.757915 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:31.757878    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8hpnj"
Sep 11 17:21:31.760411 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:31.760378    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 17:21:31.760561 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:31.760497    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 17:21:31.760561 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:31.760523    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 17:21:31.760712 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:31.760698    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 17:21:31.761583 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:31.761567    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-blz79\""
Sep 11 17:21:31.763226 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:31.763204    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8hpnj"]
Sep 11 17:21:31.885423 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:31.885384    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-crio-socket\") pod \"insights-runtime-extractor-8hpnj\" (UID: \"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb\") " pod="openshift-insights/insights-runtime-extractor-8hpnj"
Sep 11 17:21:31.885602 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:31.885461    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8hpnj\" (UID: \"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb\") " pod="openshift-insights/insights-runtime-extractor-8hpnj"
Sep 11 17:21:31.885602 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:31.885523    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-data-volume\") pod \"insights-runtime-extractor-8hpnj\" (UID: \"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb\") " pod="openshift-insights/insights-runtime-extractor-8hpnj"
Sep 11 17:21:31.885602 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:31.885597    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8hpnj\" (UID: \"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb\") " pod="openshift-insights/insights-runtime-extractor-8hpnj"
Sep 11 17:21:31.885778 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:31.885635    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s675v\" (UniqueName: \"kubernetes.io/projected/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-kube-api-access-s675v\") pod \"insights-runtime-extractor-8hpnj\" (UID: \"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb\") " pod="openshift-insights/insights-runtime-extractor-8hpnj"
Sep 11 17:21:31.986000 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:31.985965    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s675v\" (UniqueName: \"kubernetes.io/projected/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-kube-api-access-s675v\") pod \"insights-runtime-extractor-8hpnj\" (UID: \"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb\") " pod="openshift-insights/insights-runtime-extractor-8hpnj"
Sep 11 17:21:31.986180 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:31.986146    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-crio-socket\") pod \"insights-runtime-extractor-8hpnj\" (UID: \"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb\") " pod="openshift-insights/insights-runtime-extractor-8hpnj"
Sep 11 17:21:31.986237 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:31.986198    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8hpnj\" (UID: \"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb\") " pod="openshift-insights/insights-runtime-extractor-8hpnj"
Sep 11 17:21:31.986237 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:31.986226    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-data-volume\") pod \"insights-runtime-extractor-8hpnj\" (UID: \"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb\") " pod="openshift-insights/insights-runtime-extractor-8hpnj"
Sep 11 17:21:31.986334 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:31.986258    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-crio-socket\") pod \"insights-runtime-extractor-8hpnj\" (UID: \"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb\") " pod="openshift-insights/insights-runtime-extractor-8hpnj"
Sep 11 17:21:31.986334 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:31.986288    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8hpnj\" (UID: \"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb\") " pod="openshift-insights/insights-runtime-extractor-8hpnj"
Sep 11 17:21:31.986334 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:31.986322    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:21:31.986478 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:31.986373    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 17:21:31.986478 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:31.986390    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-insights-runtime-extractor-tls podName:cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb nodeName:}" failed. No retries permitted until 2026-09-11 17:21:32.486369921 +0000 UTC m=+60.290542324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8hpnj" (UID: "cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:21:31.986478 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:31.986429    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-kube-rbac-proxy-cm podName:cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb nodeName:}" failed. No retries permitted until 2026-09-11 17:21:32.48641433 +0000 UTC m=+60.290586727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8hpnj" (UID: "cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb") : configmap "kube-rbac-proxy" not found
Sep 11 17:21:31.986638 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:31.986601    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-data-volume\") pod \"insights-runtime-extractor-8hpnj\" (UID: \"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb\") " pod="openshift-insights/insights-runtime-extractor-8hpnj"
Sep 11 17:21:31.996274 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:31.996247    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s675v\" (UniqueName: \"kubernetes.io/projected/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-kube-api-access-s675v\") pod \"insights-runtime-extractor-8hpnj\" (UID: \"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb\") " pod="openshift-insights/insights-runtime-extractor-8hpnj"
Sep 11 17:21:32.086677 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:32.086637    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jd6vw" event={"ID":"b996965e-b19e-4959-b58c-a3cb4039d465","Type":"ContainerStarted","Data":"3e0623ae5d050c76aecc3b838973ea57ecd7b527a7a7f080dc6e4983ab28343c"}
Sep 11 17:21:32.088493 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:32.088473    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6fcjs_477cac01-ab9e-4441-b07d-b7e53401c616/console-operator/0.log"
Sep 11 17:21:32.088592 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:32.088509    2560 generic.go:356] "Generic (PLEG): container finished" podID="477cac01-ab9e-4441-b07d-b7e53401c616" containerID="2458a1dfddcbd331415baecfa065f7ef40e8786b7f3504befaa9eeb62f9b8494" exitCode=255
Sep 11 17:21:32.088592 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:32.088577    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-6fcjs" event={"ID":"477cac01-ab9e-4441-b07d-b7e53401c616","Type":"ContainerDied","Data":"2458a1dfddcbd331415baecfa065f7ef40e8786b7f3504befaa9eeb62f9b8494"}
Sep 11 17:21:32.088778 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:32.088762    2560 scope.go:117] "RemoveContainer" containerID="2458a1dfddcbd331415baecfa065f7ef40e8786b7f3504befaa9eeb62f9b8494"
Sep 11 17:21:32.090361 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:32.090294    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-25988" event={"ID":"c1581956-5e0a-4ed1-9258-985cf6ca7373","Type":"ContainerStarted","Data":"d937c50aa8e47a3aba1d901ebd9115cc3bbf379a91c56bb8a2f77536590ebbfc"}
Sep 11 17:21:32.104110 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:32.103996    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jd6vw" podStartSLOduration=1.6803651830000002 podStartE2EDuration="5.103976188s" totalImagesPullingTime="3.423611005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:21:32.103114511 +0000 UTC m=+59.907286930" watchObservedRunningTime="2026-09-11 17:21:32.103976188 +0000 UTC m=+59.908148607"
Sep 11 17:21:32.128570 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:32.128083    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-25988" podStartSLOduration=1.597798536 podStartE2EDuration="5.128064118s" totalImagesPullingTime="3.530265582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:21:32.126675607 +0000 UTC m=+59.930848029" watchObservedRunningTime="2026-09-11 17:21:32.128064118 +0000 UTC m=+59.932236534"
Sep 11 17:21:32.491472 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:32.491378    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8hpnj\" (UID: \"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb\") " pod="openshift-insights/insights-runtime-extractor-8hpnj"
Sep 11 17:21:32.491472 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:32.491437    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8hpnj\" (UID: \"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb\") " pod="openshift-insights/insights-runtime-extractor-8hpnj"
Sep 11 17:21:32.491648 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:32.491543    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 17:21:32.491648 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:32.491578    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:21:32.491648 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:32.491598    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-kube-rbac-proxy-cm podName:cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb nodeName:}" failed. No retries permitted until 2026-09-11 17:21:33.491584833 +0000 UTC m=+61.295757244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8hpnj" (UID: "cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb") : configmap "kube-rbac-proxy" not found
Sep 11 17:21:32.491648 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:32.491648    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-insights-runtime-extractor-tls podName:cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb nodeName:}" failed. No retries permitted until 2026-09-11 17:21:33.491628327 +0000 UTC m=+61.295800738 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8hpnj" (UID: "cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:21:33.006287 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:33.006258    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8dnzt"
Sep 11 17:21:33.095037 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:33.095007    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6fcjs_477cac01-ab9e-4441-b07d-b7e53401c616/console-operator/1.log"
Sep 11 17:21:33.095420 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:33.095407    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6fcjs_477cac01-ab9e-4441-b07d-b7e53401c616/console-operator/0.log"
Sep 11 17:21:33.095471 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:33.095442    2560 generic.go:356] "Generic (PLEG): container finished" podID="477cac01-ab9e-4441-b07d-b7e53401c616" containerID="db0601470ee4a350ddabad1a2752ac204feb814cd9a97fea78954df59243cf72" exitCode=255
Sep 11 17:21:33.095558 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:33.095537    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-6fcjs" event={"ID":"477cac01-ab9e-4441-b07d-b7e53401c616","Type":"ContainerDied","Data":"db0601470ee4a350ddabad1a2752ac204feb814cd9a97fea78954df59243cf72"}
Sep 11 17:21:33.096471 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:33.096455    2560 scope.go:117] "RemoveContainer" containerID="2458a1dfddcbd331415baecfa065f7ef40e8786b7f3504befaa9eeb62f9b8494"
Sep 11 17:21:33.096572 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:33.096555    2560 scope.go:117] "RemoveContainer" containerID="db0601470ee4a350ddabad1a2752ac204feb814cd9a97fea78954df59243cf72"
Sep 11 17:21:33.096743 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:33.096721    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-6fcjs_openshift-console-operator(477cac01-ab9e-4441-b07d-b7e53401c616)\"" pod="openshift-console-operator/console-operator-696884fb9c-6fcjs" podUID="477cac01-ab9e-4441-b07d-b7e53401c616"
Sep 11 17:21:33.490796 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:33.490717    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4mxdh_c49e0b27-b621-4879-b168-3962a9a32c8f/dns-node-resolver/0.log"
Sep 11 17:21:33.502062 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:33.502034    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8hpnj\" (UID: \"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb\") " pod="openshift-insights/insights-runtime-extractor-8hpnj"
Sep 11 17:21:33.502218 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:33.502121    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8hpnj\" (UID: \"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb\") " pod="openshift-insights/insights-runtime-extractor-8hpnj"
Sep 11 17:21:33.502218 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:33.502168    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 17:21:33.502299 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:33.502231    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:21:33.502299 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:33.502246    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-kube-rbac-proxy-cm podName:cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb nodeName:}" failed. No retries permitted until 2026-09-11 17:21:35.50222883 +0000 UTC m=+63.306401248 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8hpnj" (UID: "cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb") : configmap "kube-rbac-proxy" not found
Sep 11 17:21:33.502299 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:33.502277    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-insights-runtime-extractor-tls podName:cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb nodeName:}" failed. No retries permitted until 2026-09-11 17:21:35.502266995 +0000 UTC m=+63.306439393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8hpnj" (UID: "cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:21:34.099883 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:34.099858    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6fcjs_477cac01-ab9e-4441-b07d-b7e53401c616/console-operator/1.log"
Sep 11 17:21:34.100278 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:34.100205    2560 scope.go:117] "RemoveContainer" containerID="db0601470ee4a350ddabad1a2752ac204feb814cd9a97fea78954df59243cf72"
Sep 11 17:21:34.100374 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:34.100357    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-6fcjs_openshift-console-operator(477cac01-ab9e-4441-b07d-b7e53401c616)\"" pod="openshift-console-operator/console-operator-696884fb9c-6fcjs" podUID="477cac01-ab9e-4441-b07d-b7e53401c616"
Sep 11 17:21:34.688521 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:34.688491    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q9dr2_c11a0ba2-8654-4e79-a883-80d867a24d80/node-ca/0.log"
Sep 11 17:21:35.498449 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:35.498417    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xz968_fcf52a64-2196-4fb8-9b91-a8f0c4956cdd/serve-healthcheck-canary/0.log"
Sep 11 17:21:35.512051 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:35.512021    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-ktjg6"]
Sep 11 17:21:35.514741 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:35.514725    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-ktjg6"
Sep 11 17:21:35.516119 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:35.516089    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9c33cd13-5827-435a-9326-1e5a1eb45ab8-signing-key\") pod \"service-ca-568869fdc5-ktjg6\" (UID: \"9c33cd13-5827-435a-9326-1e5a1eb45ab8\") " pod="openshift-service-ca/service-ca-568869fdc5-ktjg6"
Sep 11 17:21:35.516219 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:35.516122    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8hpnj\" (UID: \"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb\") " pod="openshift-insights/insights-runtime-extractor-8hpnj"
Sep 11 17:21:35.516219 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:35.516148    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdtkc\" (UniqueName: \"kubernetes.io/projected/9c33cd13-5827-435a-9326-1e5a1eb45ab8-kube-api-access-wdtkc\") pod \"service-ca-568869fdc5-ktjg6\" (UID: \"9c33cd13-5827-435a-9326-1e5a1eb45ab8\") " pod="openshift-service-ca/service-ca-568869fdc5-ktjg6"
Sep 11 17:21:35.516308 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:35.516243    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 17:21:35.516341 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:35.516243    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9c33cd13-5827-435a-9326-1e5a1eb45ab8-signing-cabundle\") pod \"service-ca-568869fdc5-ktjg6\" (UID: \"9c33cd13-5827-435a-9326-1e5a1eb45ab8\") " pod="openshift-service-ca/service-ca-568869fdc5-ktjg6"
Sep 11 17:21:35.516341 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:35.516328    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-kube-rbac-proxy-cm podName:cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb nodeName:}" failed. No retries permitted until 2026-09-11 17:21:39.516309577 +0000 UTC m=+67.320481988 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8hpnj" (UID: "cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb") : configmap "kube-rbac-proxy" not found
Sep 11 17:21:35.516410 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:35.516380    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8hpnj\" (UID: \"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb\") " pod="openshift-insights/insights-runtime-extractor-8hpnj"
Sep 11 17:21:35.516501 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:35.516486    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:21:35.516556 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:35.516546    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-insights-runtime-extractor-tls podName:cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb nodeName:}" failed. No retries permitted until 2026-09-11 17:21:39.516532217 +0000 UTC m=+67.320704637 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8hpnj" (UID: "cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:21:35.519042 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:35.519025    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 11 17:21:35.519145 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:35.519026    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 11 17:21:35.519145 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:35.519127    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 11 17:21:35.519508 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:35.519494    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 11 17:21:35.520199 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:35.520183    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-htzwr\""
Sep 11 17:21:35.526790 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:35.526770    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-ktjg6"]
Sep 11 17:21:35.617470 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:35.617428    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdtkc\" (UniqueName: \"kubernetes.io/projected/9c33cd13-5827-435a-9326-1e5a1eb45ab8-kube-api-access-wdtkc\") pod \"service-ca-568869fdc5-ktjg6\" (UID: \"9c33cd13-5827-435a-9326-1e5a1eb45ab8\") " pod="openshift-service-ca/service-ca-568869fdc5-ktjg6"
Sep 11 17:21:35.617644 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:35.617506    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-registry-tls\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:35.617644 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:35.617528    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9c33cd13-5827-435a-9326-1e5a1eb45ab8-signing-cabundle\") pod \"service-ca-568869fdc5-ktjg6\" (UID: \"9c33cd13-5827-435a-9326-1e5a1eb45ab8\") " pod="openshift-service-ca/service-ca-568869fdc5-ktjg6"
Sep 11 17:21:35.617644 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:35.617598    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9c33cd13-5827-435a-9326-1e5a1eb45ab8-signing-key\") pod \"service-ca-568869fdc5-ktjg6\" (UID: \"9c33cd13-5827-435a-9326-1e5a1eb45ab8\") " pod="openshift-service-ca/service-ca-568869fdc5-ktjg6"
Sep 11 17:21:35.617759 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:35.617654    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 17:21:35.617759 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:35.617671    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-747d6b5cfd-8p2b2: secret "image-registry-tls" not found
Sep 11 17:21:35.617759 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:35.617731    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-registry-tls podName:9abdd1bd-edc8-41c3-91e6-da724784b425 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:43.617712402 +0000 UTC m=+71.421884803 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-registry-tls") pod "image-registry-747d6b5cfd-8p2b2" (UID: "9abdd1bd-edc8-41c3-91e6-da724784b425") : secret "image-registry-tls" not found
Sep 11 17:21:35.618357 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:35.618336    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9c33cd13-5827-435a-9326-1e5a1eb45ab8-signing-cabundle\") pod \"service-ca-568869fdc5-ktjg6\" (UID: \"9c33cd13-5827-435a-9326-1e5a1eb45ab8\") " pod="openshift-service-ca/service-ca-568869fdc5-ktjg6"
Sep 11 17:21:35.619996 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:35.619974    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9c33cd13-5827-435a-9326-1e5a1eb45ab8-signing-key\") pod \"service-ca-568869fdc5-ktjg6\" (UID: \"9c33cd13-5827-435a-9326-1e5a1eb45ab8\") " pod="openshift-service-ca/service-ca-568869fdc5-ktjg6"
Sep 11 17:21:35.627208 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:35.627187    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdtkc\" (UniqueName: \"kubernetes.io/projected/9c33cd13-5827-435a-9326-1e5a1eb45ab8-kube-api-access-wdtkc\") pod \"service-ca-568869fdc5-ktjg6\" (UID: \"9c33cd13-5827-435a-9326-1e5a1eb45ab8\") " pod="openshift-service-ca/service-ca-568869fdc5-ktjg6"
Sep 11 17:21:35.823445 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:35.823417    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-ktjg6"
Sep 11 17:21:36.119512 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:36.119430    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-ktjg6"]
Sep 11 17:21:36.122466 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:21:36.122437    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c33cd13_5827_435a_9326_1e5a1eb45ab8.slice/crio-50334e4205f8a49e72360a4820034a7d67984178f89a24756e341c5496a0d8f2 WatchSource:0}: Error finding container 50334e4205f8a49e72360a4820034a7d67984178f89a24756e341c5496a0d8f2: Status 404 returned error can't find the container with id 50334e4205f8a49e72360a4820034a7d67984178f89a24756e341c5496a0d8f2
Sep 11 17:21:37.108542 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:37.108508    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-ktjg6" event={"ID":"9c33cd13-5827-435a-9326-1e5a1eb45ab8","Type":"ContainerStarted","Data":"af67de318b98b90665b53d7b5b213a56312ca8853abf4c6aa39e5513ddc7e5ab"}
Sep 11 17:21:37.108954 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:37.108556    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-ktjg6" event={"ID":"9c33cd13-5827-435a-9326-1e5a1eb45ab8","Type":"ContainerStarted","Data":"50334e4205f8a49e72360a4820034a7d67984178f89a24756e341c5496a0d8f2"}
Sep 11 17:21:37.130489 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:37.130433    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-ktjg6" podStartSLOduration=2.130413534 podStartE2EDuration="2.130413534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:21:37.129577599 +0000 UTC m=+64.933750019" watchObservedRunningTime="2026-09-11 17:21:37.130413534 +0000 UTC m=+64.934585954"
Sep 11 17:21:38.108866 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:38.108818    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-6fcjs"
Sep 11 17:21:38.109499 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:38.109469    2560 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-6fcjs"
Sep 11 17:21:38.109959 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:38.109940    2560 scope.go:117] "RemoveContainer" containerID="db0601470ee4a350ddabad1a2752ac204feb814cd9a97fea78954df59243cf72"
Sep 11 17:21:38.110174 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:38.110130    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-6fcjs_openshift-console-operator(477cac01-ab9e-4441-b07d-b7e53401c616)\"" pod="openshift-console-operator/console-operator-696884fb9c-6fcjs" podUID="477cac01-ab9e-4441-b07d-b7e53401c616"
Sep 11 17:21:38.337760 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:38.337716    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1203d22c-52e3-4681-9707-66ca35b9e72a-metrics-certs\") pod \"network-metrics-daemon-sn2g7\" (UID: \"1203d22c-52e3-4681-9707-66ca35b9e72a\") " pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:21:38.340241 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:38.340218    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 17:21:38.348413 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:38.348394    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 17:21:38.348534 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:38.348480    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1203d22c-52e3-4681-9707-66ca35b9e72a-metrics-certs podName:1203d22c-52e3-4681-9707-66ca35b9e72a nodeName:}" failed. No retries permitted until 2026-09-11 17:22:42.348458031 +0000 UTC m=+130.152630428 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1203d22c-52e3-4681-9707-66ca35b9e72a-metrics-certs") pod "network-metrics-daemon-sn2g7" (UID: "1203d22c-52e3-4681-9707-66ca35b9e72a") : secret "metrics-daemon-secret" not found
Sep 11 17:21:38.438770 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:38.438687    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97afce2d-e357-4c76-b513-2cb05322bbde-original-pull-secret\") pod \"global-pull-secret-syncer-82kzs\" (UID: \"97afce2d-e357-4c76-b513-2cb05322bbde\") " pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:21:38.441662 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:38.441642    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 17:21:38.451208 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:38.451188    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97afce2d-e357-4c76-b513-2cb05322bbde-original-pull-secret\") pod \"global-pull-secret-syncer-82kzs\" (UID: \"97afce2d-e357-4c76-b513-2cb05322bbde\") " pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:21:38.540153 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:38.540110    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk9mr\" (UniqueName: \"kubernetes.io/projected/a8a65b90-c7a6-4272-a1e8-d60096120332-kube-api-access-tk9mr\") pod \"network-check-target-xjqcw\" (UID: \"a8a65b90-c7a6-4272-a1e8-d60096120332\") " pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:21:38.542734 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:38.542714    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 17:21:38.553407 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:38.553382    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 17:21:38.564294 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:38.564258    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk9mr\" (UniqueName: \"kubernetes.io/projected/a8a65b90-c7a6-4272-a1e8-d60096120332-kube-api-access-tk9mr\") pod \"network-check-target-xjqcw\" (UID: \"a8a65b90-c7a6-4272-a1e8-d60096120332\") " pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:21:38.626549 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:38.626515    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-47wvr\""
Sep 11 17:21:38.634832 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:38.634809    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:21:38.718724 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:38.718702    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82kzs"
Sep 11 17:21:38.931445 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:38.931411    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xjqcw"]
Sep 11 17:21:38.934261 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:21:38.934234    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a65b90_c7a6_4272_a1e8_d60096120332.slice/crio-9297ca9b4cb4e2bc32cf014e902ed28761d8eb3e0342a60744443dbf562525dc WatchSource:0}: Error finding container 9297ca9b4cb4e2bc32cf014e902ed28761d8eb3e0342a60744443dbf562525dc: Status 404 returned error can't find the container with id 9297ca9b4cb4e2bc32cf014e902ed28761d8eb3e0342a60744443dbf562525dc
Sep 11 17:21:39.001382 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:39.001314    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-82kzs"]
Sep 11 17:21:39.003667 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:21:39.003644    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97afce2d_e357_4c76_b513_2cb05322bbde.slice/crio-1f20f5bf669803a1d2eec81e442029c381d6d360beffc19faac37261c7fbb90a WatchSource:0}: Error finding container 1f20f5bf669803a1d2eec81e442029c381d6d360beffc19faac37261c7fbb90a: Status 404 returned error can't find the container with id 1f20f5bf669803a1d2eec81e442029c381d6d360beffc19faac37261c7fbb90a
Sep 11 17:21:39.114696 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:39.114651    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xjqcw" event={"ID":"a8a65b90-c7a6-4272-a1e8-d60096120332","Type":"ContainerStarted","Data":"9297ca9b4cb4e2bc32cf014e902ed28761d8eb3e0342a60744443dbf562525dc"}
Sep 11 17:21:39.115755 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:39.115731    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-82kzs" event={"ID":"97afce2d-e357-4c76-b513-2cb05322bbde","Type":"ContainerStarted","Data":"1f20f5bf669803a1d2eec81e442029c381d6d360beffc19faac37261c7fbb90a"}
Sep 11 17:21:39.446917 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:39.446864    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1dbb8c42-1654-4f42-8237-cf79927d51db-metrics-tls\") pod \"dns-default-mp4vh\" (UID: \"1dbb8c42-1654-4f42-8237-cf79927d51db\") " pod="openshift-dns/dns-default-mp4vh"
Sep 11 17:21:39.447096 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:39.447013    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:21:39.447096 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:39.447085    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dbb8c42-1654-4f42-8237-cf79927d51db-metrics-tls podName:1dbb8c42-1654-4f42-8237-cf79927d51db nodeName:}" failed. No retries permitted until 2026-09-11 17:22:11.447065669 +0000 UTC m=+99.251238080 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1dbb8c42-1654-4f42-8237-cf79927d51db-metrics-tls") pod "dns-default-mp4vh" (UID: "1dbb8c42-1654-4f42-8237-cf79927d51db") : secret "dns-default-metrics-tls" not found
Sep 11 17:21:39.547372 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:39.547327    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8hpnj\" (UID: \"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb\") " pod="openshift-insights/insights-runtime-extractor-8hpnj"
Sep 11 17:21:39.547559 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:39.547397    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8hpnj\" (UID: \"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb\") " pod="openshift-insights/insights-runtime-extractor-8hpnj"
Sep 11 17:21:39.547559 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:39.547504    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:21:39.547559 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:39.547538    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 17:21:39.547730 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:39.547583    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-insights-runtime-extractor-tls podName:cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb nodeName:}" failed. No retries permitted until 2026-09-11 17:21:47.547559176 +0000 UTC m=+75.351731577 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8hpnj" (UID: "cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:21:39.547730 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:39.547600    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-kube-rbac-proxy-cm podName:cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb nodeName:}" failed. No retries permitted until 2026-09-11 17:21:47.547593645 +0000 UTC m=+75.351766042 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8hpnj" (UID: "cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb") : configmap "kube-rbac-proxy" not found
Sep 11 17:21:43.682123 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:43.682089    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-registry-tls\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:43.684624 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:43.684597    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-registry-tls\") pod \"image-registry-747d6b5cfd-8p2b2\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") " pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:43.704976 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:43.704942    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nbpzt\""
Sep 11 17:21:43.713330 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:43.713307    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:43.985942 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:43.985911    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-747d6b5cfd-8p2b2"]
Sep 11 17:21:43.988597 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:21:43.988572    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9abdd1bd_edc8_41c3_91e6_da724784b425.slice/crio-882d26b53686c9ca135cbdaafc65fc5b641aca6b281f78a8b9bf246fba4f278b WatchSource:0}: Error finding container 882d26b53686c9ca135cbdaafc65fc5b641aca6b281f78a8b9bf246fba4f278b: Status 404 returned error can't find the container with id 882d26b53686c9ca135cbdaafc65fc5b641aca6b281f78a8b9bf246fba4f278b
Sep 11 17:21:44.133663 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:44.133625    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xjqcw" event={"ID":"a8a65b90-c7a6-4272-a1e8-d60096120332","Type":"ContainerStarted","Data":"f1d72aef30335cdc66cd6245f7206911d9a84e19bdb6859fb9d46dc87e778766"}
Sep 11 17:21:44.133848 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:44.133768    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:21:44.135356 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:44.135326    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-82kzs" event={"ID":"97afce2d-e357-4c76-b513-2cb05322bbde","Type":"ContainerStarted","Data":"ce59bfbabfe3c88e4446d60abfc7a46c3d9af9094e172f6a7bf4c76dcba902b6"}
Sep 11 17:21:44.136934 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:44.136907    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2" event={"ID":"9abdd1bd-edc8-41c3-91e6-da724784b425","Type":"ContainerStarted","Data":"6ed0d5ebedb978d881b1ac6ea99ca6e5523d1d306d0b848eab86f39a0735a1ed"}
Sep 11 17:21:44.137059 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:44.136945    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2" event={"ID":"9abdd1bd-edc8-41c3-91e6-da724784b425","Type":"ContainerStarted","Data":"882d26b53686c9ca135cbdaafc65fc5b641aca6b281f78a8b9bf246fba4f278b"}
Sep 11 17:21:44.137059 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:44.136998    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:21:44.148793 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:44.148594    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xjqcw" podStartSLOduration=66.57221888 podStartE2EDuration="1m11.148579863s" totalImagesPullingTime="4.576360983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:21:44.14807539 +0000 UTC m=+71.952247811" watchObservedRunningTime="2026-09-11 17:21:44.148579863 +0000 UTC m=+71.952752276"
Sep 11 17:21:44.162690 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:44.162640    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-82kzs" podStartSLOduration=66.651832689 podStartE2EDuration="1m11.16262555s" totalImagesPullingTime="4.510792861s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:21:44.162157543 +0000 UTC m=+71.966329964" watchObservedRunningTime="2026-09-11 17:21:44.16262555 +0000 UTC m=+71.966797968"
Sep 11 17:21:44.180195 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:44.180131    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2" podStartSLOduration=17.18011529 podStartE2EDuration="17.18011529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:21:44.17968019 +0000 UTC m=+71.983852610" watchObservedRunningTime="2026-09-11 17:21:44.18011529 +0000 UTC m=+71.984287708"
Sep 11 17:21:47.610548 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:47.610511    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8hpnj\" (UID: \"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb\") " pod="openshift-insights/insights-runtime-extractor-8hpnj"
Sep 11 17:21:47.610921 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:47.610558    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8hpnj\" (UID: \"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb\") " pod="openshift-insights/insights-runtime-extractor-8hpnj"
Sep 11 17:21:47.610921 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:47.610655    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:21:47.610921 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:21:47.610733    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-insights-runtime-extractor-tls podName:cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb nodeName:}" failed. No retries permitted until 2026-09-11 17:22:03.61071212 +0000 UTC m=+91.414884517 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8hpnj" (UID: "cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:21:47.611058 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:47.611039    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8hpnj\" (UID: \"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb\") " pod="openshift-insights/insights-runtime-extractor-8hpnj"
Sep 11 17:21:50.709396 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:50.709356    2560 scope.go:117] "RemoveContainer" containerID="db0601470ee4a350ddabad1a2752ac204feb814cd9a97fea78954df59243cf72"
Sep 11 17:21:51.164745 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:51.164714    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6fcjs_477cac01-ab9e-4441-b07d-b7e53401c616/console-operator/1.log"
Sep 11 17:21:51.164934 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:51.164788    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-6fcjs" event={"ID":"477cac01-ab9e-4441-b07d-b7e53401c616","Type":"ContainerStarted","Data":"b4598484aae346c49296c5483e53cd3bee51a739c073979af516d8c774f7626f"}
Sep 11 17:21:51.165072 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:51.165048    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-6fcjs"
Sep 11 17:21:51.188234 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:51.188178    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-6fcjs" podStartSLOduration=20.784086482 podStartE2EDuration="24.188160238s" totalImagesPullingTime="3.404073756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:21:51.186500574 +0000 UTC m=+78.990672992" watchObservedRunningTime="2026-09-11 17:21:51.188160238 +0000 UTC m=+78.992332656"
Sep 11 17:21:51.750747 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:51.750717    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-6fcjs"
Sep 11 17:21:58.370431 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:58.370318    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5957d5cb9b-dn42r"]
Sep 11 17:21:58.374602 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:58.374586    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5957d5cb9b-dn42r"
Sep 11 17:21:58.379872 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:58.379844    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 17:21:58.380851 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:58.380829    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 17:21:58.380851 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:58.380851    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 11 17:21:58.381058 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:58.380954    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 17:21:58.382597 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:58.382575    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-747d6b5cfd-8p2b2"]
Sep 11 17:21:58.387794 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:58.387773    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5957d5cb9b-dn42r"]
Sep 11 17:21:58.495886 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:58.495843    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdhtp\" (UniqueName: \"kubernetes.io/projected/90d11333-da9d-41e7-b00a-83dbb11c7431-kube-api-access-mdhtp\") pod \"klusterlet-addon-workmgr-5957d5cb9b-dn42r\" (UID: \"90d11333-da9d-41e7-b00a-83dbb11c7431\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5957d5cb9b-dn42r"
Sep 11 17:21:58.496077 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:58.495932    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90d11333-da9d-41e7-b00a-83dbb11c7431-tmp\") pod \"klusterlet-addon-workmgr-5957d5cb9b-dn42r\" (UID: \"90d11333-da9d-41e7-b00a-83dbb11c7431\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5957d5cb9b-dn42r"
Sep 11 17:21:58.496077 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:58.495968    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/90d11333-da9d-41e7-b00a-83dbb11c7431-klusterlet-config\") pod \"klusterlet-addon-workmgr-5957d5cb9b-dn42r\" (UID: \"90d11333-da9d-41e7-b00a-83dbb11c7431\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5957d5cb9b-dn42r"
Sep 11 17:21:58.596550 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:58.596512    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdhtp\" (UniqueName: \"kubernetes.io/projected/90d11333-da9d-41e7-b00a-83dbb11c7431-kube-api-access-mdhtp\") pod \"klusterlet-addon-workmgr-5957d5cb9b-dn42r\" (UID: \"90d11333-da9d-41e7-b00a-83dbb11c7431\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5957d5cb9b-dn42r"
Sep 11 17:21:58.596725 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:58.596570    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90d11333-da9d-41e7-b00a-83dbb11c7431-tmp\") pod \"klusterlet-addon-workmgr-5957d5cb9b-dn42r\" (UID: \"90d11333-da9d-41e7-b00a-83dbb11c7431\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5957d5cb9b-dn42r"
Sep 11 17:21:58.596725 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:58.596593    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/90d11333-da9d-41e7-b00a-83dbb11c7431-klusterlet-config\") pod \"klusterlet-addon-workmgr-5957d5cb9b-dn42r\" (UID: \"90d11333-da9d-41e7-b00a-83dbb11c7431\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5957d5cb9b-dn42r"
Sep 11 17:21:58.597050 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:58.597030    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90d11333-da9d-41e7-b00a-83dbb11c7431-tmp\") pod \"klusterlet-addon-workmgr-5957d5cb9b-dn42r\" (UID: \"90d11333-da9d-41e7-b00a-83dbb11c7431\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5957d5cb9b-dn42r"
Sep 11 17:21:58.599109 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:58.599087    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/90d11333-da9d-41e7-b00a-83dbb11c7431-klusterlet-config\") pod \"klusterlet-addon-workmgr-5957d5cb9b-dn42r\" (UID: \"90d11333-da9d-41e7-b00a-83dbb11c7431\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5957d5cb9b-dn42r"
Sep 11 17:21:58.605248 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:58.605224    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdhtp\" (UniqueName: \"kubernetes.io/projected/90d11333-da9d-41e7-b00a-83dbb11c7431-kube-api-access-mdhtp\") pod \"klusterlet-addon-workmgr-5957d5cb9b-dn42r\" (UID: \"90d11333-da9d-41e7-b00a-83dbb11c7431\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5957d5cb9b-dn42r"
Sep 11 17:21:58.684192 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:58.684106    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5957d5cb9b-dn42r"
Sep 11 17:21:58.955076 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:58.954932    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5957d5cb9b-dn42r"]
Sep 11 17:21:58.958765 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:21:58.958732    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d11333_da9d_41e7_b00a_83dbb11c7431.slice/crio-456d5c6872ca4275b34b3746b0f16b1c9aa865ca3e85988648ce082cc8055d22 WatchSource:0}: Error finding container 456d5c6872ca4275b34b3746b0f16b1c9aa865ca3e85988648ce082cc8055d22: Status 404 returned error can't find the container with id 456d5c6872ca4275b34b3746b0f16b1c9aa865ca3e85988648ce082cc8055d22
Sep 11 17:21:59.191192 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:21:59.191159    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5957d5cb9b-dn42r" event={"ID":"90d11333-da9d-41e7-b00a-83dbb11c7431","Type":"ContainerStarted","Data":"456d5c6872ca4275b34b3746b0f16b1c9aa865ca3e85988648ce082cc8055d22"}
Sep 11 17:22:02.753700 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:02.753665    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-kh7vx"]
Sep 11 17:22:02.758145 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:02.758127    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-kh7vx"
Sep 11 17:22:02.760658 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:02.760638    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 11 17:22:02.760658 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:02.760652    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-njwj2\""
Sep 11 17:22:02.765948 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:02.765922    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-kh7vx"]
Sep 11 17:22:02.833761 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:02.833713    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/58c88952-e2eb-4729-a21b-251d7d1e6278-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-kh7vx\" (UID: \"58c88952-e2eb-4729-a21b-251d7d1e6278\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-kh7vx"
Sep 11 17:22:02.934168 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:02.934131    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/58c88952-e2eb-4729-a21b-251d7d1e6278-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-kh7vx\" (UID: \"58c88952-e2eb-4729-a21b-251d7d1e6278\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-kh7vx"
Sep 11 17:22:02.936553 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:02.936526    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/58c88952-e2eb-4729-a21b-251d7d1e6278-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-kh7vx\" (UID: \"58c88952-e2eb-4729-a21b-251d7d1e6278\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-kh7vx"
Sep 11 17:22:03.068295 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:03.068204    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-kh7vx"
Sep 11 17:22:03.343806 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:03.343772    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-kh7vx"]
Sep 11 17:22:03.346746 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:22:03.346718    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c88952_e2eb_4729_a21b_251d7d1e6278.slice/crio-0ed385e477d13c28f48ea0ccd9f619e52ff828cb4149dcd7afe9093b8fec5f4f WatchSource:0}: Error finding container 0ed385e477d13c28f48ea0ccd9f619e52ff828cb4149dcd7afe9093b8fec5f4f: Status 404 returned error can't find the container with id 0ed385e477d13c28f48ea0ccd9f619e52ff828cb4149dcd7afe9093b8fec5f4f
Sep 11 17:22:03.643266 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:03.643170    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8hpnj\" (UID: \"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb\") " pod="openshift-insights/insights-runtime-extractor-8hpnj"
Sep 11 17:22:03.645519 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:03.645496    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8hpnj\" (UID: \"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb\") " pod="openshift-insights/insights-runtime-extractor-8hpnj"
Sep 11 17:22:03.874241 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:03.874213    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-blz79\""
Sep 11 17:22:03.878924 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:03.878908    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8hpnj"
Sep 11 17:22:04.152701 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:04.152664    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8hpnj"]
Sep 11 17:22:04.157883 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:22:04.157852    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8c333a_4c92_4b4d_8cc4_8acf75d3aafb.slice/crio-7b5b52d9d9117f17539c504de158aa910ddfdb8eca2ff58c1531715908afc940 WatchSource:0}: Error finding container 7b5b52d9d9117f17539c504de158aa910ddfdb8eca2ff58c1531715908afc940: Status 404 returned error can't find the container with id 7b5b52d9d9117f17539c504de158aa910ddfdb8eca2ff58c1531715908afc940
Sep 11 17:22:04.208285 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:04.208254    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-kh7vx" event={"ID":"58c88952-e2eb-4729-a21b-251d7d1e6278","Type":"ContainerStarted","Data":"0ed385e477d13c28f48ea0ccd9f619e52ff828cb4149dcd7afe9093b8fec5f4f"}
Sep 11 17:22:04.209772 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:04.209743    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8hpnj" event={"ID":"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb","Type":"ContainerStarted","Data":"7b5b52d9d9117f17539c504de158aa910ddfdb8eca2ff58c1531715908afc940"}
Sep 11 17:22:05.218193 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:05.218097    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-kh7vx" event={"ID":"58c88952-e2eb-4729-a21b-251d7d1e6278","Type":"ContainerStarted","Data":"aca40a88107037a84b13e1be39d5723cc3d67ec7416b325367423a4fdc4c8586"}
Sep 11 17:22:05.218193 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:05.218189    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-kh7vx"
Sep 11 17:22:05.220051 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:05.220021    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8hpnj" event={"ID":"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb","Type":"ContainerStarted","Data":"bb2ff0a9e938b467536ee2b02612b617c04781bcedebbed0b6f2d2b5af68fa34"}
Sep 11 17:22:05.220177 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:05.220077    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8hpnj" event={"ID":"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb","Type":"ContainerStarted","Data":"2f614794e8370616dbdf62b21fd03d15b5a17e32daeb269456b090dc1592d720"}
Sep 11 17:22:05.223247 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:05.223226    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-kh7vx"
Sep 11 17:22:05.235434 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:05.235389    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-kh7vx" podStartSLOduration=1.936096914 podStartE2EDuration="3.235375451s" totalImagesPullingTime="1.299278537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:05.234993315 +0000 UTC m=+93.039165736" watchObservedRunningTime="2026-09-11 17:22:05.235375451 +0000 UTC m=+93.039547870"
Sep 11 17:22:05.807387 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:05.807353    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-cc4pf"]
Sep 11 17:22:05.823075 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:05.823028    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-cc4pf"]
Sep 11 17:22:05.823244 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:05.823169    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-cc4pf"
Sep 11 17:22:05.826087 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:05.826061    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 17:22:05.826237 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:05.826208    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 17:22:05.826582 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:05.826561    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 11 17:22:05.826695 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:05.826588    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-s2gmx\""
Sep 11 17:22:05.827164 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:05.827148    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 17:22:05.827215 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:05.827203    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 11 17:22:05.961359 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:05.961322    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25wb5\" (UniqueName: \"kubernetes.io/projected/dc2bf6b3-f8a2-4b4b-972c-919fb11c9000-kube-api-access-25wb5\") pod \"prometheus-operator-68bcd7f987-cc4pf\" (UID: \"dc2bf6b3-f8a2-4b4b-972c-919fb11c9000\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cc4pf"
Sep 11 17:22:05.961541 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:05.961402    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc2bf6b3-f8a2-4b4b-972c-919fb11c9000-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-cc4pf\" (UID: \"dc2bf6b3-f8a2-4b4b-972c-919fb11c9000\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cc4pf"
Sep 11 17:22:05.961541 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:05.961428    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc2bf6b3-f8a2-4b4b-972c-919fb11c9000-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-cc4pf\" (UID: \"dc2bf6b3-f8a2-4b4b-972c-919fb11c9000\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cc4pf"
Sep 11 17:22:05.961541 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:05.961448    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc2bf6b3-f8a2-4b4b-972c-919fb11c9000-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-cc4pf\" (UID: \"dc2bf6b3-f8a2-4b4b-972c-919fb11c9000\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cc4pf"
Sep 11 17:22:06.062323 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:06.062242    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc2bf6b3-f8a2-4b4b-972c-919fb11c9000-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-cc4pf\" (UID: \"dc2bf6b3-f8a2-4b4b-972c-919fb11c9000\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cc4pf"
Sep 11 17:22:06.062323 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:06.062286    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc2bf6b3-f8a2-4b4b-972c-919fb11c9000-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-cc4pf\" (UID: \"dc2bf6b3-f8a2-4b4b-972c-919fb11c9000\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cc4pf"
Sep 11 17:22:06.062323 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:06.062322    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc2bf6b3-f8a2-4b4b-972c-919fb11c9000-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-cc4pf\" (UID: \"dc2bf6b3-f8a2-4b4b-972c-919fb11c9000\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cc4pf"
Sep 11 17:22:06.062599 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:06.062376    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25wb5\" (UniqueName: \"kubernetes.io/projected/dc2bf6b3-f8a2-4b4b-972c-919fb11c9000-kube-api-access-25wb5\") pod \"prometheus-operator-68bcd7f987-cc4pf\" (UID: \"dc2bf6b3-f8a2-4b4b-972c-919fb11c9000\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cc4pf"
Sep 11 17:22:06.062599 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:22:06.062425    2560 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 11 17:22:06.062599 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:22:06.062505    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc2bf6b3-f8a2-4b4b-972c-919fb11c9000-prometheus-operator-tls podName:dc2bf6b3-f8a2-4b4b-972c-919fb11c9000 nodeName:}" failed. No retries permitted until 2026-09-11 17:22:06.562483918 +0000 UTC m=+94.366656319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/dc2bf6b3-f8a2-4b4b-972c-919fb11c9000-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-cc4pf" (UID: "dc2bf6b3-f8a2-4b4b-972c-919fb11c9000") : secret "prometheus-operator-tls" not found
Sep 11 17:22:06.063147 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:06.063124    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc2bf6b3-f8a2-4b4b-972c-919fb11c9000-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-cc4pf\" (UID: \"dc2bf6b3-f8a2-4b4b-972c-919fb11c9000\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cc4pf"
Sep 11 17:22:06.065284 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:06.065261    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc2bf6b3-f8a2-4b4b-972c-919fb11c9000-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-cc4pf\" (UID: \"dc2bf6b3-f8a2-4b4b-972c-919fb11c9000\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cc4pf"
Sep 11 17:22:06.073268 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:06.073245    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25wb5\" (UniqueName: \"kubernetes.io/projected/dc2bf6b3-f8a2-4b4b-972c-919fb11c9000-kube-api-access-25wb5\") pod \"prometheus-operator-68bcd7f987-cc4pf\" (UID: \"dc2bf6b3-f8a2-4b4b-972c-919fb11c9000\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cc4pf"
Sep 11 17:22:06.567043 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:06.567005    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc2bf6b3-f8a2-4b4b-972c-919fb11c9000-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-cc4pf\" (UID: \"dc2bf6b3-f8a2-4b4b-972c-919fb11c9000\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cc4pf"
Sep 11 17:22:06.569914 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:06.569864    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc2bf6b3-f8a2-4b4b-972c-919fb11c9000-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-cc4pf\" (UID: \"dc2bf6b3-f8a2-4b4b-972c-919fb11c9000\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cc4pf"
Sep 11 17:22:06.732818 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:06.732788    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-cc4pf"
Sep 11 17:22:07.010058 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:07.009888    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-cc4pf"]
Sep 11 17:22:07.012449 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:22:07.012424    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2bf6b3_f8a2_4b4b_972c_919fb11c9000.slice/crio-8590f32e823bd2e7274ee497dcd79726bbf891c933fee8ed9017d50047aef0ac WatchSource:0}: Error finding container 8590f32e823bd2e7274ee497dcd79726bbf891c933fee8ed9017d50047aef0ac: Status 404 returned error can't find the container with id 8590f32e823bd2e7274ee497dcd79726bbf891c933fee8ed9017d50047aef0ac
Sep 11 17:22:07.228446 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:07.228350    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-cc4pf" event={"ID":"dc2bf6b3-f8a2-4b4b-972c-919fb11c9000","Type":"ContainerStarted","Data":"8590f32e823bd2e7274ee497dcd79726bbf891c933fee8ed9017d50047aef0ac"}
Sep 11 17:22:07.230406 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:07.230376    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8hpnj" event={"ID":"cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb","Type":"ContainerStarted","Data":"1678a4aec09ebb1c0c8285973bd0af300cff280af404f7adc30eec83d7ed10b0"}
Sep 11 17:22:07.250691 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:07.250637    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8hpnj" podStartSLOduration=33.604577791 podStartE2EDuration="36.2506192s" totalImagesPullingTime="2.646041409s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:21:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:07.250339399 +0000 UTC m=+95.054511818" watchObservedRunningTime="2026-09-11 17:22:07.2506192 +0000 UTC m=+95.054791629"
Sep 11 17:22:08.390136 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:08.390097    2560 patch_prober.go:28] interesting pod/image-registry-747d6b5cfd-8p2b2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 17:22:08.390506 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:08.390156    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2" podUID="9abdd1bd-edc8-41c3-91e6-da724784b425" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 17:22:09.244510 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:09.244468    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-cc4pf" event={"ID":"dc2bf6b3-f8a2-4b4b-972c-919fb11c9000","Type":"ContainerStarted","Data":"6fd2718f1e6ff8fec2d9a46cb6504c0e5695dd897102a0fa1400428e5e677710"}
Sep 11 17:22:09.244813 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:09.244567    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-cc4pf" event={"ID":"dc2bf6b3-f8a2-4b4b-972c-919fb11c9000","Type":"ContainerStarted","Data":"7ea7e3919f1f59339f2c17cd9c73b3dc3d83d304a98debc6506d941cb2a11706"}
Sep 11 17:22:09.265702 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:09.265631    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-cc4pf" podStartSLOduration=2.824643403 podStartE2EDuration="4.265608745s" totalImagesPullingTime="1.440965342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:09.264160437 +0000 UTC m=+97.068332863" watchObservedRunningTime="2026-09-11 17:22:09.265608745 +0000 UTC m=+97.069781164"
Sep 11 17:22:10.443241 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.443198    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5578774b47-z7pkw"]
Sep 11 17:22:10.452514 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.452480    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:22:10.455556 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.455419    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 17:22:10.455556 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.455464    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 11 17:22:10.456364 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.456337    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 17:22:10.456497 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.456388    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 17:22:10.456559 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.456546    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 11 17:22:10.456686 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.456668    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 17:22:10.456806 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.456547    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9tzbh\""
Sep 11 17:22:10.457080 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.457059    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 17:22:10.459466 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.459402    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5578774b47-z7pkw"]
Sep 11 17:22:10.463517 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.463496    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 11 17:22:10.600495 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.600450    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-console-oauth-config\") pod \"console-5578774b47-z7pkw\" (UID: \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\") " pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:22:10.600675 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.600539    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkb7x\" (UniqueName: \"kubernetes.io/projected/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-kube-api-access-dkb7x\") pod \"console-5578774b47-z7pkw\" (UID: \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\") " pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:22:10.600675 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.600603    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-console-serving-cert\") pod \"console-5578774b47-z7pkw\" (UID: \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\") " pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:22:10.600675 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.600631    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-service-ca\") pod \"console-5578774b47-z7pkw\" (UID: \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\") " pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:22:10.600920 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.600743    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-console-config\") pod \"console-5578774b47-z7pkw\" (UID: \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\") " pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:22:10.600920 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.600803    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-trusted-ca-bundle\") pod \"console-5578774b47-z7pkw\" (UID: \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\") " pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:22:10.600920 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.600837    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-oauth-serving-cert\") pod \"console-5578774b47-z7pkw\" (UID: \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\") " pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:22:10.702120 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.702027    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-console-oauth-config\") pod \"console-5578774b47-z7pkw\" (UID: \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\") " pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:22:10.702120 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.702083    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkb7x\" (UniqueName: \"kubernetes.io/projected/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-kube-api-access-dkb7x\") pod \"console-5578774b47-z7pkw\" (UID: \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\") " pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:22:10.702315 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.702127    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-console-serving-cert\") pod \"console-5578774b47-z7pkw\" (UID: \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\") " pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:22:10.703057 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.702436    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-service-ca\") pod \"console-5578774b47-z7pkw\" (UID: \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\") " pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:22:10.703057 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.702527    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-console-config\") pod \"console-5578774b47-z7pkw\" (UID: \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\") " pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:22:10.703057 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.702561    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-trusted-ca-bundle\") pod \"console-5578774b47-z7pkw\" (UID: \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\") " pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:22:10.703057 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.702586    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-oauth-serving-cert\") pod \"console-5578774b47-z7pkw\" (UID: \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\") " pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:22:10.703334 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.703237    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-service-ca\") pod \"console-5578774b47-z7pkw\" (UID: \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\") " pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:22:10.703836 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.703813    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-console-config\") pod \"console-5578774b47-z7pkw\" (UID: \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\") " pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:22:10.703952 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.703824    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-oauth-serving-cert\") pod \"console-5578774b47-z7pkw\" (UID: \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\") " pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:22:10.704805 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.704781    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-trusted-ca-bundle\") pod \"console-5578774b47-z7pkw\" (UID: \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\") " pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:22:10.705142 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.705117    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-console-serving-cert\") pod \"console-5578774b47-z7pkw\" (UID: \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\") " pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:22:10.705457 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.705433    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-console-oauth-config\") pod \"console-5578774b47-z7pkw\" (UID: \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\") " pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:22:10.716644 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.716616    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkb7x\" (UniqueName: \"kubernetes.io/projected/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-kube-api-access-dkb7x\") pod \"console-5578774b47-z7pkw\" (UID: \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\") " pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:22:10.765722 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:10.765685    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:22:11.053593 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.053495    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5578774b47-z7pkw"]
Sep 11 17:22:11.235014 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.234973    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-sczrs"]
Sep 11 17:22:11.238878 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.238846    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sczrs"
Sep 11 17:22:11.242055 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.241954    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 11 17:22:11.242055 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.242040    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-fjrtp\""
Sep 11 17:22:11.242279 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.242209    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 11 17:22:11.255599 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.255519    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-sczrs"]
Sep 11 17:22:11.295330 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.295295    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7hvgz"]
Sep 11 17:22:11.299222 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.299194    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:11.302245 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.302225    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 17:22:11.303019 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.302616    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 17:22:11.303019 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.302626    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tszjt\""
Sep 11 17:22:11.303019 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.302857    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 17:22:11.307889 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.307829    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nxwn\" (UniqueName: \"kubernetes.io/projected/b98862eb-9553-4b1d-b808-810adda98bc8-kube-api-access-8nxwn\") pod \"openshift-state-metrics-67dcf8c979-sczrs\" (UID: \"b98862eb-9553-4b1d-b808-810adda98bc8\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sczrs"
Sep 11 17:22:11.307889 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.307861    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b98862eb-9553-4b1d-b808-810adda98bc8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-sczrs\" (UID: \"b98862eb-9553-4b1d-b808-810adda98bc8\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sczrs"
Sep 11 17:22:11.307889 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.307882    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b98862eb-9553-4b1d-b808-810adda98bc8-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-sczrs\" (UID: \"b98862eb-9553-4b1d-b808-810adda98bc8\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sczrs"
Sep 11 17:22:11.308451 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.307947    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b98862eb-9553-4b1d-b808-810adda98bc8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-sczrs\" (UID: \"b98862eb-9553-4b1d-b808-810adda98bc8\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sczrs"
Sep 11 17:22:11.409199 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.409150    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6dadf403-2628-4ab2-83d7-3bf3a3dac462-node-exporter-tls\") pod \"node-exporter-7hvgz\" (UID: \"6dadf403-2628-4ab2-83d7-3bf3a3dac462\") " pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:11.409396 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.409206    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6dadf403-2628-4ab2-83d7-3bf3a3dac462-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7hvgz\" (UID: \"6dadf403-2628-4ab2-83d7-3bf3a3dac462\") " pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:11.409396 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.409233    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j26tg\" (UniqueName: \"kubernetes.io/projected/6dadf403-2628-4ab2-83d7-3bf3a3dac462-kube-api-access-j26tg\") pod \"node-exporter-7hvgz\" (UID: \"6dadf403-2628-4ab2-83d7-3bf3a3dac462\") " pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:11.409396 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.409331    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nxwn\" (UniqueName: \"kubernetes.io/projected/b98862eb-9553-4b1d-b808-810adda98bc8-kube-api-access-8nxwn\") pod \"openshift-state-metrics-67dcf8c979-sczrs\" (UID: \"b98862eb-9553-4b1d-b808-810adda98bc8\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sczrs"
Sep 11 17:22:11.409396 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.409369    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b98862eb-9553-4b1d-b808-810adda98bc8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-sczrs\" (UID: \"b98862eb-9553-4b1d-b808-810adda98bc8\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sczrs"
Sep 11 17:22:11.409620 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.409395    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6dadf403-2628-4ab2-83d7-3bf3a3dac462-node-exporter-textfile\") pod \"node-exporter-7hvgz\" (UID: \"6dadf403-2628-4ab2-83d7-3bf3a3dac462\") " pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:11.409620 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.409445    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b98862eb-9553-4b1d-b808-810adda98bc8-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-sczrs\" (UID: \"b98862eb-9553-4b1d-b808-810adda98bc8\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sczrs"
Sep 11 17:22:11.409620 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.409479    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6dadf403-2628-4ab2-83d7-3bf3a3dac462-node-exporter-wtmp\") pod \"node-exporter-7hvgz\" (UID: \"6dadf403-2628-4ab2-83d7-3bf3a3dac462\") " pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:11.409620 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.409505    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6dadf403-2628-4ab2-83d7-3bf3a3dac462-sys\") pod \"node-exporter-7hvgz\" (UID: \"6dadf403-2628-4ab2-83d7-3bf3a3dac462\") " pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:11.409620 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.409538    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dadf403-2628-4ab2-83d7-3bf3a3dac462-metrics-client-ca\") pod \"node-exporter-7hvgz\" (UID: \"6dadf403-2628-4ab2-83d7-3bf3a3dac462\") " pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:11.409620 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.409569    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6dadf403-2628-4ab2-83d7-3bf3a3dac462-node-exporter-accelerators-collector-config\") pod \"node-exporter-7hvgz\" (UID: \"6dadf403-2628-4ab2-83d7-3bf3a3dac462\") " pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:11.409620 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.409595    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6dadf403-2628-4ab2-83d7-3bf3a3dac462-root\") pod \"node-exporter-7hvgz\" (UID: \"6dadf403-2628-4ab2-83d7-3bf3a3dac462\") " pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:11.409987 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.409929    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b98862eb-9553-4b1d-b808-810adda98bc8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-sczrs\" (UID: \"b98862eb-9553-4b1d-b808-810adda98bc8\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sczrs"
Sep 11 17:22:11.410650 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.410619    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b98862eb-9553-4b1d-b808-810adda98bc8-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-sczrs\" (UID: \"b98862eb-9553-4b1d-b808-810adda98bc8\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sczrs"
Sep 11 17:22:11.412437 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.412384    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b98862eb-9553-4b1d-b808-810adda98bc8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-sczrs\" (UID: \"b98862eb-9553-4b1d-b808-810adda98bc8\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sczrs"
Sep 11 17:22:11.412950 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.412926    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b98862eb-9553-4b1d-b808-810adda98bc8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-sczrs\" (UID: \"b98862eb-9553-4b1d-b808-810adda98bc8\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sczrs"
Sep 11 17:22:11.423568 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.423538    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nxwn\" (UniqueName: \"kubernetes.io/projected/b98862eb-9553-4b1d-b808-810adda98bc8-kube-api-access-8nxwn\") pod \"openshift-state-metrics-67dcf8c979-sczrs\" (UID: \"b98862eb-9553-4b1d-b808-810adda98bc8\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sczrs"
Sep 11 17:22:11.511352 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.510325    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1dbb8c42-1654-4f42-8237-cf79927d51db-metrics-tls\") pod \"dns-default-mp4vh\" (UID: \"1dbb8c42-1654-4f42-8237-cf79927d51db\") " pod="openshift-dns/dns-default-mp4vh"
Sep 11 17:22:11.511352 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.510406    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6dadf403-2628-4ab2-83d7-3bf3a3dac462-node-exporter-textfile\") pod \"node-exporter-7hvgz\" (UID: \"6dadf403-2628-4ab2-83d7-3bf3a3dac462\") " pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:11.511352 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.510451    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6dadf403-2628-4ab2-83d7-3bf3a3dac462-node-exporter-wtmp\") pod \"node-exporter-7hvgz\" (UID: \"6dadf403-2628-4ab2-83d7-3bf3a3dac462\") " pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:11.511352 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.510478    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6dadf403-2628-4ab2-83d7-3bf3a3dac462-sys\") pod \"node-exporter-7hvgz\" (UID: \"6dadf403-2628-4ab2-83d7-3bf3a3dac462\") " pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:11.511352 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.510510    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dadf403-2628-4ab2-83d7-3bf3a3dac462-metrics-client-ca\") pod \"node-exporter-7hvgz\" (UID: \"6dadf403-2628-4ab2-83d7-3bf3a3dac462\") " pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:11.511352 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.510539    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6dadf403-2628-4ab2-83d7-3bf3a3dac462-node-exporter-accelerators-collector-config\") pod \"node-exporter-7hvgz\" (UID: \"6dadf403-2628-4ab2-83d7-3bf3a3dac462\") " pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:11.511352 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.510569    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6dadf403-2628-4ab2-83d7-3bf3a3dac462-root\") pod \"node-exporter-7hvgz\" (UID: \"6dadf403-2628-4ab2-83d7-3bf3a3dac462\") " pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:11.511352 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.510610    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6dadf403-2628-4ab2-83d7-3bf3a3dac462-node-exporter-tls\") pod \"node-exporter-7hvgz\" (UID: \"6dadf403-2628-4ab2-83d7-3bf3a3dac462\") " pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:11.511352 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.510635    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6dadf403-2628-4ab2-83d7-3bf3a3dac462-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7hvgz\" (UID: \"6dadf403-2628-4ab2-83d7-3bf3a3dac462\") " pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:11.511352 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.510641    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6dadf403-2628-4ab2-83d7-3bf3a3dac462-node-exporter-wtmp\") pod \"node-exporter-7hvgz\" (UID: \"6dadf403-2628-4ab2-83d7-3bf3a3dac462\") " pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:11.511352 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.510659    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j26tg\" (UniqueName: \"kubernetes.io/projected/6dadf403-2628-4ab2-83d7-3bf3a3dac462-kube-api-access-j26tg\") pod \"node-exporter-7hvgz\" (UID: \"6dadf403-2628-4ab2-83d7-3bf3a3dac462\") " pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:11.511352 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.510749    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6dadf403-2628-4ab2-83d7-3bf3a3dac462-node-exporter-textfile\") pod \"node-exporter-7hvgz\" (UID: \"6dadf403-2628-4ab2-83d7-3bf3a3dac462\") " pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:11.511352 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.510807    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6dadf403-2628-4ab2-83d7-3bf3a3dac462-sys\") pod \"node-exporter-7hvgz\" (UID: \"6dadf403-2628-4ab2-83d7-3bf3a3dac462\") " pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:11.511352 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:22:11.510917    2560 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 11 17:22:11.511352 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:22:11.510978    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dadf403-2628-4ab2-83d7-3bf3a3dac462-node-exporter-tls podName:6dadf403-2628-4ab2-83d7-3bf3a3dac462 nodeName:}" failed. No retries permitted until 2026-09-11 17:22:12.010958949 +0000 UTC m=+99.815131362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/6dadf403-2628-4ab2-83d7-3bf3a3dac462-node-exporter-tls") pod "node-exporter-7hvgz" (UID: "6dadf403-2628-4ab2-83d7-3bf3a3dac462") : secret "node-exporter-tls" not found
Sep 11 17:22:11.511352 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.511047    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6dadf403-2628-4ab2-83d7-3bf3a3dac462-root\") pod \"node-exporter-7hvgz\" (UID: \"6dadf403-2628-4ab2-83d7-3bf3a3dac462\") " pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:11.511352 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.511220    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6dadf403-2628-4ab2-83d7-3bf3a3dac462-node-exporter-accelerators-collector-config\") pod \"node-exporter-7hvgz\" (UID: \"6dadf403-2628-4ab2-83d7-3bf3a3dac462\") " pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:11.512690 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.511613    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dadf403-2628-4ab2-83d7-3bf3a3dac462-metrics-client-ca\") pod \"node-exporter-7hvgz\" (UID: \"6dadf403-2628-4ab2-83d7-3bf3a3dac462\") " pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:11.513609 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.513587    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6dadf403-2628-4ab2-83d7-3bf3a3dac462-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7hvgz\" (UID: \"6dadf403-2628-4ab2-83d7-3bf3a3dac462\") " pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:11.513739 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.513597    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1dbb8c42-1654-4f42-8237-cf79927d51db-metrics-tls\") pod \"dns-default-mp4vh\" (UID: \"1dbb8c42-1654-4f42-8237-cf79927d51db\") " pod="openshift-dns/dns-default-mp4vh"
Sep 11 17:22:11.523037 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.523009    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j26tg\" (UniqueName: \"kubernetes.io/projected/6dadf403-2628-4ab2-83d7-3bf3a3dac462-kube-api-access-j26tg\") pod \"node-exporter-7hvgz\" (UID: \"6dadf403-2628-4ab2-83d7-3bf3a3dac462\") " pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:11.551464 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.551421    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sczrs"
Sep 11 17:22:11.805369 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.805336    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n6xk4\""
Sep 11 17:22:11.812529 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:11.812393    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mp4vh"
Sep 11 17:22:12.015730 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:12.015689    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6dadf403-2628-4ab2-83d7-3bf3a3dac462-node-exporter-tls\") pod \"node-exporter-7hvgz\" (UID: \"6dadf403-2628-4ab2-83d7-3bf3a3dac462\") " pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:12.018551 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:12.018524    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6dadf403-2628-4ab2-83d7-3bf3a3dac462-node-exporter-tls\") pod \"node-exporter-7hvgz\" (UID: \"6dadf403-2628-4ab2-83d7-3bf3a3dac462\") " pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:12.212376 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:12.212335    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7hvgz"
Sep 11 17:22:12.239437 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:22:12.239405    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11d64f96_b6f7_41f3_8dde_c8dd2fb2f4e5.slice/crio-44724f980a1d8fcd3d2fe4bf4c0ebe47897e3436b8bfb893b31b62458d7c145b WatchSource:0}: Error finding container 44724f980a1d8fcd3d2fe4bf4c0ebe47897e3436b8bfb893b31b62458d7c145b: Status 404 returned error can't find the container with id 44724f980a1d8fcd3d2fe4bf4c0ebe47897e3436b8bfb893b31b62458d7c145b
Sep 11 17:22:12.251878 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:22:12.251843    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dadf403_2628_4ab2_83d7_3bf3a3dac462.slice/crio-1abad75dd2839f10ca0aaeb9773879abf0f3ae49f26f406475ac06b6112a74d8 WatchSource:0}: Error finding container 1abad75dd2839f10ca0aaeb9773879abf0f3ae49f26f406475ac06b6112a74d8: Status 404 returned error can't find the container with id 1abad75dd2839f10ca0aaeb9773879abf0f3ae49f26f406475ac06b6112a74d8
Sep 11 17:22:12.258610 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:12.258577    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7hvgz" event={"ID":"6dadf403-2628-4ab2-83d7-3bf3a3dac462","Type":"ContainerStarted","Data":"1abad75dd2839f10ca0aaeb9773879abf0f3ae49f26f406475ac06b6112a74d8"}
Sep 11 17:22:12.260250 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:12.260223    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5578774b47-z7pkw" event={"ID":"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5","Type":"ContainerStarted","Data":"44724f980a1d8fcd3d2fe4bf4c0ebe47897e3436b8bfb893b31b62458d7c145b"}
Sep 11 17:22:12.547555 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:12.547518    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-sczrs"]
Sep 11 17:22:12.549047 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:12.549019    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mp4vh"]
Sep 11 17:22:12.549142 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:22:12.549092    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98862eb_9553_4b1d_b808_810adda98bc8.slice/crio-7831428676aa813825c53ad65b922448d7296429eae819bd7f69cd2631595882 WatchSource:0}: Error finding container 7831428676aa813825c53ad65b922448d7296429eae819bd7f69cd2631595882: Status 404 returned error can't find the container with id 7831428676aa813825c53ad65b922448d7296429eae819bd7f69cd2631595882
Sep 11 17:22:12.549744 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:22:12.549721    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dbb8c42_1654_4f42_8237_cf79927d51db.slice/crio-8db31a9aeaeef213d0c854190bc52833d3c6639fa147cbd036fe086da4e4b498 WatchSource:0}: Error finding container 8db31a9aeaeef213d0c854190bc52833d3c6639fa147cbd036fe086da4e4b498: Status 404 returned error can't find the container with id 8db31a9aeaeef213d0c854190bc52833d3c6639fa147cbd036fe086da4e4b498
Sep 11 17:22:13.268051 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.267732    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sczrs" event={"ID":"b98862eb-9553-4b1d-b808-810adda98bc8","Type":"ContainerStarted","Data":"30f6793b5cd321bfa44a08041d3024c0d013e069a98ac602a38d4d3e4143d171"}
Sep 11 17:22:13.268051 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.267810    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sczrs" event={"ID":"b98862eb-9553-4b1d-b808-810adda98bc8","Type":"ContainerStarted","Data":"c3a00955e4b887b77bd48300b5bc41e5b15021269d27e70a6dca12996b2cfec1"}
Sep 11 17:22:13.268051 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.267840    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sczrs" event={"ID":"b98862eb-9553-4b1d-b808-810adda98bc8","Type":"ContainerStarted","Data":"7831428676aa813825c53ad65b922448d7296429eae819bd7f69cd2631595882"}
Sep 11 17:22:13.270481 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.270451    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5957d5cb9b-dn42r" event={"ID":"90d11333-da9d-41e7-b00a-83dbb11c7431","Type":"ContainerStarted","Data":"082565a6d058b0b3aae8aef065de8c9fab9fc47dbb94ffd1b0bf299657cefd74"}
Sep 11 17:22:13.271029 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.271005    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5957d5cb9b-dn42r"
Sep 11 17:22:13.272585 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.272559    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mp4vh" event={"ID":"1dbb8c42-1654-4f42-8237-cf79927d51db","Type":"ContainerStarted","Data":"8db31a9aeaeef213d0c854190bc52833d3c6639fa147cbd036fe086da4e4b498"}
Sep 11 17:22:13.272794 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.272771    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5957d5cb9b-dn42r"
Sep 11 17:22:13.359058 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.358847    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5957d5cb9b-dn42r" podStartSLOduration=1.971744357 podStartE2EDuration="15.358822846s" totalImagesPullingTime="13.387078489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:21:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:13.322227067 +0000 UTC m=+101.126399483" watchObservedRunningTime="2026-09-11 17:22:13.358822846 +0000 UTC m=+101.162995266"
Sep 11 17:22:13.445129 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.443275    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"]
Sep 11 17:22:13.453050 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.451824    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"
Sep 11 17:22:13.457835 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.457808    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 11 17:22:13.460476 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.460449    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 11 17:22:13.465393 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.464761    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 11 17:22:13.465393 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.465069    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2rjhg2hain5gc\""
Sep 11 17:22:13.465393 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.465314    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 11 17:22:13.466421 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.466398    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 11 17:22:13.466510 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.466444    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-br4xb\""
Sep 11 17:22:13.478823 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.478795    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"]
Sep 11 17:22:13.531365 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.531272    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5b2f4bf0-4752-4905-ba63-3879494596bc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f8d4ccfc-7n5f4\" (UID: \"5b2f4bf0-4752-4905-ba63-3879494596bc\") " pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"
Sep 11 17:22:13.531365 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.531352    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5b2f4bf0-4752-4905-ba63-3879494596bc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f8d4ccfc-7n5f4\" (UID: \"5b2f4bf0-4752-4905-ba63-3879494596bc\") " pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"
Sep 11 17:22:13.531583 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.531419    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b2f4bf0-4752-4905-ba63-3879494596bc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f8d4ccfc-7n5f4\" (UID: \"5b2f4bf0-4752-4905-ba63-3879494596bc\") " pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"
Sep 11 17:22:13.531583 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.531475    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z552\" (UniqueName: \"kubernetes.io/projected/5b2f4bf0-4752-4905-ba63-3879494596bc-kube-api-access-2z552\") pod \"thanos-querier-7f8d4ccfc-7n5f4\" (UID: \"5b2f4bf0-4752-4905-ba63-3879494596bc\") " pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"
Sep 11 17:22:13.531583 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.531506    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b2f4bf0-4752-4905-ba63-3879494596bc-metrics-client-ca\") pod \"thanos-querier-7f8d4ccfc-7n5f4\" (UID: \"5b2f4bf0-4752-4905-ba63-3879494596bc\") " pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"
Sep 11 17:22:13.531746 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.531616    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b2f4bf0-4752-4905-ba63-3879494596bc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f8d4ccfc-7n5f4\" (UID: \"5b2f4bf0-4752-4905-ba63-3879494596bc\") " pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"
Sep 11 17:22:13.531746 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.531652    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5b2f4bf0-4752-4905-ba63-3879494596bc-secret-grpc-tls\") pod \"thanos-querier-7f8d4ccfc-7n5f4\" (UID: \"5b2f4bf0-4752-4905-ba63-3879494596bc\") " pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"
Sep 11 17:22:13.531746 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.531731    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5b2f4bf0-4752-4905-ba63-3879494596bc-secret-thanos-querier-tls\") pod \"thanos-querier-7f8d4ccfc-7n5f4\" (UID: \"5b2f4bf0-4752-4905-ba63-3879494596bc\") " pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"
Sep 11 17:22:13.632630 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.632588    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b2f4bf0-4752-4905-ba63-3879494596bc-metrics-client-ca\") pod \"thanos-querier-7f8d4ccfc-7n5f4\" (UID: \"5b2f4bf0-4752-4905-ba63-3879494596bc\") " pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"
Sep 11 17:22:13.633135 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.632665    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b2f4bf0-4752-4905-ba63-3879494596bc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f8d4ccfc-7n5f4\" (UID: \"5b2f4bf0-4752-4905-ba63-3879494596bc\") " pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"
Sep 11 17:22:13.633135 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.632693    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5b2f4bf0-4752-4905-ba63-3879494596bc-secret-grpc-tls\") pod \"thanos-querier-7f8d4ccfc-7n5f4\" (UID: \"5b2f4bf0-4752-4905-ba63-3879494596bc\") " pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"
Sep 11 17:22:13.633135 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.632742    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5b2f4bf0-4752-4905-ba63-3879494596bc-secret-thanos-querier-tls\") pod \"thanos-querier-7f8d4ccfc-7n5f4\" (UID: \"5b2f4bf0-4752-4905-ba63-3879494596bc\") " pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"
Sep 11 17:22:13.633135 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.632774    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5b2f4bf0-4752-4905-ba63-3879494596bc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f8d4ccfc-7n5f4\" (UID: \"5b2f4bf0-4752-4905-ba63-3879494596bc\") " pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"
Sep 11 17:22:13.633135 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.632838    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5b2f4bf0-4752-4905-ba63-3879494596bc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f8d4ccfc-7n5f4\" (UID: \"5b2f4bf0-4752-4905-ba63-3879494596bc\") " pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"
Sep 11 17:22:13.633135 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.632865    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b2f4bf0-4752-4905-ba63-3879494596bc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f8d4ccfc-7n5f4\" (UID: \"5b2f4bf0-4752-4905-ba63-3879494596bc\") " pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"
Sep 11 17:22:13.633135 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.632924    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z552\" (UniqueName: \"kubernetes.io/projected/5b2f4bf0-4752-4905-ba63-3879494596bc-kube-api-access-2z552\") pod \"thanos-querier-7f8d4ccfc-7n5f4\" (UID: \"5b2f4bf0-4752-4905-ba63-3879494596bc\") " pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"
Sep 11 17:22:13.633483 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.633459    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b2f4bf0-4752-4905-ba63-3879494596bc-metrics-client-ca\") pod \"thanos-querier-7f8d4ccfc-7n5f4\" (UID: \"5b2f4bf0-4752-4905-ba63-3879494596bc\") " pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"
Sep 11 17:22:13.636853 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.636821    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b2f4bf0-4752-4905-ba63-3879494596bc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f8d4ccfc-7n5f4\" (UID: \"5b2f4bf0-4752-4905-ba63-3879494596bc\") " pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"
Sep 11 17:22:13.637135 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.637111    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b2f4bf0-4752-4905-ba63-3879494596bc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f8d4ccfc-7n5f4\" (UID: \"5b2f4bf0-4752-4905-ba63-3879494596bc\") " pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"
Sep 11 17:22:13.638046 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.638023    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5b2f4bf0-4752-4905-ba63-3879494596bc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f8d4ccfc-7n5f4\" (UID: \"5b2f4bf0-4752-4905-ba63-3879494596bc\") " pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"
Sep 11 17:22:13.638140 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.638031    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5b2f4bf0-4752-4905-ba63-3879494596bc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f8d4ccfc-7n5f4\" (UID: \"5b2f4bf0-4752-4905-ba63-3879494596bc\") " pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"
Sep 11 17:22:13.638554 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.638507    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5b2f4bf0-4752-4905-ba63-3879494596bc-secret-grpc-tls\") pod \"thanos-querier-7f8d4ccfc-7n5f4\" (UID: \"5b2f4bf0-4752-4905-ba63-3879494596bc\") " pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"
Sep 11 17:22:13.638782 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.638758    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5b2f4bf0-4752-4905-ba63-3879494596bc-secret-thanos-querier-tls\") pod \"thanos-querier-7f8d4ccfc-7n5f4\" (UID: \"5b2f4bf0-4752-4905-ba63-3879494596bc\") " pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"
Sep 11 17:22:13.644969 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.644936    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z552\" (UniqueName: \"kubernetes.io/projected/5b2f4bf0-4752-4905-ba63-3879494596bc-kube-api-access-2z552\") pod \"thanos-querier-7f8d4ccfc-7n5f4\" (UID: \"5b2f4bf0-4752-4905-ba63-3879494596bc\") " pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"
Sep 11 17:22:13.769439 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:13.769390    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"
Sep 11 17:22:15.145405 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:15.145374    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xjqcw"
Sep 11 17:22:16.289576 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:16.288329    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sczrs" event={"ID":"b98862eb-9553-4b1d-b808-810adda98bc8","Type":"ContainerStarted","Data":"a558a9410885a2ca18a6517263ab1a37691dfe49367b37114a40f8e2edbe7646"}
Sep 11 17:22:16.296137 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:16.296114    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7hvgz" event={"ID":"6dadf403-2628-4ab2-83d7-3bf3a3dac462","Type":"ContainerStarted","Data":"f611f50a0c86d08756bebfb184117429a8142dd8a59d7756a0b46c54b6701f0f"}
Sep 11 17:22:16.299194 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:16.299139    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5578774b47-z7pkw" event={"ID":"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5","Type":"ContainerStarted","Data":"1275d3317d30ea4f97718806c5bd0c23fb03c9479df48b1c62975563cd0d92a8"}
Sep 11 17:22:16.325012 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:16.324765    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sczrs" podStartSLOduration=1.9946954730000002 podStartE2EDuration="5.324743755s" totalImagesPullingTime="3.330048282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:22:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:16.323888998 +0000 UTC m=+104.128061419" watchObservedRunningTime="2026-09-11 17:22:16.324743755 +0000 UTC m=+104.128916175"
Sep 11 17:22:16.378248 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:16.378196    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5578774b47-z7pkw" podStartSLOduration=2.46667133 podStartE2EDuration="6.37817805s" totalImagesPullingTime="3.91150672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:22:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:16.377189796 +0000 UTC m=+104.181362215" watchObservedRunningTime="2026-09-11 17:22:16.37817805 +0000 UTC m=+104.182350469"
Sep 11 17:22:16.437748 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:16.437641    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"]
Sep 11 17:22:16.441734 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:22:16.441688    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b2f4bf0_4752_4905_ba63_3879494596bc.slice/crio-c5ed98f6d4a944e232afc64fc876547995ff5c0e43be2c5e40a13333584a9bbb WatchSource:0}: Error finding container c5ed98f6d4a944e232afc64fc876547995ff5c0e43be2c5e40a13333584a9bbb: Status 404 returned error can't find the container with id c5ed98f6d4a944e232afc64fc876547995ff5c0e43be2c5e40a13333584a9bbb
Sep 11 17:22:17.304008 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.303968    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mp4vh" event={"ID":"1dbb8c42-1654-4f42-8237-cf79927d51db","Type":"ContainerStarted","Data":"33805c8834b0a2907a911aec8b5ccb96e68d44485d22d95fdeaaaac1df66d330"}
Sep 11 17:22:17.304451 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.304021    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mp4vh" event={"ID":"1dbb8c42-1654-4f42-8237-cf79927d51db","Type":"ContainerStarted","Data":"cd2f47af610d670c7f90659d28dd8cc343f8ceba45e6cc3dd8a9e30d6f40d3b9"}
Sep 11 17:22:17.304451 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.304084    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mp4vh"
Sep 11 17:22:17.305237 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.305216    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4" event={"ID":"5b2f4bf0-4752-4905-ba63-3879494596bc","Type":"ContainerStarted","Data":"c5ed98f6d4a944e232afc64fc876547995ff5c0e43be2c5e40a13333584a9bbb"}
Sep 11 17:22:17.306636 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.306613    2560 generic.go:356] "Generic (PLEG): container finished" podID="6dadf403-2628-4ab2-83d7-3bf3a3dac462" containerID="f611f50a0c86d08756bebfb184117429a8142dd8a59d7756a0b46c54b6701f0f" exitCode=0
Sep 11 17:22:17.306733 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.306676    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7hvgz" event={"ID":"6dadf403-2628-4ab2-83d7-3bf3a3dac462","Type":"ContainerDied","Data":"f611f50a0c86d08756bebfb184117429a8142dd8a59d7756a0b46c54b6701f0f"}
Sep 11 17:22:17.325855 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.325810    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mp4vh" podStartSLOduration=66.729478401 podStartE2EDuration="1m10.325794638s" totalImagesPullingTime="3.596316237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:21:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:17.324875336 +0000 UTC m=+105.129047757" watchObservedRunningTime="2026-09-11 17:22:17.325794638 +0000 UTC m=+105.129967056"
Sep 11 17:22:17.727723 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.727640    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 17:22:17.731097 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.731070    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.739012 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:22:17.738963    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"prometheus-k8s-tls\" is forbidden: User \"system:node:ip-10-0-130-89.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-89.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" type="*v1.Secret"
Sep 11 17:22:17.739242 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:22:17.739214    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"prometheus-k8s\" is forbidden: User \"system:node:ip-10-0-130-89.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-89.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" type="*v1.Secret"
Sep 11 17:22:17.739315 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:22:17.739239    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"prometheus-k8s-kube-rbac-proxy-web\" is forbidden: User \"system:node:ip-10-0-130-89.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-89.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" type="*v1.Secret"
Sep 11 17:22:17.740540 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.740519    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 17:22:17.742002 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.741888    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 17:22:17.742002 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.741980    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 17:22:17.743058 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:22:17.743018    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"prometheus-k8s-0\" is forbidden: User \"system:node:ip-10-0-130-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-89.ec2.internal' and this object" podUID="5c4004f2-61ce-4890-9c2a-766558008eba" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.745381 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.745359    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a228j75t456ke\""
Sep 11 17:22:17.745503 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.745406    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 17:22:17.745503 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.745492    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 17:22:17.745838 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.745820    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 17:22:17.753987 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.753965    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 17:22:17.754175 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.754138    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 17:22:17.754175 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.754169    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 17:22:17.754389 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.754372    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xxtwh\""
Sep 11 17:22:17.754641 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.754608    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 17:22:17.767556 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.767532    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 17:22:17.771259 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.771234    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 17:22:17.872745 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.872712    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c4004f2-61ce-4890-9c2a-766558008eba-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.872745 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.872754    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5c4004f2-61ce-4890-9c2a-766558008eba-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.873008 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.872790    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c4004f2-61ce-4890-9c2a-766558008eba-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.873008 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.872816    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c4004f2-61ce-4890-9c2a-766558008eba-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.873008 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.872842    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5c4004f2-61ce-4890-9c2a-766558008eba-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.873008 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.872867    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5c4004f2-61ce-4890-9c2a-766558008eba-config\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.873008 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.872913    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c4004f2-61ce-4890-9c2a-766558008eba-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.873008 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.872938    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5c4004f2-61ce-4890-9c2a-766558008eba-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.873008 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.872964    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5c4004f2-61ce-4890-9c2a-766558008eba-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.873008 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.872994    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf96v\" (UniqueName: \"kubernetes.io/projected/5c4004f2-61ce-4890-9c2a-766558008eba-kube-api-access-qf96v\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.873337 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.873078    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5c4004f2-61ce-4890-9c2a-766558008eba-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.873337 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.873148    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5c4004f2-61ce-4890-9c2a-766558008eba-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.873337 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.873183    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5c4004f2-61ce-4890-9c2a-766558008eba-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.873337 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.873206    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c4004f2-61ce-4890-9c2a-766558008eba-web-config\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.873337 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.873232    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5c4004f2-61ce-4890-9c2a-766558008eba-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.873337 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.873258    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c4004f2-61ce-4890-9c2a-766558008eba-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.873337 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.873285    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c4004f2-61ce-4890-9c2a-766558008eba-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.873337 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.873333    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c4004f2-61ce-4890-9c2a-766558008eba-config-out\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.873740 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.873375    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 17:22:17.873740 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.873375    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c4004f2-61ce-4890-9c2a-766558008eba-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.873740 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.873491    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5c4004f2-61ce-4890-9c2a-766558008eba-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.974345 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.974305    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5c4004f2-61ce-4890-9c2a-766558008eba-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.974510 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.974366    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5c4004f2-61ce-4890-9c2a-766558008eba-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.974510 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.974407    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c4004f2-61ce-4890-9c2a-766558008eba-web-config\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.974510 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.974435    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5c4004f2-61ce-4890-9c2a-766558008eba-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.974734 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.974709    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c4004f2-61ce-4890-9c2a-766558008eba-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.974823 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.974754    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c4004f2-61ce-4890-9c2a-766558008eba-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.974823 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.974792    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c4004f2-61ce-4890-9c2a-766558008eba-config-out\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.974823 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.974818    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c4004f2-61ce-4890-9c2a-766558008eba-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.974993 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.974834    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5c4004f2-61ce-4890-9c2a-766558008eba-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.974993 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.974841    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5c4004f2-61ce-4890-9c2a-766558008eba-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.974993 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.974890    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c4004f2-61ce-4890-9c2a-766558008eba-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.974993 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.974940    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5c4004f2-61ce-4890-9c2a-766558008eba-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.974993 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.974978    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c4004f2-61ce-4890-9c2a-766558008eba-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.975220 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.975005    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c4004f2-61ce-4890-9c2a-766558008eba-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.975220 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.975036    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5c4004f2-61ce-4890-9c2a-766558008eba-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.975764 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.975721    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c4004f2-61ce-4890-9c2a-766558008eba-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.975947 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.975779    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c4004f2-61ce-4890-9c2a-766558008eba-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.976354 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.976327    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5c4004f2-61ce-4890-9c2a-766558008eba-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.977073 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.977049    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c4004f2-61ce-4890-9c2a-766558008eba-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.977169 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.977146    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5c4004f2-61ce-4890-9c2a-766558008eba-config\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.977237 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.977180    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c4004f2-61ce-4890-9c2a-766558008eba-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.977237 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.977206    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5c4004f2-61ce-4890-9c2a-766558008eba-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.977347 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.977236    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5c4004f2-61ce-4890-9c2a-766558008eba-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.977347 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.977264    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf96v\" (UniqueName: \"kubernetes.io/projected/5c4004f2-61ce-4890-9c2a-766558008eba-kube-api-access-qf96v\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.977347 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.977308    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5c4004f2-61ce-4890-9c2a-766558008eba-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.977773 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.977712    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c4004f2-61ce-4890-9c2a-766558008eba-config-out\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.978472 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.978084    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5c4004f2-61ce-4890-9c2a-766558008eba-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.978472 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.978292    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5c4004f2-61ce-4890-9c2a-766558008eba-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.978472 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.978384    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c4004f2-61ce-4890-9c2a-766558008eba-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.979953 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.979910    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c4004f2-61ce-4890-9c2a-766558008eba-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.980207 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.980154    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c4004f2-61ce-4890-9c2a-766558008eba-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.980323 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.980211    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5c4004f2-61ce-4890-9c2a-766558008eba-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.980971 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.980947    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5c4004f2-61ce-4890-9c2a-766558008eba-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.981154 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.981110    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5c4004f2-61ce-4890-9c2a-766558008eba-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.981322 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.981300    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c4004f2-61ce-4890-9c2a-766558008eba-web-config\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.982928 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.982858    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5c4004f2-61ce-4890-9c2a-766558008eba-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:17.988946 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:17.988857    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf96v\" (UniqueName: \"kubernetes.io/projected/5c4004f2-61ce-4890-9c2a-766558008eba-kube-api-access-qf96v\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:18.313362 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:18.313252    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7hvgz" event={"ID":"6dadf403-2628-4ab2-83d7-3bf3a3dac462","Type":"ContainerStarted","Data":"d1f1899e40794f87ad6512f9a670ebf5b49e9755bc6024442e1f86003ad3e42e"}
Sep 11 17:22:18.313362 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:18.313322    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7hvgz" event={"ID":"6dadf403-2628-4ab2-83d7-3bf3a3dac462","Type":"ContainerStarted","Data":"3490bc09ee7b207c54428eb9a6b8f7d22055cbaf4854c6b42293fdb309749dc4"}
Sep 11 17:22:18.352282 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:18.352231    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7hvgz" podStartSLOduration=3.371534092 podStartE2EDuration="7.352212878s" totalImagesPullingTime="3.89160465s" totalInitContainerRuntime="89.074136ms" isStatefulPod=true podCreationTimestamp="2026-09-11 17:22:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:18.351519198 +0000 UTC m=+106.155691617" watchObservedRunningTime="2026-09-11 17:22:18.352212878 +0000 UTC m=+106.156385297"
Sep 11 17:22:18.388012 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:18.387978    2560 patch_prober.go:28] interesting pod/image-registry-747d6b5cfd-8p2b2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 17:22:18.388169 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:18.388031    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2" podUID="9abdd1bd-edc8-41c3-91e6-da724784b425" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 17:22:18.607842 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:18.607800    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 17:22:18.620183 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:18.620150    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5c4004f2-61ce-4890-9c2a-766558008eba-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:18.636784 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:18.636748    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 17:22:18.641473 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:18.641431    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c4004f2-61ce-4890-9c2a-766558008eba-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:18.875979 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:18.875948    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 17:22:18.881636 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:18.881616    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5c4004f2-61ce-4890-9c2a-766558008eba-config\") pod \"prometheus-k8s-0\" (UID: \"5c4004f2-61ce-4890-9c2a-766558008eba\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:18.943944 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:18.943917    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:19.244197 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:19.244111    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 17:22:19.247410 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:22:19.247386    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4004f2_61ce_4890_9c2a_766558008eba.slice/crio-e53ee751c33202b06b4ca6eace028155aa75712f3923ab974de1b3a187fe9434 WatchSource:0}: Error finding container e53ee751c33202b06b4ca6eace028155aa75712f3923ab974de1b3a187fe9434: Status 404 returned error can't find the container with id e53ee751c33202b06b4ca6eace028155aa75712f3923ab974de1b3a187fe9434
Sep 11 17:22:19.319326 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:19.319281    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c4004f2-61ce-4890-9c2a-766558008eba","Type":"ContainerStarted","Data":"e53ee751c33202b06b4ca6eace028155aa75712f3923ab974de1b3a187fe9434"}
Sep 11 17:22:19.321794 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:19.321768    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4" event={"ID":"5b2f4bf0-4752-4905-ba63-3879494596bc","Type":"ContainerStarted","Data":"c4c1629d49ef2f85699ace4c7f4957e12384d2243812872bfb5dd930c506b4e4"}
Sep 11 17:22:19.321940 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:19.321829    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4" event={"ID":"5b2f4bf0-4752-4905-ba63-3879494596bc","Type":"ContainerStarted","Data":"102eee908d5f5d3ab154a0dc02b46726c17aba0e09efc04fc2d03fec9b10f887"}
Sep 11 17:22:19.321940 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:19.321857    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4" event={"ID":"5b2f4bf0-4752-4905-ba63-3879494596bc","Type":"ContainerStarted","Data":"757e5e421dad5505615b83018be0f40577ba230135228795c74e48f0a0fae64e"}
Sep 11 17:22:20.328745 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:20.328705    2560 generic.go:356] "Generic (PLEG): container finished" podID="5c4004f2-61ce-4890-9c2a-766558008eba" containerID="cee12ba257841ef2e358a7f1be4eda3564b9905f3a6de235c8bf00ee45504952" exitCode=0
Sep 11 17:22:20.329217 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:20.328779    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c4004f2-61ce-4890-9c2a-766558008eba","Type":"ContainerDied","Data":"cee12ba257841ef2e358a7f1be4eda3564b9905f3a6de235c8bf00ee45504952"}
Sep 11 17:22:20.765802 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:20.765774    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:22:20.765918 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:20.765846    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:22:20.771087 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:20.771068    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:22:21.337292 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:21.337250    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4" event={"ID":"5b2f4bf0-4752-4905-ba63-3879494596bc","Type":"ContainerStarted","Data":"c3d82436e826f5825cb8f3a36ea38645c60152aafc1ad147bbb6c49402099fc4"}
Sep 11 17:22:21.337721 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:21.337332    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4" event={"ID":"5b2f4bf0-4752-4905-ba63-3879494596bc","Type":"ContainerStarted","Data":"82d95d30d3d64864d15d9cdcf90ab81e193b0bbc8cc14c6346a059be75cd18c5"}
Sep 11 17:22:21.337721 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:21.337373    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4" event={"ID":"5b2f4bf0-4752-4905-ba63-3879494596bc","Type":"ContainerStarted","Data":"1da0615b756509ed1336178b9129b901b4ea3aaf362e4d1ed5e14db775d31e1f"}
Sep 11 17:22:21.337721 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:21.337438    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"
Sep 11 17:22:21.341715 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:21.341692    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:22:21.364620 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:21.364568    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4" podStartSLOduration=4.358640283 podStartE2EDuration="8.364548947s" totalImagesPullingTime="4.005908664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:22:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:21.36265843 +0000 UTC m=+109.166830848" watchObservedRunningTime="2026-09-11 17:22:21.364548947 +0000 UTC m=+109.168721365"
Sep 11 17:22:23.403072 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:23.403028    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2" podUID="9abdd1bd-edc8-41c3-91e6-da724784b425" containerName="registry" containerID="cri-o://6ed0d5ebedb978d881b1ac6ea99ca6e5523d1d306d0b848eab86f39a0735a1ed" gracePeriod=30
Sep 11 17:22:24.028644 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.028619    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:22:24.141884 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.139267    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-bound-sa-token\") pod \"9abdd1bd-edc8-41c3-91e6-da724784b425\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") "
Sep 11 17:22:24.141884 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.139346    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fksz\" (UniqueName: \"kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-kube-api-access-8fksz\") pod \"9abdd1bd-edc8-41c3-91e6-da724784b425\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") "
Sep 11 17:22:24.141884 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.139379    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9abdd1bd-edc8-41c3-91e6-da724784b425-ca-trust-extracted\") pod \"9abdd1bd-edc8-41c3-91e6-da724784b425\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") "
Sep 11 17:22:24.141884 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.139499    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9abdd1bd-edc8-41c3-91e6-da724784b425-registry-certificates\") pod \"9abdd1bd-edc8-41c3-91e6-da724784b425\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") "
Sep 11 17:22:24.141884 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.139548    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9abdd1bd-edc8-41c3-91e6-da724784b425-installation-pull-secrets\") pod \"9abdd1bd-edc8-41c3-91e6-da724784b425\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") "
Sep 11 17:22:24.141884 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.139576    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9abdd1bd-edc8-41c3-91e6-da724784b425-trusted-ca\") pod \"9abdd1bd-edc8-41c3-91e6-da724784b425\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") "
Sep 11 17:22:24.141884 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.139625    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-registry-tls\") pod \"9abdd1bd-edc8-41c3-91e6-da724784b425\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") "
Sep 11 17:22:24.141884 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.139662    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9abdd1bd-edc8-41c3-91e6-da724784b425-image-registry-private-configuration\") pod \"9abdd1bd-edc8-41c3-91e6-da724784b425\" (UID: \"9abdd1bd-edc8-41c3-91e6-da724784b425\") "
Sep 11 17:22:24.141884 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.140203    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9abdd1bd-edc8-41c3-91e6-da724784b425-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "9abdd1bd-edc8-41c3-91e6-da724784b425" (UID: "9abdd1bd-edc8-41c3-91e6-da724784b425"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:22:24.141884 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.140426    2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9abdd1bd-edc8-41c3-91e6-da724784b425-registry-certificates\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:22:24.141884 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.140489    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9abdd1bd-edc8-41c3-91e6-da724784b425-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "9abdd1bd-edc8-41c3-91e6-da724784b425" (UID: "9abdd1bd-edc8-41c3-91e6-da724784b425"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:22:24.144709 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.144617    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9abdd1bd-edc8-41c3-91e6-da724784b425-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "9abdd1bd-edc8-41c3-91e6-da724784b425" (UID: "9abdd1bd-edc8-41c3-91e6-da724784b425"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:22:24.145979 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.145106    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9abdd1bd-edc8-41c3-91e6-da724784b425" (UID: "9abdd1bd-edc8-41c3-91e6-da724784b425"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:22:24.148471 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.148239    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9abdd1bd-edc8-41c3-91e6-da724784b425-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "9abdd1bd-edc8-41c3-91e6-da724784b425" (UID: "9abdd1bd-edc8-41c3-91e6-da724784b425"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:22:24.149944 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.149389    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "9abdd1bd-edc8-41c3-91e6-da724784b425" (UID: "9abdd1bd-edc8-41c3-91e6-da724784b425"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:22:24.149944 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.149561    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-kube-api-access-8fksz" (OuterVolumeSpecName: "kube-api-access-8fksz") pod "9abdd1bd-edc8-41c3-91e6-da724784b425" (UID: "9abdd1bd-edc8-41c3-91e6-da724784b425"). InnerVolumeSpecName "kube-api-access-8fksz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:22:24.151829 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.151792    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abdd1bd-edc8-41c3-91e6-da724784b425-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "9abdd1bd-edc8-41c3-91e6-da724784b425" (UID: "9abdd1bd-edc8-41c3-91e6-da724784b425"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:22:24.241196 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.241170    2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-bound-sa-token\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:22:24.241289 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.241202    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fksz\" (UniqueName: \"kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-kube-api-access-8fksz\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:22:24.241289 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.241218    2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9abdd1bd-edc8-41c3-91e6-da724784b425-ca-trust-extracted\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:22:24.241289 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.241231    2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9abdd1bd-edc8-41c3-91e6-da724784b425-installation-pull-secrets\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:22:24.241289 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.241247    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9abdd1bd-edc8-41c3-91e6-da724784b425-trusted-ca\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:22:24.241289 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.241259    2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9abdd1bd-edc8-41c3-91e6-da724784b425-registry-tls\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:22:24.241289 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.241270    2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9abdd1bd-edc8-41c3-91e6-da724784b425-image-registry-private-configuration\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:22:24.354327 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.354289    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c4004f2-61ce-4890-9c2a-766558008eba","Type":"ContainerStarted","Data":"7f633e41fdd048761fd96dd52f982fd203972294ec1657f54d07666fcc48204b"}
Sep 11 17:22:24.354469 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.354377    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c4004f2-61ce-4890-9c2a-766558008eba","Type":"ContainerStarted","Data":"995f325af33b3f37e4c000b9546742992b524e7f573e8caa6eb0d5dc5d595eca"}
Sep 11 17:22:24.354469 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.354421    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c4004f2-61ce-4890-9c2a-766558008eba","Type":"ContainerStarted","Data":"30a575f935189b737e1624af4ca40470f6a8e0c1d24808c227fa4546c59d5a74"}
Sep 11 17:22:24.354469 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.354463    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c4004f2-61ce-4890-9c2a-766558008eba","Type":"ContainerStarted","Data":"31f156a4f5eb9527c7767fc3d60a17b6c781de566c5e9d913d50c1b3ef2dcd1b"}
Sep 11 17:22:24.354613 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.354503    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c4004f2-61ce-4890-9c2a-766558008eba","Type":"ContainerStarted","Data":"c4e13943b75aab27bb4b9d8a9c7211f05f179ceb38c162623ea6a160424de62f"}
Sep 11 17:22:24.354613 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.354541    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c4004f2-61ce-4890-9c2a-766558008eba","Type":"ContainerStarted","Data":"4ccaa8b052e8e3d72768b3569385bd1482c275769b55b07a299ee6b3b3eae78b"}
Sep 11 17:22:24.355596 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.355568    2560 generic.go:356] "Generic (PLEG): container finished" podID="9abdd1bd-edc8-41c3-91e6-da724784b425" containerID="6ed0d5ebedb978d881b1ac6ea99ca6e5523d1d306d0b848eab86f39a0735a1ed" exitCode=0
Sep 11 17:22:24.355695 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.355603    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2"
Sep 11 17:22:24.355695 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.355627    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2" event={"ID":"9abdd1bd-edc8-41c3-91e6-da724784b425","Type":"ContainerDied","Data":"6ed0d5ebedb978d881b1ac6ea99ca6e5523d1d306d0b848eab86f39a0735a1ed"}
Sep 11 17:22:24.355695 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.355666    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-747d6b5cfd-8p2b2" event={"ID":"9abdd1bd-edc8-41c3-91e6-da724784b425","Type":"ContainerDied","Data":"882d26b53686c9ca135cbdaafc65fc5b641aca6b281f78a8b9bf246fba4f278b"}
Sep 11 17:22:24.355695 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.355692    2560 scope.go:117] "RemoveContainer" containerID="6ed0d5ebedb978d881b1ac6ea99ca6e5523d1d306d0b848eab86f39a0735a1ed"
Sep 11 17:22:24.365023 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.364747    2560 scope.go:117] "RemoveContainer" containerID="6ed0d5ebedb978d881b1ac6ea99ca6e5523d1d306d0b848eab86f39a0735a1ed"
Sep 11 17:22:24.365259 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:22:24.365214    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ed0d5ebedb978d881b1ac6ea99ca6e5523d1d306d0b848eab86f39a0735a1ed\": container with ID starting with 6ed0d5ebedb978d881b1ac6ea99ca6e5523d1d306d0b848eab86f39a0735a1ed not found: ID does not exist" containerID="6ed0d5ebedb978d881b1ac6ea99ca6e5523d1d306d0b848eab86f39a0735a1ed"
Sep 11 17:22:24.365327 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.365248    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ed0d5ebedb978d881b1ac6ea99ca6e5523d1d306d0b848eab86f39a0735a1ed"} err="failed to get container status \"6ed0d5ebedb978d881b1ac6ea99ca6e5523d1d306d0b848eab86f39a0735a1ed\": rpc error: code = NotFound desc = could not find container \"6ed0d5ebedb978d881b1ac6ea99ca6e5523d1d306d0b848eab86f39a0735a1ed\": container with ID starting with 6ed0d5ebedb978d881b1ac6ea99ca6e5523d1d306d0b848eab86f39a0735a1ed not found: ID does not exist"
Sep 11 17:22:24.387828 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.387777    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.863233201 podStartE2EDuration="7.387758457s" totalImagesPullingTime="4.506952882s" totalInitContainerRuntime="17.572374ms" isStatefulPod=false podCreationTimestamp="2026-09-11 17:22:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:24.385724969 +0000 UTC m=+112.189897429" watchObservedRunningTime="2026-09-11 17:22:24.387758457 +0000 UTC m=+112.191930876"
Sep 11 17:22:24.399344 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.399274    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-747d6b5cfd-8p2b2"]
Sep 11 17:22:24.406761 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.406735    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-747d6b5cfd-8p2b2"]
Sep 11 17:22:24.715058 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:24.714981    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9abdd1bd-edc8-41c3-91e6-da724784b425" path="/var/lib/kubelet/pods/9abdd1bd-edc8-41c3-91e6-da724784b425/volumes"
Sep 11 17:22:27.315516 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:27.315482    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mp4vh"
Sep 11 17:22:27.351064 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:27.351039    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7f8d4ccfc-7n5f4"
Sep 11 17:22:28.944422 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:28.944385    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:22:36.489404 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:36.489365    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5578774b47-z7pkw"]
Sep 11 17:22:42.404099 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:42.404056    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1203d22c-52e3-4681-9707-66ca35b9e72a-metrics-certs\") pod \"network-metrics-daemon-sn2g7\" (UID: \"1203d22c-52e3-4681-9707-66ca35b9e72a\") " pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:22:42.406355 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:42.406334    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1203d22c-52e3-4681-9707-66ca35b9e72a-metrics-certs\") pod \"network-metrics-daemon-sn2g7\" (UID: \"1203d22c-52e3-4681-9707-66ca35b9e72a\") " pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:22:42.521693 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:42.521662    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f9zbq\""
Sep 11 17:22:42.529891 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:42.529872    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sn2g7"
Sep 11 17:22:42.803523 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:42.803472    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sn2g7"]
Sep 11 17:22:42.805971 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:22:42.805942    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1203d22c_52e3_4681_9707_66ca35b9e72a.slice/crio-58c345d5294ceca62760346796fa237b9c0d39a15d061e084440efe1ee143845 WatchSource:0}: Error finding container 58c345d5294ceca62760346796fa237b9c0d39a15d061e084440efe1ee143845: Status 404 returned error can't find the container with id 58c345d5294ceca62760346796fa237b9c0d39a15d061e084440efe1ee143845
Sep 11 17:22:43.435875 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:43.435836    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sn2g7" event={"ID":"1203d22c-52e3-4681-9707-66ca35b9e72a","Type":"ContainerStarted","Data":"58c345d5294ceca62760346796fa237b9c0d39a15d061e084440efe1ee143845"}
Sep 11 17:22:45.448576 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:45.448484    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sn2g7" event={"ID":"1203d22c-52e3-4681-9707-66ca35b9e72a","Type":"ContainerStarted","Data":"73a49ec0d66961e702f867aa0fb85dfa5a7e6e4f1c9e1cffe000b0db03ed7537"}
Sep 11 17:22:45.448576 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:45.448553    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sn2g7" event={"ID":"1203d22c-52e3-4681-9707-66ca35b9e72a","Type":"ContainerStarted","Data":"e1e499dfce5b41046a79908a84dd7569a8f042df986f65ee1862b4cb576e77d0"}
Sep 11 17:22:52.477160 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:52.477126    2560 generic.go:356] "Generic (PLEG): container finished" podID="c1581956-5e0a-4ed1-9258-985cf6ca7373" containerID="d937c50aa8e47a3aba1d901ebd9115cc3bbf379a91c56bb8a2f77536590ebbfc" exitCode=0
Sep 11 17:22:52.477559 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:52.477201    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-25988" event={"ID":"c1581956-5e0a-4ed1-9258-985cf6ca7373","Type":"ContainerDied","Data":"d937c50aa8e47a3aba1d901ebd9115cc3bbf379a91c56bb8a2f77536590ebbfc"}
Sep 11 17:22:52.477559 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:52.477525    2560 scope.go:117] "RemoveContainer" containerID="d937c50aa8e47a3aba1d901ebd9115cc3bbf379a91c56bb8a2f77536590ebbfc"
Sep 11 17:22:52.494664 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:52.494619    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sn2g7" podStartSLOduration=137.489241039 podStartE2EDuration="2m19.494604613s" totalImagesPullingTime="2.005363574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:45.467863634 +0000 UTC m=+133.272036054" watchObservedRunningTime="2026-09-11 17:22:52.494604613 +0000 UTC m=+140.298777169"
Sep 11 17:22:53.482673 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:53.482634    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-25988" event={"ID":"c1581956-5e0a-4ed1-9258-985cf6ca7373","Type":"ContainerStarted","Data":"389b1da2ae4f7627d28b8423e46e6682ca25730ae0b77c5af7a19765a65d7cb6"}
Sep 11 17:22:57.500365 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:57.500333    2560 generic.go:356] "Generic (PLEG): container finished" podID="b996965e-b19e-4959-b58c-a3cb4039d465" containerID="3e0623ae5d050c76aecc3b838973ea57ecd7b527a7a7f080dc6e4983ab28343c" exitCode=0
Sep 11 17:22:57.500744 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:57.500413    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jd6vw" event={"ID":"b996965e-b19e-4959-b58c-a3cb4039d465","Type":"ContainerDied","Data":"3e0623ae5d050c76aecc3b838973ea57ecd7b527a7a7f080dc6e4983ab28343c"}
Sep 11 17:22:57.500789 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:57.500752    2560 scope.go:117] "RemoveContainer" containerID="3e0623ae5d050c76aecc3b838973ea57ecd7b527a7a7f080dc6e4983ab28343c"
Sep 11 17:22:58.506525 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:22:58.506486    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jd6vw" event={"ID":"b996965e-b19e-4959-b58c-a3cb4039d465","Type":"ContainerStarted","Data":"f312fd8efeda4143d56bea6f78c6dbec078352d00e28244d815316d04861ee98"}
Sep 11 17:23:01.508213 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:01.508159    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5578774b47-z7pkw" podUID="11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5" containerName="console" containerID="cri-o://1275d3317d30ea4f97718806c5bd0c23fb03c9479df48b1c62975563cd0d92a8" gracePeriod=15
Sep 11 17:23:01.755270 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:01.755245    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:23:01.879839 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:01.879795    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-console-oauth-config\") pod \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\" (UID: \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\") "
Sep 11 17:23:01.880068 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:01.879857    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-trusted-ca-bundle\") pod \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\" (UID: \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\") "
Sep 11 17:23:01.880068 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:01.879889    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-service-ca\") pod \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\" (UID: \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\") "
Sep 11 17:23:01.880068 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:01.879939    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-console-config\") pod \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\" (UID: \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\") "
Sep 11 17:23:01.880068 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:01.880053    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-oauth-serving-cert\") pod \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\" (UID: \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\") "
Sep 11 17:23:01.880286 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:01.880139    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-console-serving-cert\") pod \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\" (UID: \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\") "
Sep 11 17:23:01.880286 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:01.880181    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkb7x\" (UniqueName: \"kubernetes.io/projected/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-kube-api-access-dkb7x\") pod \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\" (UID: \"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5\") "
Sep 11 17:23:01.880404 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:01.880347    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5" (UID: "11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:23:01.880404 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:01.880375    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-console-config" (OuterVolumeSpecName: "console-config") pod "11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5" (UID: "11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:23:01.880493 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:01.880412    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-service-ca" (OuterVolumeSpecName: "service-ca") pod "11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5" (UID: "11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:23:01.880530 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:01.880486    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5" (UID: "11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:23:01.882303 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:01.882269    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5" (UID: "11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:23:01.882405 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:01.882317    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-kube-api-access-dkb7x" (OuterVolumeSpecName: "kube-api-access-dkb7x") pod "11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5" (UID: "11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5"). InnerVolumeSpecName "kube-api-access-dkb7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:23:01.882405 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:01.882317    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5" (UID: "11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:23:01.981669 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:01.981630    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkb7x\" (UniqueName: \"kubernetes.io/projected/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-kube-api-access-dkb7x\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:23:01.981669 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:01.981661    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-console-oauth-config\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:23:01.981669 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:01.981670    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-trusted-ca-bundle\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:23:01.981669 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:01.981679    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-service-ca\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:23:01.981949 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:01.981688    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-console-config\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:23:01.981949 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:01.981696    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-oauth-serving-cert\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:23:01.981949 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:01.981704    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5-console-serving-cert\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:23:02.524290 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:02.524255    2560 generic.go:356] "Generic (PLEG): container finished" podID="11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5" containerID="1275d3317d30ea4f97718806c5bd0c23fb03c9479df48b1c62975563cd0d92a8" exitCode=0
Sep 11 17:23:02.524701 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:02.524310    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5578774b47-z7pkw"
Sep 11 17:23:02.524701 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:02.524352    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5578774b47-z7pkw" event={"ID":"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5","Type":"ContainerDied","Data":"1275d3317d30ea4f97718806c5bd0c23fb03c9479df48b1c62975563cd0d92a8"}
Sep 11 17:23:02.524701 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:02.524415    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5578774b47-z7pkw" event={"ID":"11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5","Type":"ContainerDied","Data":"44724f980a1d8fcd3d2fe4bf4c0ebe47897e3436b8bfb893b31b62458d7c145b"}
Sep 11 17:23:02.524701 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:02.524437    2560 scope.go:117] "RemoveContainer" containerID="1275d3317d30ea4f97718806c5bd0c23fb03c9479df48b1c62975563cd0d92a8"
Sep 11 17:23:02.533446 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:02.533425    2560 scope.go:117] "RemoveContainer" containerID="1275d3317d30ea4f97718806c5bd0c23fb03c9479df48b1c62975563cd0d92a8"
Sep 11 17:23:02.533762 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:23:02.533739    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1275d3317d30ea4f97718806c5bd0c23fb03c9479df48b1c62975563cd0d92a8\": container with ID starting with 1275d3317d30ea4f97718806c5bd0c23fb03c9479df48b1c62975563cd0d92a8 not found: ID does not exist" containerID="1275d3317d30ea4f97718806c5bd0c23fb03c9479df48b1c62975563cd0d92a8"
Sep 11 17:23:02.533869 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:02.533767    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1275d3317d30ea4f97718806c5bd0c23fb03c9479df48b1c62975563cd0d92a8"} err="failed to get container status \"1275d3317d30ea4f97718806c5bd0c23fb03c9479df48b1c62975563cd0d92a8\": rpc error: code = NotFound desc = could not find container \"1275d3317d30ea4f97718806c5bd0c23fb03c9479df48b1c62975563cd0d92a8\": container with ID starting with 1275d3317d30ea4f97718806c5bd0c23fb03c9479df48b1c62975563cd0d92a8 not found: ID does not exist"
Sep 11 17:23:02.550070 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:02.550037    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5578774b47-z7pkw"]
Sep 11 17:23:02.554807 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:02.554784    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5578774b47-z7pkw"]
Sep 11 17:23:02.714051 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:02.714020    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5" path="/var/lib/kubelet/pods/11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5/volumes"
Sep 11 17:23:18.944329 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:18.944291    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:23:18.963725 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:18.963697    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:23:19.607602 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:23:19.607576    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 17:25:09.201914 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:09.201810    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lbdgf"]
Sep 11 17:25:09.202357 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:09.202127    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9abdd1bd-edc8-41c3-91e6-da724784b425" containerName="registry"
Sep 11 17:25:09.202357 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:09.202137    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9abdd1bd-edc8-41c3-91e6-da724784b425" containerName="registry"
Sep 11 17:25:09.202357 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:09.202149    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5" containerName="console"
Sep 11 17:25:09.202357 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:09.202154    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5" containerName="console"
Sep 11 17:25:09.202357 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:09.202211    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="11d64f96-b6f7-41f3-8dde-c8dd2fb2f4e5" containerName="console"
Sep 11 17:25:09.202357 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:09.202220    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9abdd1bd-edc8-41c3-91e6-da724784b425" containerName="registry"
Sep 11 17:25:09.205080 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:09.205063    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lbdgf"
Sep 11 17:25:09.207605 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:09.207581    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:25:09.208691 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:09.208673    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 17:25:09.208792 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:09.208745    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-b9lfn\""
Sep 11 17:25:09.214403 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:09.214377    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lbdgf"]
Sep 11 17:25:09.224415 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:09.224393    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39e8377d-780d-400a-bf5b-b73ec7e3a97d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lbdgf\" (UID: \"39e8377d-780d-400a-bf5b-b73ec7e3a97d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lbdgf"
Sep 11 17:25:09.224517 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:09.224430    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v624n\" (UniqueName: \"kubernetes.io/projected/39e8377d-780d-400a-bf5b-b73ec7e3a97d-kube-api-access-v624n\") pod \"openshift-lws-operator-fd8ccff4c-lbdgf\" (UID: \"39e8377d-780d-400a-bf5b-b73ec7e3a97d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lbdgf"
Sep 11 17:25:09.325448 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:09.325399    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39e8377d-780d-400a-bf5b-b73ec7e3a97d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lbdgf\" (UID: \"39e8377d-780d-400a-bf5b-b73ec7e3a97d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lbdgf"
Sep 11 17:25:09.325448 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:09.325447    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v624n\" (UniqueName: \"kubernetes.io/projected/39e8377d-780d-400a-bf5b-b73ec7e3a97d-kube-api-access-v624n\") pod \"openshift-lws-operator-fd8ccff4c-lbdgf\" (UID: \"39e8377d-780d-400a-bf5b-b73ec7e3a97d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lbdgf"
Sep 11 17:25:09.325805 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:09.325786    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39e8377d-780d-400a-bf5b-b73ec7e3a97d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lbdgf\" (UID: \"39e8377d-780d-400a-bf5b-b73ec7e3a97d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lbdgf"
Sep 11 17:25:09.334557 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:09.334528    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v624n\" (UniqueName: \"kubernetes.io/projected/39e8377d-780d-400a-bf5b-b73ec7e3a97d-kube-api-access-v624n\") pod \"openshift-lws-operator-fd8ccff4c-lbdgf\" (UID: \"39e8377d-780d-400a-bf5b-b73ec7e3a97d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lbdgf"
Sep 11 17:25:09.524917 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:09.524813    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lbdgf"
Sep 11 17:25:09.802012 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:09.801949    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lbdgf"]
Sep 11 17:25:09.804628 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:25:09.804600    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39e8377d_780d_400a_bf5b_b73ec7e3a97d.slice/crio-23693aa7e2e2ba10939944b85c353769b2957345c7ef22973d1431a4ec916da7 WatchSource:0}: Error finding container 23693aa7e2e2ba10939944b85c353769b2957345c7ef22973d1431a4ec916da7: Status 404 returned error can't find the container with id 23693aa7e2e2ba10939944b85c353769b2957345c7ef22973d1431a4ec916da7
Sep 11 17:25:10.004191 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:10.004156    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lbdgf" event={"ID":"39e8377d-780d-400a-bf5b-b73ec7e3a97d","Type":"ContainerStarted","Data":"23693aa7e2e2ba10939944b85c353769b2957345c7ef22973d1431a4ec916da7"}
Sep 11 17:25:13.023990 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:13.019454    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lbdgf" event={"ID":"39e8377d-780d-400a-bf5b-b73ec7e3a97d","Type":"ContainerStarted","Data":"0b7b1568fff893e756a36590e2da723c086410017b968f2b19c1a7ad1559e19b"}
Sep 11 17:25:13.040267 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:13.040214    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lbdgf" podStartSLOduration=1.834647557 podStartE2EDuration="4.040198185s" totalImagesPullingTime="2.205550628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:25:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:25:13.039675011 +0000 UTC m=+280.843847429" watchObservedRunningTime="2026-09-11 17:25:13.040198185 +0000 UTC m=+280.844370603"
Sep 11 17:25:28.495506 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:28.495474    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-2xvmq"]
Sep 11 17:25:28.498854 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:28.498834    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-2xvmq"
Sep 11 17:25:28.501738 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:28.501708    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 17:25:28.501854 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:28.501826    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 17:25:28.501940 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:28.501876    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 17:25:28.502010 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:28.501967    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hw64m\""
Sep 11 17:25:28.502226 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:28.502211    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 17:25:28.532872 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:28.532842    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-2xvmq"]
Sep 11 17:25:28.587102 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:28.587071    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d178cc4-c757-4046-ad96-82495b5f9530-tmp\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-2xvmq\" (UID: \"0d178cc4-c757-4046-ad96-82495b5f9530\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-2xvmq"
Sep 11 17:25:28.587270 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:28.587121    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxpkr\" (UniqueName: \"kubernetes.io/projected/0d178cc4-c757-4046-ad96-82495b5f9530-kube-api-access-xxpkr\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-2xvmq\" (UID: \"0d178cc4-c757-4046-ad96-82495b5f9530\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-2xvmq"
Sep 11 17:25:28.587270 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:28.587144    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d178cc4-c757-4046-ad96-82495b5f9530-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-2xvmq\" (UID: \"0d178cc4-c757-4046-ad96-82495b5f9530\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-2xvmq"
Sep 11 17:25:28.587270 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:28.587189    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0d178cc4-c757-4046-ad96-82495b5f9530-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-2xvmq\" (UID: \"0d178cc4-c757-4046-ad96-82495b5f9530\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-2xvmq"
Sep 11 17:25:28.587270 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:28.587217    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0d178cc4-c757-4046-ad96-82495b5f9530-manifests\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-2xvmq\" (UID: \"0d178cc4-c757-4046-ad96-82495b5f9530\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-2xvmq"
Sep 11 17:25:28.688064 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:28.688023    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d178cc4-c757-4046-ad96-82495b5f9530-tmp\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-2xvmq\" (UID: \"0d178cc4-c757-4046-ad96-82495b5f9530\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-2xvmq"
Sep 11 17:25:28.688064 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:28.688070    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxpkr\" (UniqueName: \"kubernetes.io/projected/0d178cc4-c757-4046-ad96-82495b5f9530-kube-api-access-xxpkr\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-2xvmq\" (UID: \"0d178cc4-c757-4046-ad96-82495b5f9530\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-2xvmq"
Sep 11 17:25:28.688262 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:28.688177    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d178cc4-c757-4046-ad96-82495b5f9530-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-2xvmq\" (UID: \"0d178cc4-c757-4046-ad96-82495b5f9530\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-2xvmq"
Sep 11 17:25:28.688262 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:28.688212    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0d178cc4-c757-4046-ad96-82495b5f9530-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-2xvmq\" (UID: \"0d178cc4-c757-4046-ad96-82495b5f9530\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-2xvmq"
Sep 11 17:25:28.688262 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:28.688248    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0d178cc4-c757-4046-ad96-82495b5f9530-manifests\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-2xvmq\" (UID: \"0d178cc4-c757-4046-ad96-82495b5f9530\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-2xvmq"
Sep 11 17:25:28.688642 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:28.688618    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0d178cc4-c757-4046-ad96-82495b5f9530-manifests\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-2xvmq\" (UID: \"0d178cc4-c757-4046-ad96-82495b5f9530\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-2xvmq"
Sep 11 17:25:28.690377 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:28.690353    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d178cc4-c757-4046-ad96-82495b5f9530-tmp\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-2xvmq\" (UID: \"0d178cc4-c757-4046-ad96-82495b5f9530\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-2xvmq"
Sep 11 17:25:28.690716 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:28.690694    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d178cc4-c757-4046-ad96-82495b5f9530-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-2xvmq\" (UID: \"0d178cc4-c757-4046-ad96-82495b5f9530\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-2xvmq"
Sep 11 17:25:28.690757 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:28.690694    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0d178cc4-c757-4046-ad96-82495b5f9530-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-2xvmq\" (UID: \"0d178cc4-c757-4046-ad96-82495b5f9530\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-2xvmq"
Sep 11 17:25:28.699547 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:28.699522    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxpkr\" (UniqueName: \"kubernetes.io/projected/0d178cc4-c757-4046-ad96-82495b5f9530-kube-api-access-xxpkr\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-2xvmq\" (UID: \"0d178cc4-c757-4046-ad96-82495b5f9530\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-2xvmq"
Sep 11 17:25:28.808742 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:28.808649    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-2xvmq"
Sep 11 17:25:29.101581 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:29.101501    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-2xvmq"]
Sep 11 17:25:29.103671 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:25:29.103642    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d178cc4_c757_4046_ad96_82495b5f9530.slice/crio-5680fb5747530edd2b63e73e12067f4726e308916a4a3621bab98786014a40ae WatchSource:0}: Error finding container 5680fb5747530edd2b63e73e12067f4726e308916a4a3621bab98786014a40ae: Status 404 returned error can't find the container with id 5680fb5747530edd2b63e73e12067f4726e308916a4a3621bab98786014a40ae
Sep 11 17:25:30.089308 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:30.089272    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-2xvmq" event={"ID":"0d178cc4-c757-4046-ad96-82495b5f9530","Type":"ContainerStarted","Data":"5680fb5747530edd2b63e73e12067f4726e308916a4a3621bab98786014a40ae"}
Sep 11 17:25:32.550544 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:32.550465    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6fcjs_477cac01-ab9e-4441-b07d-b7e53401c616/console-operator/1.log"
Sep 11 17:25:32.554037 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:32.554003    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6fcjs_477cac01-ab9e-4441-b07d-b7e53401c616/console-operator/1.log"
Sep 11 17:25:32.570267 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:32.570240    2560 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 17:25:33.105655 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:33.105614    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-2xvmq" event={"ID":"0d178cc4-c757-4046-ad96-82495b5f9530","Type":"ContainerStarted","Data":"fcb1dc4614f2cc0409914c5775e2f96bb3e231bb9967aa31f881301322d19b80"}
Sep 11 17:25:34.112049 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:34.112011    2560 generic.go:356] "Generic (PLEG): container finished" podID="0d178cc4-c757-4046-ad96-82495b5f9530" containerID="fcb1dc4614f2cc0409914c5775e2f96bb3e231bb9967aa31f881301322d19b80" exitCode=0
Sep 11 17:25:34.112444 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:34.112097    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-2xvmq" event={"ID":"0d178cc4-c757-4046-ad96-82495b5f9530","Type":"ContainerDied","Data":"fcb1dc4614f2cc0409914c5775e2f96bb3e231bb9967aa31f881301322d19b80"}
Sep 11 17:25:34.112567 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:34.112549    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:25:35.118230 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:35.118189    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-2xvmq" event={"ID":"0d178cc4-c757-4046-ad96-82495b5f9530","Type":"ContainerStarted","Data":"9dec6ac6d2b8eaff6bbbc9c8c4414177f914054fe36d2581dc7fc1c972a67eef"}
Sep 11 17:25:35.118699 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:35.118325    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-2xvmq"
Sep 11 17:25:35.152239 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:35.152186    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-2xvmq" podStartSLOduration=2.4901530579999998 podStartE2EDuration="7.152169991s" totalImagesPullingTime="3.559222361s" totalInitContainerRuntime="1.102794572s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:25:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:25:35.149577506 +0000 UTC m=+302.953749925" watchObservedRunningTime="2026-09-11 17:25:35.152169991 +0000 UTC m=+302.956342410"
Sep 11 17:25:46.125666 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:25:46.125631    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-2xvmq"
Sep 11 17:26:08.442702 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.442661    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5786cf49b6-qzhhh"]
Sep 11 17:26:08.445104 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.445084    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-qzhhh"
Sep 11 17:26:08.448577 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.448550    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-v94wv\""
Sep 11 17:26:08.448711 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.448563    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 17:26:08.448711 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.448594    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 17:26:08.448711 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.448563    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 17:26:08.457043 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.457020    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5786cf49b6-qzhhh"]
Sep 11 17:26:08.530531 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.530490    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/28e028d7-1f3a-4335-91ad-835ee47f0635-metrics-cert\") pod \"lws-controller-manager-5786cf49b6-qzhhh\" (UID: \"28e028d7-1f3a-4335-91ad-835ee47f0635\") " pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-qzhhh"
Sep 11 17:26:08.530706 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.530558    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/28e028d7-1f3a-4335-91ad-835ee47f0635-manager-config\") pod \"lws-controller-manager-5786cf49b6-qzhhh\" (UID: \"28e028d7-1f3a-4335-91ad-835ee47f0635\") " pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-qzhhh"
Sep 11 17:26:08.530706 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.530635    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28e028d7-1f3a-4335-91ad-835ee47f0635-cert\") pod \"lws-controller-manager-5786cf49b6-qzhhh\" (UID: \"28e028d7-1f3a-4335-91ad-835ee47f0635\") " pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-qzhhh"
Sep 11 17:26:08.530706 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.530674    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddxcz\" (UniqueName: \"kubernetes.io/projected/28e028d7-1f3a-4335-91ad-835ee47f0635-kube-api-access-ddxcz\") pod \"lws-controller-manager-5786cf49b6-qzhhh\" (UID: \"28e028d7-1f3a-4335-91ad-835ee47f0635\") " pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-qzhhh"
Sep 11 17:26:08.632047 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.632007    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28e028d7-1f3a-4335-91ad-835ee47f0635-cert\") pod \"lws-controller-manager-5786cf49b6-qzhhh\" (UID: \"28e028d7-1f3a-4335-91ad-835ee47f0635\") " pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-qzhhh"
Sep 11 17:26:08.632047 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.632049    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddxcz\" (UniqueName: \"kubernetes.io/projected/28e028d7-1f3a-4335-91ad-835ee47f0635-kube-api-access-ddxcz\") pod \"lws-controller-manager-5786cf49b6-qzhhh\" (UID: \"28e028d7-1f3a-4335-91ad-835ee47f0635\") " pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-qzhhh"
Sep 11 17:26:08.632286 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.632092    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/28e028d7-1f3a-4335-91ad-835ee47f0635-metrics-cert\") pod \"lws-controller-manager-5786cf49b6-qzhhh\" (UID: \"28e028d7-1f3a-4335-91ad-835ee47f0635\") " pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-qzhhh"
Sep 11 17:26:08.632286 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.632129    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/28e028d7-1f3a-4335-91ad-835ee47f0635-manager-config\") pod \"lws-controller-manager-5786cf49b6-qzhhh\" (UID: \"28e028d7-1f3a-4335-91ad-835ee47f0635\") " pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-qzhhh"
Sep 11 17:26:08.632838 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.632812    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/28e028d7-1f3a-4335-91ad-835ee47f0635-manager-config\") pod \"lws-controller-manager-5786cf49b6-qzhhh\" (UID: \"28e028d7-1f3a-4335-91ad-835ee47f0635\") " pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-qzhhh"
Sep 11 17:26:08.634566 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.634545    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/28e028d7-1f3a-4335-91ad-835ee47f0635-metrics-cert\") pod \"lws-controller-manager-5786cf49b6-qzhhh\" (UID: \"28e028d7-1f3a-4335-91ad-835ee47f0635\") " pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-qzhhh"
Sep 11 17:26:08.634643 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.634612    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28e028d7-1f3a-4335-91ad-835ee47f0635-cert\") pod \"lws-controller-manager-5786cf49b6-qzhhh\" (UID: \"28e028d7-1f3a-4335-91ad-835ee47f0635\") " pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-qzhhh"
Sep 11 17:26:08.646268 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.646245    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddxcz\" (UniqueName: \"kubernetes.io/projected/28e028d7-1f3a-4335-91ad-835ee47f0635-kube-api-access-ddxcz\") pod \"lws-controller-manager-5786cf49b6-qzhhh\" (UID: \"28e028d7-1f3a-4335-91ad-835ee47f0635\") " pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-qzhhh"
Sep 11 17:26:08.754363 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.754288    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-qzhhh"
Sep 11 17:26:08.840100 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.840067    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns"]
Sep 11 17:26:08.842445 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.842428    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns"
Sep 11 17:26:08.845786 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.845763    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 11 17:26:08.845932 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.845802    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 17:26:08.845932 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.845883    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 11 17:26:08.846065 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.845998    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-5b6fj\""
Sep 11 17:26:08.846065 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.846029    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 17:26:08.846530 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.846513    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 17:26:08.846631 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.846516    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 11 17:26:08.867945 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.867915    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns"]
Sep 11 17:26:08.935327 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.935290    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81e4c195-c79e-4719-b867-cb1cdc7f5d40-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-xfdns\" (UID: \"81e4c195-c79e-4719-b867-cb1cdc7f5d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns"
Sep 11 17:26:08.935500 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.935337    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81e4c195-c79e-4719-b867-cb1cdc7f5d40-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-xfdns\" (UID: \"81e4c195-c79e-4719-b867-cb1cdc7f5d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns"
Sep 11 17:26:08.935500 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.935443    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81e4c195-c79e-4719-b867-cb1cdc7f5d40-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-xfdns\" (UID: \"81e4c195-c79e-4719-b867-cb1cdc7f5d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns"
Sep 11 17:26:08.935500 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.935481    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c66cz\" (UniqueName: \"kubernetes.io/projected/81e4c195-c79e-4719-b867-cb1cdc7f5d40-kube-api-access-c66cz\") pod \"istiod-openshift-gateway-5677ccb49b-xfdns\" (UID: \"81e4c195-c79e-4719-b867-cb1cdc7f5d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns"
Sep 11 17:26:08.935500 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.935498    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e4c195-c79e-4719-b867-cb1cdc7f5d40-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-xfdns\" (UID: \"81e4c195-c79e-4719-b867-cb1cdc7f5d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns"
Sep 11 17:26:08.935640 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.935552    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81e4c195-c79e-4719-b867-cb1cdc7f5d40-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-xfdns\" (UID: \"81e4c195-c79e-4719-b867-cb1cdc7f5d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns"
Sep 11 17:26:08.935640 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:08.935612    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/81e4c195-c79e-4719-b867-cb1cdc7f5d40-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-xfdns\" (UID: \"81e4c195-c79e-4719-b867-cb1cdc7f5d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns"
Sep 11 17:26:09.036535 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:09.036452    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/81e4c195-c79e-4719-b867-cb1cdc7f5d40-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-xfdns\" (UID: \"81e4c195-c79e-4719-b867-cb1cdc7f5d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns"
Sep 11 17:26:09.036535 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:09.036492    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81e4c195-c79e-4719-b867-cb1cdc7f5d40-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-xfdns\" (UID: \"81e4c195-c79e-4719-b867-cb1cdc7f5d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns"
Sep 11 17:26:09.036535 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:09.036511    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81e4c195-c79e-4719-b867-cb1cdc7f5d40-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-xfdns\" (UID: \"81e4c195-c79e-4719-b867-cb1cdc7f5d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns"
Sep 11 17:26:09.036768 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:09.036554    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81e4c195-c79e-4719-b867-cb1cdc7f5d40-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-xfdns\" (UID: \"81e4c195-c79e-4719-b867-cb1cdc7f5d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns"
Sep 11 17:26:09.036768 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:09.036573    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c66cz\" (UniqueName: \"kubernetes.io/projected/81e4c195-c79e-4719-b867-cb1cdc7f5d40-kube-api-access-c66cz\") pod \"istiod-openshift-gateway-5677ccb49b-xfdns\" (UID: \"81e4c195-c79e-4719-b867-cb1cdc7f5d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns"
Sep 11 17:26:09.036768 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:09.036588    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e4c195-c79e-4719-b867-cb1cdc7f5d40-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-xfdns\" (UID: \"81e4c195-c79e-4719-b867-cb1cdc7f5d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns"
Sep 11 17:26:09.036768 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:09.036622    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81e4c195-c79e-4719-b867-cb1cdc7f5d40-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-xfdns\" (UID: \"81e4c195-c79e-4719-b867-cb1cdc7f5d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns"
Sep 11 17:26:09.037448 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:09.037411    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81e4c195-c79e-4719-b867-cb1cdc7f5d40-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-xfdns\" (UID: \"81e4c195-c79e-4719-b867-cb1cdc7f5d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns"
Sep 11 17:26:09.038927 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:09.038885    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e4c195-c79e-4719-b867-cb1cdc7f5d40-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-xfdns\" (UID: \"81e4c195-c79e-4719-b867-cb1cdc7f5d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns"
Sep 11 17:26:09.039005 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:09.038969    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/81e4c195-c79e-4719-b867-cb1cdc7f5d40-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-xfdns\" (UID: \"81e4c195-c79e-4719-b867-cb1cdc7f5d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns"
Sep 11 17:26:09.039005 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:09.038992    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81e4c195-c79e-4719-b867-cb1cdc7f5d40-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-xfdns\" (UID: \"81e4c195-c79e-4719-b867-cb1cdc7f5d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns"
Sep 11 17:26:09.039070 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:09.039056    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81e4c195-c79e-4719-b867-cb1cdc7f5d40-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-xfdns\" (UID: \"81e4c195-c79e-4719-b867-cb1cdc7f5d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns"
Sep 11 17:26:09.048889 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:09.048819    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5786cf49b6-qzhhh"]
Sep 11 17:26:09.051544 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:26:09.051521    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28e028d7_1f3a_4335_91ad_835ee47f0635.slice/crio-617d51bd854c0876c99075bcc70fa782b4c60b292023331d2e773f1b196ad68e WatchSource:0}: Error finding container 617d51bd854c0876c99075bcc70fa782b4c60b292023331d2e773f1b196ad68e: Status 404 returned error can't find the container with id 617d51bd854c0876c99075bcc70fa782b4c60b292023331d2e773f1b196ad68e
Sep 11 17:26:09.059907 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:09.059871    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c66cz\" (UniqueName: \"kubernetes.io/projected/81e4c195-c79e-4719-b867-cb1cdc7f5d40-kube-api-access-c66cz\") pod \"istiod-openshift-gateway-5677ccb49b-xfdns\" (UID: \"81e4c195-c79e-4719-b867-cb1cdc7f5d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns"
Sep 11 17:26:09.061689 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:09.061668    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81e4c195-c79e-4719-b867-cb1cdc7f5d40-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-xfdns\" (UID: \"81e4c195-c79e-4719-b867-cb1cdc7f5d40\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns"
Sep 11 17:26:09.152629 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:09.152592    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns"
Sep 11 17:26:09.266163 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:09.266129    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-qzhhh" event={"ID":"28e028d7-1f3a-4335-91ad-835ee47f0635","Type":"ContainerStarted","Data":"617d51bd854c0876c99075bcc70fa782b4c60b292023331d2e773f1b196ad68e"}
Sep 11 17:26:09.428917 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:09.428876    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns"]
Sep 11 17:26:09.431110 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:26:09.431085    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e4c195_c79e_4719_b867_cb1cdc7f5d40.slice/crio-27262b0177287cf6d7379b0e56f72bef53296c26c9a7aa096e845094ac1a263f WatchSource:0}: Error finding container 27262b0177287cf6d7379b0e56f72bef53296c26c9a7aa096e845094ac1a263f: Status 404 returned error can't find the container with id 27262b0177287cf6d7379b0e56f72bef53296c26c9a7aa096e845094ac1a263f
Sep 11 17:26:10.274839 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:10.274764    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns" event={"ID":"81e4c195-c79e-4719-b867-cb1cdc7f5d40","Type":"ContainerStarted","Data":"27262b0177287cf6d7379b0e56f72bef53296c26c9a7aa096e845094ac1a263f"}
Sep 11 17:26:12.027821 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:12.027772    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:26:12.028183 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:12.027865    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:26:12.285958 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:12.285826    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-qzhhh" event={"ID":"28e028d7-1f3a-4335-91ad-835ee47f0635","Type":"ContainerStarted","Data":"825e9a89b8e4a197a99e26370e43f3469de11968b00d46b9d67edf57ba6712c2"}
Sep 11 17:26:12.286121 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:12.285951    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-qzhhh"
Sep 11 17:26:12.287581 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:12.287560    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns" event={"ID":"81e4c195-c79e-4719-b867-cb1cdc7f5d40","Type":"ContainerStarted","Data":"e6503439837c9176249bd4f02d0f560a0af8626470840f51112c13898aca5c6b"}
Sep 11 17:26:12.287710 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:12.287687    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns"
Sep 11 17:26:12.343339 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:12.343288    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-qzhhh" podStartSLOduration=1.429495005 podStartE2EDuration="4.34327131s" totalImagesPullingTime="2.913776305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:26:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:26:12.341228447 +0000 UTC m=+340.145400870" watchObservedRunningTime="2026-09-11 17:26:12.34327131 +0000 UTC m=+340.147443729"
Sep 11 17:26:12.380620 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:12.380566    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns" podStartSLOduration=1.786252902 podStartE2EDuration="4.380550701s" totalImagesPullingTime="2.594297799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:26:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:26:12.379529267 +0000 UTC m=+340.183701689" watchObservedRunningTime="2026-09-11 17:26:12.380550701 +0000 UTC m=+340.184723119"
Sep 11 17:26:13.293373 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:13.293343    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xfdns"
Sep 11 17:26:23.294338 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:26:23.294304    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-qzhhh"
Sep 11 17:27:41.235337 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:41.235298    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-v2j7t"]
Sep 11 17:27:41.238603 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:41.238578    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-v2j7t"
Sep 11 17:27:41.238990 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:41.238963    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7qdg\" (UniqueName: \"kubernetes.io/projected/c9a6d33d-fba9-4377-b3cf-d92a0e878d36-kube-api-access-d7qdg\") pod \"kuadrant-operator-controller-manager-5f685c496f-v2j7t\" (UID: \"c9a6d33d-fba9-4377-b3cf-d92a0e878d36\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-v2j7t"
Sep 11 17:27:41.239084 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:41.239029    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9a6d33d-fba9-4377-b3cf-d92a0e878d36-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-v2j7t\" (UID: \"c9a6d33d-fba9-4377-b3cf-d92a0e878d36\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-v2j7t"
Sep 11 17:27:41.241196 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:41.241173    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 17:27:41.241313 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:41.241204    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 17:27:41.241313 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:41.241177    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xg8wc\""
Sep 11 17:27:41.250637 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:41.250612    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-v2j7t"]
Sep 11 17:27:41.340137 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:41.340100    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7qdg\" (UniqueName: \"kubernetes.io/projected/c9a6d33d-fba9-4377-b3cf-d92a0e878d36-kube-api-access-d7qdg\") pod \"kuadrant-operator-controller-manager-5f685c496f-v2j7t\" (UID: \"c9a6d33d-fba9-4377-b3cf-d92a0e878d36\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-v2j7t"
Sep 11 17:27:41.340301 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:41.340143    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9a6d33d-fba9-4377-b3cf-d92a0e878d36-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-v2j7t\" (UID: \"c9a6d33d-fba9-4377-b3cf-d92a0e878d36\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-v2j7t"
Sep 11 17:27:41.340503 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:41.340487    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9a6d33d-fba9-4377-b3cf-d92a0e878d36-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-v2j7t\" (UID: \"c9a6d33d-fba9-4377-b3cf-d92a0e878d36\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-v2j7t"
Sep 11 17:27:41.349416 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:41.349381    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7qdg\" (UniqueName: \"kubernetes.io/projected/c9a6d33d-fba9-4377-b3cf-d92a0e878d36-kube-api-access-d7qdg\") pod \"kuadrant-operator-controller-manager-5f685c496f-v2j7t\" (UID: \"c9a6d33d-fba9-4377-b3cf-d92a0e878d36\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-v2j7t"
Sep 11 17:27:41.548875 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:41.548776    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-v2j7t"
Sep 11 17:27:41.832230 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:41.832205    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-v2j7t"]
Sep 11 17:27:41.834942 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:27:41.834912    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9a6d33d_fba9_4377_b3cf_d92a0e878d36.slice/crio-bd8a1b06ef4468a6ae9caf67041b5ec0bc7aec67c4fb01a3f46020d3be129d82 WatchSource:0}: Error finding container bd8a1b06ef4468a6ae9caf67041b5ec0bc7aec67c4fb01a3f46020d3be129d82: Status 404 returned error can't find the container with id bd8a1b06ef4468a6ae9caf67041b5ec0bc7aec67c4fb01a3f46020d3be129d82
Sep 11 17:27:41.891961 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:41.891925    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-v2j7t"]
Sep 11 17:27:41.906597 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:41.906566    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-v2j7t"]
Sep 11 17:27:41.926041 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:41.926008    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7p57m"]
Sep 11 17:27:41.930305 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:41.930286    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7p57m"
Sep 11 17:27:41.944572 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:41.944545    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9d7f\" (UniqueName: \"kubernetes.io/projected/23fe443f-b8b6-4814-b64f-b7eee4bf99f5-kube-api-access-z9d7f\") pod \"kuadrant-operator-controller-manager-5f685c496f-7p57m\" (UID: \"23fe443f-b8b6-4814-b64f-b7eee4bf99f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7p57m"
Sep 11 17:27:41.944713 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:41.944597    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23fe443f-b8b6-4814-b64f-b7eee4bf99f5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-7p57m\" (UID: \"23fe443f-b8b6-4814-b64f-b7eee4bf99f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7p57m"
Sep 11 17:27:41.948663 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:41.948635    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7p57m"]
Sep 11 17:27:42.045726 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:42.045694    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9d7f\" (UniqueName: \"kubernetes.io/projected/23fe443f-b8b6-4814-b64f-b7eee4bf99f5-kube-api-access-z9d7f\") pod \"kuadrant-operator-controller-manager-5f685c496f-7p57m\" (UID: \"23fe443f-b8b6-4814-b64f-b7eee4bf99f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7p57m"
Sep 11 17:27:42.045921 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:42.045785    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23fe443f-b8b6-4814-b64f-b7eee4bf99f5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-7p57m\" (UID: \"23fe443f-b8b6-4814-b64f-b7eee4bf99f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7p57m"
Sep 11 17:27:42.046199 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:42.046175    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23fe443f-b8b6-4814-b64f-b7eee4bf99f5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-7p57m\" (UID: \"23fe443f-b8b6-4814-b64f-b7eee4bf99f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7p57m"
Sep 11 17:27:42.113046 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:42.112975    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9d7f\" (UniqueName: \"kubernetes.io/projected/23fe443f-b8b6-4814-b64f-b7eee4bf99f5-kube-api-access-z9d7f\") pod \"kuadrant-operator-controller-manager-5f685c496f-7p57m\" (UID: \"23fe443f-b8b6-4814-b64f-b7eee4bf99f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7p57m"
Sep 11 17:27:42.240945 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:42.240885    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7p57m"
Sep 11 17:27:42.534340 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:42.534238    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7p57m"]
Sep 11 17:27:42.539199 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:27:42.539171    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23fe443f_b8b6_4814_b64f_b7eee4bf99f5.slice/crio-9bea3361ec0fce19667dc223ad2c55ab42549ba583c4050291989738cead77a7 WatchSource:0}: Error finding container 9bea3361ec0fce19667dc223ad2c55ab42549ba583c4050291989738cead77a7: Status 404 returned error can't find the container with id 9bea3361ec0fce19667dc223ad2c55ab42549ba583c4050291989738cead77a7
Sep 11 17:27:42.680840 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:42.680795    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7p57m" event={"ID":"23fe443f-b8b6-4814-b64f-b7eee4bf99f5","Type":"ContainerStarted","Data":"9bea3361ec0fce19667dc223ad2c55ab42549ba583c4050291989738cead77a7"}
Sep 11 17:27:46.707476 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:46.707437    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7p57m" event={"ID":"23fe443f-b8b6-4814-b64f-b7eee4bf99f5","Type":"ContainerStarted","Data":"52856504f425c4df62baa2d2344ed4506339d266ca7765b6fa343c99f03216f6"}
Sep 11 17:27:46.707928 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:46.707594    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7p57m"
Sep 11 17:27:46.709197 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:46.709168    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-v2j7t" podUID="c9a6d33d-fba9-4377-b3cf-d92a0e878d36" containerName="manager" containerID="cri-o://dc45911da0eaed99ffcf47978521f204f65764f3b3014beb781fee0ef6001f01" gracePeriod=2
Sep 11 17:27:46.733143 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:46.733098    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7p57m" podStartSLOduration=2.302011826 podStartE2EDuration="5.73308294s" totalImagesPullingTime="3.431071114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:27:46.730344547 +0000 UTC m=+434.534516966" watchObservedRunningTime="2026-09-11 17:27:46.73308294 +0000 UTC m=+434.537255359"
Sep 11 17:27:46.955876 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:46.955849    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-v2j7t"
Sep 11 17:27:46.991252 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:46.991163    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9a6d33d-fba9-4377-b3cf-d92a0e878d36-extensions-socket-volume\") pod \"c9a6d33d-fba9-4377-b3cf-d92a0e878d36\" (UID: \"c9a6d33d-fba9-4377-b3cf-d92a0e878d36\") "
Sep 11 17:27:46.991404 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:46.991283    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7qdg\" (UniqueName: \"kubernetes.io/projected/c9a6d33d-fba9-4377-b3cf-d92a0e878d36-kube-api-access-d7qdg\") pod \"c9a6d33d-fba9-4377-b3cf-d92a0e878d36\" (UID: \"c9a6d33d-fba9-4377-b3cf-d92a0e878d36\") "
Sep 11 17:27:46.991500 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:46.991471    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a6d33d-fba9-4377-b3cf-d92a0e878d36-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c9a6d33d-fba9-4377-b3cf-d92a0e878d36" (UID: "c9a6d33d-fba9-4377-b3cf-d92a0e878d36"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:27:46.993964 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:46.993930    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a6d33d-fba9-4377-b3cf-d92a0e878d36-kube-api-access-d7qdg" (OuterVolumeSpecName: "kube-api-access-d7qdg") pod "c9a6d33d-fba9-4377-b3cf-d92a0e878d36" (UID: "c9a6d33d-fba9-4377-b3cf-d92a0e878d36"). InnerVolumeSpecName "kube-api-access-d7qdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:27:47.091827 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:47.091791    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7qdg\" (UniqueName: \"kubernetes.io/projected/c9a6d33d-fba9-4377-b3cf-d92a0e878d36-kube-api-access-d7qdg\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:27:47.091827 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:47.091818    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9a6d33d-fba9-4377-b3cf-d92a0e878d36-extensions-socket-volume\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:27:47.715792 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:47.715753    2560 generic.go:356] "Generic (PLEG): container finished" podID="c9a6d33d-fba9-4377-b3cf-d92a0e878d36" containerID="dc45911da0eaed99ffcf47978521f204f65764f3b3014beb781fee0ef6001f01" exitCode=2
Sep 11 17:27:47.715792 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:47.715795    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-v2j7t"
Sep 11 17:27:47.716320 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:47.715855    2560 scope.go:117] "RemoveContainer" containerID="dc45911da0eaed99ffcf47978521f204f65764f3b3014beb781fee0ef6001f01"
Sep 11 17:27:47.718081 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:27:47.718057    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-v2j7t\" is forbidden: User \"system:node:ip-10-0-130-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-89.ec2.internal' and this object" podUID="c9a6d33d-fba9-4377-b3cf-d92a0e878d36" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-v2j7t"
Sep 11 17:27:47.724989 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:47.724969    2560 scope.go:117] "RemoveContainer" containerID="dc45911da0eaed99ffcf47978521f204f65764f3b3014beb781fee0ef6001f01"
Sep 11 17:27:47.725268 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:27:47.725249    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc45911da0eaed99ffcf47978521f204f65764f3b3014beb781fee0ef6001f01\": container with ID starting with dc45911da0eaed99ffcf47978521f204f65764f3b3014beb781fee0ef6001f01 not found: ID does not exist" containerID="dc45911da0eaed99ffcf47978521f204f65764f3b3014beb781fee0ef6001f01"
Sep 11 17:27:47.725324 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:47.725276    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc45911da0eaed99ffcf47978521f204f65764f3b3014beb781fee0ef6001f01"} err="failed to get container status \"dc45911da0eaed99ffcf47978521f204f65764f3b3014beb781fee0ef6001f01\": rpc error: code = NotFound desc = could not find container \"dc45911da0eaed99ffcf47978521f204f65764f3b3014beb781fee0ef6001f01\": container with ID starting with dc45911da0eaed99ffcf47978521f204f65764f3b3014beb781fee0ef6001f01 not found: ID does not exist"
Sep 11 17:27:47.726936 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:27:47.726888    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-v2j7t\" is forbidden: User \"system:node:ip-10-0-130-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-89.ec2.internal' and this object" podUID="c9a6d33d-fba9-4377-b3cf-d92a0e878d36" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-v2j7t"
Sep 11 17:27:48.714780 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:48.714746    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9a6d33d-fba9-4377-b3cf-d92a0e878d36" path="/var/lib/kubelet/pods/c9a6d33d-fba9-4377-b3cf-d92a0e878d36/volumes"
Sep 11 17:27:57.718522 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:27:57.718442    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7p57m"
Sep 11 17:28:09.868965 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:09.868931    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7p57m"]
Sep 11 17:28:09.869443 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:09.869132    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7p57m" podUID="23fe443f-b8b6-4814-b64f-b7eee4bf99f5" containerName="manager" containerID="cri-o://52856504f425c4df62baa2d2344ed4506339d266ca7765b6fa343c99f03216f6" gracePeriod=10
Sep 11 17:28:10.123551 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:10.123485    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7p57m"
Sep 11 17:28:10.203946 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:10.203886    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23fe443f-b8b6-4814-b64f-b7eee4bf99f5-extensions-socket-volume\") pod \"23fe443f-b8b6-4814-b64f-b7eee4bf99f5\" (UID: \"23fe443f-b8b6-4814-b64f-b7eee4bf99f5\") "
Sep 11 17:28:10.204141 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:10.204017    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9d7f\" (UniqueName: \"kubernetes.io/projected/23fe443f-b8b6-4814-b64f-b7eee4bf99f5-kube-api-access-z9d7f\") pod \"23fe443f-b8b6-4814-b64f-b7eee4bf99f5\" (UID: \"23fe443f-b8b6-4814-b64f-b7eee4bf99f5\") "
Sep 11 17:28:10.204395 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:10.204366    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23fe443f-b8b6-4814-b64f-b7eee4bf99f5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "23fe443f-b8b6-4814-b64f-b7eee4bf99f5" (UID: "23fe443f-b8b6-4814-b64f-b7eee4bf99f5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:28:10.206103 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:10.206075    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23fe443f-b8b6-4814-b64f-b7eee4bf99f5-kube-api-access-z9d7f" (OuterVolumeSpecName: "kube-api-access-z9d7f") pod "23fe443f-b8b6-4814-b64f-b7eee4bf99f5" (UID: "23fe443f-b8b6-4814-b64f-b7eee4bf99f5"). InnerVolumeSpecName "kube-api-access-z9d7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:28:10.305351 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:10.305314    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23fe443f-b8b6-4814-b64f-b7eee4bf99f5-extensions-socket-volume\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:28:10.305351 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:10.305344    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9d7f\" (UniqueName: \"kubernetes.io/projected/23fe443f-b8b6-4814-b64f-b7eee4bf99f5-kube-api-access-z9d7f\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:28:10.822653 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:10.822619    2560 generic.go:356] "Generic (PLEG): container finished" podID="23fe443f-b8b6-4814-b64f-b7eee4bf99f5" containerID="52856504f425c4df62baa2d2344ed4506339d266ca7765b6fa343c99f03216f6" exitCode=0
Sep 11 17:28:10.822807 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:10.822674    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7p57m"
Sep 11 17:28:10.822807 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:10.822693    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7p57m" event={"ID":"23fe443f-b8b6-4814-b64f-b7eee4bf99f5","Type":"ContainerDied","Data":"52856504f425c4df62baa2d2344ed4506339d266ca7765b6fa343c99f03216f6"}
Sep 11 17:28:10.822807 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:10.822761    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7p57m" event={"ID":"23fe443f-b8b6-4814-b64f-b7eee4bf99f5","Type":"ContainerDied","Data":"9bea3361ec0fce19667dc223ad2c55ab42549ba583c4050291989738cead77a7"}
Sep 11 17:28:10.822807 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:10.822782    2560 scope.go:117] "RemoveContainer" containerID="52856504f425c4df62baa2d2344ed4506339d266ca7765b6fa343c99f03216f6"
Sep 11 17:28:10.830838 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:10.830821    2560 scope.go:117] "RemoveContainer" containerID="52856504f425c4df62baa2d2344ed4506339d266ca7765b6fa343c99f03216f6"
Sep 11 17:28:10.831131 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:28:10.831109    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52856504f425c4df62baa2d2344ed4506339d266ca7765b6fa343c99f03216f6\": container with ID starting with 52856504f425c4df62baa2d2344ed4506339d266ca7765b6fa343c99f03216f6 not found: ID does not exist" containerID="52856504f425c4df62baa2d2344ed4506339d266ca7765b6fa343c99f03216f6"
Sep 11 17:28:10.831194 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:10.831138    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52856504f425c4df62baa2d2344ed4506339d266ca7765b6fa343c99f03216f6"} err="failed to get container status \"52856504f425c4df62baa2d2344ed4506339d266ca7765b6fa343c99f03216f6\": rpc error: code = NotFound desc = could not find container \"52856504f425c4df62baa2d2344ed4506339d266ca7765b6fa343c99f03216f6\": container with ID starting with 52856504f425c4df62baa2d2344ed4506339d266ca7765b6fa343c99f03216f6 not found: ID does not exist"
Sep 11 17:28:10.844858 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:10.844832    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7p57m"]
Sep 11 17:28:10.850991 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:10.850962    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7p57m"]
Sep 11 17:28:12.714595 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:12.714565    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23fe443f-b8b6-4814-b64f-b7eee4bf99f5" path="/var/lib/kubelet/pods/23fe443f-b8b6-4814-b64f-b7eee4bf99f5/volumes"
Sep 11 17:28:28.079280 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:28.079244    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-dz59f"]
Sep 11 17:28:28.079647 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:28.079550    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23fe443f-b8b6-4814-b64f-b7eee4bf99f5" containerName="manager"
Sep 11 17:28:28.079647 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:28.079562    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="23fe443f-b8b6-4814-b64f-b7eee4bf99f5" containerName="manager"
Sep 11 17:28:28.079647 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:28.079584    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9a6d33d-fba9-4377-b3cf-d92a0e878d36" containerName="manager"
Sep 11 17:28:28.079647 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:28.079590    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9a6d33d-fba9-4377-b3cf-d92a0e878d36" containerName="manager"
Sep 11 17:28:28.079647 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:28.079643    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="23fe443f-b8b6-4814-b64f-b7eee4bf99f5" containerName="manager"
Sep 11 17:28:28.079805 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:28.079652    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9a6d33d-fba9-4377-b3cf-d92a0e878d36" containerName="manager"
Sep 11 17:28:28.082535 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:28.082519    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-dz59f"
Sep 11 17:28:28.085041 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:28.085017    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 17:28:28.085174 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:28.085044    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 17:28:28.085174 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:28.085065    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 17:28:28.086011 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:28.085983    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nnjqp\""
Sep 11 17:28:28.093226 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:28.093202    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-dz59f"]
Sep 11 17:28:28.163554 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:28.163518    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2cqj\" (UniqueName: \"kubernetes.io/projected/1303a070-5f85-48f8-bd5a-e2be73d3b762-kube-api-access-k2cqj\") pod \"limitador-limitador-8588749977-dz59f\" (UID: \"1303a070-5f85-48f8-bd5a-e2be73d3b762\") " pod="kuadrant-system/limitador-limitador-8588749977-dz59f"
Sep 11 17:28:28.163729 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:28.163595    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1303a070-5f85-48f8-bd5a-e2be73d3b762-config-file\") pod \"limitador-limitador-8588749977-dz59f\" (UID: \"1303a070-5f85-48f8-bd5a-e2be73d3b762\") " pod="kuadrant-system/limitador-limitador-8588749977-dz59f"
Sep 11 17:28:28.175913 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:28.175865    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-dz59f"]
Sep 11 17:28:28.264529 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:28.264487    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2cqj\" (UniqueName: \"kubernetes.io/projected/1303a070-5f85-48f8-bd5a-e2be73d3b762-kube-api-access-k2cqj\") pod \"limitador-limitador-8588749977-dz59f\" (UID: \"1303a070-5f85-48f8-bd5a-e2be73d3b762\") " pod="kuadrant-system/limitador-limitador-8588749977-dz59f"
Sep 11 17:28:28.264698 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:28.264569    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1303a070-5f85-48f8-bd5a-e2be73d3b762-config-file\") pod \"limitador-limitador-8588749977-dz59f\" (UID: \"1303a070-5f85-48f8-bd5a-e2be73d3b762\") " pod="kuadrant-system/limitador-limitador-8588749977-dz59f"
Sep 11 17:28:28.265570 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:28.265549    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1303a070-5f85-48f8-bd5a-e2be73d3b762-config-file\") pod \"limitador-limitador-8588749977-dz59f\" (UID: \"1303a070-5f85-48f8-bd5a-e2be73d3b762\") " pod="kuadrant-system/limitador-limitador-8588749977-dz59f"
Sep 11 17:28:28.272817 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:28.272794    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2cqj\" (UniqueName: \"kubernetes.io/projected/1303a070-5f85-48f8-bd5a-e2be73d3b762-kube-api-access-k2cqj\") pod \"limitador-limitador-8588749977-dz59f\" (UID: \"1303a070-5f85-48f8-bd5a-e2be73d3b762\") " pod="kuadrant-system/limitador-limitador-8588749977-dz59f"
Sep 11 17:28:28.393645 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:28.393555    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-dz59f"
Sep 11 17:28:28.663390 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:28.663351    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-dz59f"]
Sep 11 17:28:28.665391 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:28:28.665360    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1303a070_5f85_48f8_bd5a_e2be73d3b762.slice/crio-8bda3cde5881dadea9047a102fc5b2636856048dff8bb69bc8c050e59cfa4629 WatchSource:0}: Error finding container 8bda3cde5881dadea9047a102fc5b2636856048dff8bb69bc8c050e59cfa4629: Status 404 returned error can't find the container with id 8bda3cde5881dadea9047a102fc5b2636856048dff8bb69bc8c050e59cfa4629
Sep 11 17:28:28.910737 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:28.910188    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-dz59f" event={"ID":"1303a070-5f85-48f8-bd5a-e2be73d3b762","Type":"ContainerStarted","Data":"8bda3cde5881dadea9047a102fc5b2636856048dff8bb69bc8c050e59cfa4629"}
Sep 11 17:28:30.920272 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:30.920233    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-dz59f" event={"ID":"1303a070-5f85-48f8-bd5a-e2be73d3b762","Type":"ContainerStarted","Data":"a791d36d93652382b2477a9abdff52f6265b040bf0ff7ddbb0ea7527e3899f0f"}
Sep 11 17:28:30.920665 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:30.920318    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-dz59f"
Sep 11 17:28:30.937580 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:30.937525    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-dz59f" podStartSLOduration=1.235193248 podStartE2EDuration="2.937507747s" totalImagesPullingTime="1.702314499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:28:30.936118448 +0000 UTC m=+478.740290869" watchObservedRunningTime="2026-09-11 17:28:30.937507747 +0000 UTC m=+478.741680194"
Sep 11 17:28:41.925977 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:41.925942    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-dz59f"
Sep 11 17:28:42.035941 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:42.035889    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-dz59f"]
Sep 11 17:28:42.036102 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:42.036082    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-dz59f" podUID="1303a070-5f85-48f8-bd5a-e2be73d3b762" containerName="limitador" containerID="cri-o://a791d36d93652382b2477a9abdff52f6265b040bf0ff7ddbb0ea7527e3899f0f" gracePeriod=30
Sep 11 17:28:42.589556 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:42.589525    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-dz59f"
Sep 11 17:28:42.688177 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:42.688099    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1303a070-5f85-48f8-bd5a-e2be73d3b762-config-file\") pod \"1303a070-5f85-48f8-bd5a-e2be73d3b762\" (UID: \"1303a070-5f85-48f8-bd5a-e2be73d3b762\") "
Sep 11 17:28:42.688347 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:42.688196    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2cqj\" (UniqueName: \"kubernetes.io/projected/1303a070-5f85-48f8-bd5a-e2be73d3b762-kube-api-access-k2cqj\") pod \"1303a070-5f85-48f8-bd5a-e2be73d3b762\" (UID: \"1303a070-5f85-48f8-bd5a-e2be73d3b762\") "
Sep 11 17:28:42.688493 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:42.688472    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1303a070-5f85-48f8-bd5a-e2be73d3b762-config-file" (OuterVolumeSpecName: "config-file") pod "1303a070-5f85-48f8-bd5a-e2be73d3b762" (UID: "1303a070-5f85-48f8-bd5a-e2be73d3b762"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:28:42.690279 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:42.690255    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1303a070-5f85-48f8-bd5a-e2be73d3b762-kube-api-access-k2cqj" (OuterVolumeSpecName: "kube-api-access-k2cqj") pod "1303a070-5f85-48f8-bd5a-e2be73d3b762" (UID: "1303a070-5f85-48f8-bd5a-e2be73d3b762"). InnerVolumeSpecName "kube-api-access-k2cqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:28:42.789560 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:42.789519    2560 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1303a070-5f85-48f8-bd5a-e2be73d3b762-config-file\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:28:42.789560 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:42.789552    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2cqj\" (UniqueName: \"kubernetes.io/projected/1303a070-5f85-48f8-bd5a-e2be73d3b762-kube-api-access-k2cqj\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:28:42.971292 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:42.971203    2560 generic.go:356] "Generic (PLEG): container finished" podID="1303a070-5f85-48f8-bd5a-e2be73d3b762" containerID="a791d36d93652382b2477a9abdff52f6265b040bf0ff7ddbb0ea7527e3899f0f" exitCode=0
Sep 11 17:28:42.971292 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:42.971247    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-dz59f"
Sep 11 17:28:42.971810 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:42.971292    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-dz59f" event={"ID":"1303a070-5f85-48f8-bd5a-e2be73d3b762","Type":"ContainerDied","Data":"a791d36d93652382b2477a9abdff52f6265b040bf0ff7ddbb0ea7527e3899f0f"}
Sep 11 17:28:42.971810 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:42.971352    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-dz59f" event={"ID":"1303a070-5f85-48f8-bd5a-e2be73d3b762","Type":"ContainerDied","Data":"8bda3cde5881dadea9047a102fc5b2636856048dff8bb69bc8c050e59cfa4629"}
Sep 11 17:28:42.971810 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:42.971383    2560 scope.go:117] "RemoveContainer" containerID="a791d36d93652382b2477a9abdff52f6265b040bf0ff7ddbb0ea7527e3899f0f"
Sep 11 17:28:42.979385 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:42.979365    2560 scope.go:117] "RemoveContainer" containerID="a791d36d93652382b2477a9abdff52f6265b040bf0ff7ddbb0ea7527e3899f0f"
Sep 11 17:28:42.979670 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:28:42.979651    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a791d36d93652382b2477a9abdff52f6265b040bf0ff7ddbb0ea7527e3899f0f\": container with ID starting with a791d36d93652382b2477a9abdff52f6265b040bf0ff7ddbb0ea7527e3899f0f not found: ID does not exist" containerID="a791d36d93652382b2477a9abdff52f6265b040bf0ff7ddbb0ea7527e3899f0f"
Sep 11 17:28:42.979719 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:42.979679    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a791d36d93652382b2477a9abdff52f6265b040bf0ff7ddbb0ea7527e3899f0f"} err="failed to get container status \"a791d36d93652382b2477a9abdff52f6265b040bf0ff7ddbb0ea7527e3899f0f\": rpc error: code = NotFound desc = could not find container \"a791d36d93652382b2477a9abdff52f6265b040bf0ff7ddbb0ea7527e3899f0f\": container with ID starting with a791d36d93652382b2477a9abdff52f6265b040bf0ff7ddbb0ea7527e3899f0f not found: ID does not exist"
Sep 11 17:28:42.988247 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:42.988226    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-dz59f"]
Sep 11 17:28:42.992537 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:42.992518    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-dz59f"]
Sep 11 17:28:44.714946 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:44.714890    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1303a070-5f85-48f8-bd5a-e2be73d3b762" path="/var/lib/kubelet/pods/1303a070-5f85-48f8-bd5a-e2be73d3b762/volumes"
Sep 11 17:28:48.211344 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:48.211311    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wx4sf"]
Sep 11 17:28:48.211802 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:48.211787    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1303a070-5f85-48f8-bd5a-e2be73d3b762" containerName="limitador"
Sep 11 17:28:48.211846 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:48.211805    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1303a070-5f85-48f8-bd5a-e2be73d3b762" containerName="limitador"
Sep 11 17:28:48.211939 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:48.211926    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1303a070-5f85-48f8-bd5a-e2be73d3b762" containerName="limitador"
Sep 11 17:28:48.216374 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:48.216350    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wx4sf"
Sep 11 17:28:48.219015 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:48.218994    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 11 17:28:48.219132 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:48.219042    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 17:28:48.219838 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:48.219822    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-x58fl\""
Sep 11 17:28:48.219936 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:48.219826    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:28:48.225207 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:48.225185    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wx4sf"]
Sep 11 17:28:48.342607 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:48.342568    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfmcm\" (UniqueName: \"kubernetes.io/projected/62cdf1c0-2378-4cdf-a084-11992f088ab8-kube-api-access-lfmcm\") pod \"postgres-69fb4594cf-wx4sf\" (UID: \"62cdf1c0-2378-4cdf-a084-11992f088ab8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wx4sf"
Sep 11 17:28:48.342764 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:48.342625    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/62cdf1c0-2378-4cdf-a084-11992f088ab8-data\") pod \"postgres-69fb4594cf-wx4sf\" (UID: \"62cdf1c0-2378-4cdf-a084-11992f088ab8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wx4sf"
Sep 11 17:28:48.443821 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:48.443780    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/62cdf1c0-2378-4cdf-a084-11992f088ab8-data\") pod \"postgres-69fb4594cf-wx4sf\" (UID: \"62cdf1c0-2378-4cdf-a084-11992f088ab8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wx4sf"
Sep 11 17:28:48.444001 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:48.443917    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfmcm\" (UniqueName: \"kubernetes.io/projected/62cdf1c0-2378-4cdf-a084-11992f088ab8-kube-api-access-lfmcm\") pod \"postgres-69fb4594cf-wx4sf\" (UID: \"62cdf1c0-2378-4cdf-a084-11992f088ab8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wx4sf"
Sep 11 17:28:48.444276 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:48.444253    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/62cdf1c0-2378-4cdf-a084-11992f088ab8-data\") pod \"postgres-69fb4594cf-wx4sf\" (UID: \"62cdf1c0-2378-4cdf-a084-11992f088ab8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wx4sf"
Sep 11 17:28:48.452704 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:48.452675    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfmcm\" (UniqueName: \"kubernetes.io/projected/62cdf1c0-2378-4cdf-a084-11992f088ab8-kube-api-access-lfmcm\") pod \"postgres-69fb4594cf-wx4sf\" (UID: \"62cdf1c0-2378-4cdf-a084-11992f088ab8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wx4sf"
Sep 11 17:28:48.527570 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:48.527487    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wx4sf"
Sep 11 17:28:48.795617 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:48.795532    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wx4sf"]
Sep 11 17:28:48.798457 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:28:48.798418    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62cdf1c0_2378_4cdf_a084_11992f088ab8.slice/crio-a0faa21a083fddfa47b6f150bd7f82f1a42fe5887ac1daa65415f70c5abe50ae WatchSource:0}: Error finding container a0faa21a083fddfa47b6f150bd7f82f1a42fe5887ac1daa65415f70c5abe50ae: Status 404 returned error can't find the container with id a0faa21a083fddfa47b6f150bd7f82f1a42fe5887ac1daa65415f70c5abe50ae
Sep 11 17:28:48.998372 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:48.998338    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wx4sf" event={"ID":"62cdf1c0-2378-4cdf-a084-11992f088ab8","Type":"ContainerStarted","Data":"a0faa21a083fddfa47b6f150bd7f82f1a42fe5887ac1daa65415f70c5abe50ae"}
Sep 11 17:28:54.028192 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:54.028156    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wx4sf" event={"ID":"62cdf1c0-2378-4cdf-a084-11992f088ab8","Type":"ContainerStarted","Data":"d1668411e9a1193735832e1ca625ba666dd551f842b43e45d550d63d888f1e53"}
Sep 11 17:28:54.028552 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:54.028289    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wx4sf"
Sep 11 17:28:54.045641 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:28:54.045590    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wx4sf" podStartSLOduration=0.955720671 podStartE2EDuration="6.045574812s" totalImagesPullingTime="5.089854141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:28:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:28:54.043377159 +0000 UTC m=+501.847549578" watchObservedRunningTime="2026-09-11 17:28:54.045574812 +0000 UTC m=+501.849747231"
Sep 11 17:29:00.060930 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:00.060879    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wx4sf"
Sep 11 17:29:07.245002 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:07.244956    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5d6579744d-92qcp"]
Sep 11 17:29:07.264856 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:07.264817    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d6579744d-92qcp"]
Sep 11 17:29:07.265050 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:07.264958    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d6579744d-92qcp"
Sep 11 17:29:07.267448 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:07.267425    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 17:29:07.267589 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:07.267449    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-r7knj\""
Sep 11 17:29:07.267589 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:07.267431    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 17:29:07.267715 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:07.267704    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 17:29:07.403953 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:07.403908    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6ddbd79556-qd9nn"]
Sep 11 17:29:07.412577 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:07.411990    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ddbd79556-qd9nn"
Sep 11 17:29:07.416869 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:07.416840    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6ddbd79556-qd9nn"]
Sep 11 17:29:07.420153 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:07.420053    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e110b6ea-4472-4a07-973e-40c599278432-webhook-cert\") pod \"maas-controller-5d6579744d-92qcp\" (UID: \"e110b6ea-4472-4a07-973e-40c599278432\") " pod="opendatahub/maas-controller-5d6579744d-92qcp"
Sep 11 17:29:07.420290 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:07.420247    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e110b6ea-4472-4a07-973e-40c599278432-metrics-certs\") pod \"maas-controller-5d6579744d-92qcp\" (UID: \"e110b6ea-4472-4a07-973e-40c599278432\") " pod="opendatahub/maas-controller-5d6579744d-92qcp"
Sep 11 17:29:07.420356 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:07.420312    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkwtp\" (UniqueName: \"kubernetes.io/projected/e110b6ea-4472-4a07-973e-40c599278432-kube-api-access-hkwtp\") pod \"maas-controller-5d6579744d-92qcp\" (UID: \"e110b6ea-4472-4a07-973e-40c599278432\") " pod="opendatahub/maas-controller-5d6579744d-92qcp"
Sep 11 17:29:07.521838 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:07.521742    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e110b6ea-4472-4a07-973e-40c599278432-webhook-cert\") pod \"maas-controller-5d6579744d-92qcp\" (UID: \"e110b6ea-4472-4a07-973e-40c599278432\") " pod="opendatahub/maas-controller-5d6579744d-92qcp"
Sep 11 17:29:07.521838 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:07.521786    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70ff87e0-8bf9-4f4a-b32b-242baf27ceaa-webhook-cert\") pod \"maas-controller-6ddbd79556-qd9nn\" (UID: \"70ff87e0-8bf9-4f4a-b32b-242baf27ceaa\") " pod="opendatahub/maas-controller-6ddbd79556-qd9nn"
Sep 11 17:29:07.521838 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:07.521835    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e110b6ea-4472-4a07-973e-40c599278432-metrics-certs\") pod \"maas-controller-5d6579744d-92qcp\" (UID: \"e110b6ea-4472-4a07-973e-40c599278432\") " pod="opendatahub/maas-controller-5d6579744d-92qcp"
Sep 11 17:29:07.522157 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:07.521855    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5p5r\" (UniqueName: \"kubernetes.io/projected/70ff87e0-8bf9-4f4a-b32b-242baf27ceaa-kube-api-access-b5p5r\") pod \"maas-controller-6ddbd79556-qd9nn\" (UID: \"70ff87e0-8bf9-4f4a-b32b-242baf27ceaa\") " pod="opendatahub/maas-controller-6ddbd79556-qd9nn"
Sep 11 17:29:07.522157 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:07.521880    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkwtp\" (UniqueName: \"kubernetes.io/projected/e110b6ea-4472-4a07-973e-40c599278432-kube-api-access-hkwtp\") pod \"maas-controller-5d6579744d-92qcp\" (UID: \"e110b6ea-4472-4a07-973e-40c599278432\") " pod="opendatahub/maas-controller-5d6579744d-92qcp"
Sep 11 17:29:07.522157 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:29:07.521939    2560 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 11 17:29:07.522157 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:29:07.522031    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e110b6ea-4472-4a07-973e-40c599278432-webhook-cert podName:e110b6ea-4472-4a07-973e-40c599278432 nodeName:}" failed. No retries permitted until 2026-09-11 17:29:08.022008595 +0000 UTC m=+515.826181000 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/e110b6ea-4472-4a07-973e-40c599278432-webhook-cert") pod "maas-controller-5d6579744d-92qcp" (UID: "e110b6ea-4472-4a07-973e-40c599278432") : secret "maas-controller-webhook-cert" not found
Sep 11 17:29:07.522157 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:29:07.522059    2560 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 11 17:29:07.522157 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:29:07.522142    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e110b6ea-4472-4a07-973e-40c599278432-metrics-certs podName:e110b6ea-4472-4a07-973e-40c599278432 nodeName:}" failed. No retries permitted until 2026-09-11 17:29:08.022122972 +0000 UTC m=+515.826295386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e110b6ea-4472-4a07-973e-40c599278432-metrics-certs") pod "maas-controller-5d6579744d-92qcp" (UID: "e110b6ea-4472-4a07-973e-40c599278432") : secret "maas-controller-metrics-tls" not found
Sep 11 17:29:07.522376 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:07.522166    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70ff87e0-8bf9-4f4a-b32b-242baf27ceaa-metrics-certs\") pod \"maas-controller-6ddbd79556-qd9nn\" (UID: \"70ff87e0-8bf9-4f4a-b32b-242baf27ceaa\") " pod="opendatahub/maas-controller-6ddbd79556-qd9nn"
Sep 11 17:29:07.530924 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:07.530863    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkwtp\" (UniqueName: \"kubernetes.io/projected/e110b6ea-4472-4a07-973e-40c599278432-kube-api-access-hkwtp\") pod \"maas-controller-5d6579744d-92qcp\" (UID: \"e110b6ea-4472-4a07-973e-40c599278432\") " pod="opendatahub/maas-controller-5d6579744d-92qcp"
Sep 11 17:29:07.623337 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:07.623305    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70ff87e0-8bf9-4f4a-b32b-242baf27ceaa-metrics-certs\") pod \"maas-controller-6ddbd79556-qd9nn\" (UID: \"70ff87e0-8bf9-4f4a-b32b-242baf27ceaa\") " pod="opendatahub/maas-controller-6ddbd79556-qd9nn"
Sep 11 17:29:07.623512 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:07.623355    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70ff87e0-8bf9-4f4a-b32b-242baf27ceaa-webhook-cert\") pod \"maas-controller-6ddbd79556-qd9nn\" (UID: \"70ff87e0-8bf9-4f4a-b32b-242baf27ceaa\") " pod="opendatahub/maas-controller-6ddbd79556-qd9nn"
Sep 11 17:29:07.623512 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:07.623407    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5p5r\" (UniqueName: \"kubernetes.io/projected/70ff87e0-8bf9-4f4a-b32b-242baf27ceaa-kube-api-access-b5p5r\") pod \"maas-controller-6ddbd79556-qd9nn\" (UID: \"70ff87e0-8bf9-4f4a-b32b-242baf27ceaa\") " pod="opendatahub/maas-controller-6ddbd79556-qd9nn"
Sep 11 17:29:07.623512 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:29:07.623475    2560 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 11 17:29:07.623630 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:29:07.623518    2560 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 11 17:29:07.623630 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:29:07.623554    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70ff87e0-8bf9-4f4a-b32b-242baf27ceaa-metrics-certs podName:70ff87e0-8bf9-4f4a-b32b-242baf27ceaa nodeName:}" failed. No retries permitted until 2026-09-11 17:29:08.123534404 +0000 UTC m=+515.927706808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70ff87e0-8bf9-4f4a-b32b-242baf27ceaa-metrics-certs") pod "maas-controller-6ddbd79556-qd9nn" (UID: "70ff87e0-8bf9-4f4a-b32b-242baf27ceaa") : secret "maas-controller-metrics-tls" not found
Sep 11 17:29:07.623630 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:29:07.623579    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70ff87e0-8bf9-4f4a-b32b-242baf27ceaa-webhook-cert podName:70ff87e0-8bf9-4f4a-b32b-242baf27ceaa nodeName:}" failed. No retries permitted until 2026-09-11 17:29:08.123563374 +0000 UTC m=+515.927735776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/70ff87e0-8bf9-4f4a-b32b-242baf27ceaa-webhook-cert") pod "maas-controller-6ddbd79556-qd9nn" (UID: "70ff87e0-8bf9-4f4a-b32b-242baf27ceaa") : secret "maas-controller-webhook-cert" not found
Sep 11 17:29:07.634487 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:07.634430    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5p5r\" (UniqueName: \"kubernetes.io/projected/70ff87e0-8bf9-4f4a-b32b-242baf27ceaa-kube-api-access-b5p5r\") pod \"maas-controller-6ddbd79556-qd9nn\" (UID: \"70ff87e0-8bf9-4f4a-b32b-242baf27ceaa\") " pod="opendatahub/maas-controller-6ddbd79556-qd9nn"
Sep 11 17:29:08.028081 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:08.028042    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e110b6ea-4472-4a07-973e-40c599278432-webhook-cert\") pod \"maas-controller-5d6579744d-92qcp\" (UID: \"e110b6ea-4472-4a07-973e-40c599278432\") " pod="opendatahub/maas-controller-5d6579744d-92qcp"
Sep 11 17:29:08.028262 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:08.028120    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e110b6ea-4472-4a07-973e-40c599278432-metrics-certs\") pod \"maas-controller-5d6579744d-92qcp\" (UID: \"e110b6ea-4472-4a07-973e-40c599278432\") " pod="opendatahub/maas-controller-5d6579744d-92qcp"
Sep 11 17:29:08.030598 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:08.030569    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e110b6ea-4472-4a07-973e-40c599278432-webhook-cert\") pod \"maas-controller-5d6579744d-92qcp\" (UID: \"e110b6ea-4472-4a07-973e-40c599278432\") " pod="opendatahub/maas-controller-5d6579744d-92qcp"
Sep 11 17:29:08.030711 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:08.030657    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e110b6ea-4472-4a07-973e-40c599278432-metrics-certs\") pod \"maas-controller-5d6579744d-92qcp\" (UID: \"e110b6ea-4472-4a07-973e-40c599278432\") " pod="opendatahub/maas-controller-5d6579744d-92qcp"
Sep 11 17:29:08.129120 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:08.129076    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70ff87e0-8bf9-4f4a-b32b-242baf27ceaa-metrics-certs\") pod \"maas-controller-6ddbd79556-qd9nn\" (UID: \"70ff87e0-8bf9-4f4a-b32b-242baf27ceaa\") " pod="opendatahub/maas-controller-6ddbd79556-qd9nn"
Sep 11 17:29:08.129120 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:08.129129    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70ff87e0-8bf9-4f4a-b32b-242baf27ceaa-webhook-cert\") pod \"maas-controller-6ddbd79556-qd9nn\" (UID: \"70ff87e0-8bf9-4f4a-b32b-242baf27ceaa\") " pod="opendatahub/maas-controller-6ddbd79556-qd9nn"
Sep 11 17:29:08.131678 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:08.131645    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70ff87e0-8bf9-4f4a-b32b-242baf27ceaa-metrics-certs\") pod \"maas-controller-6ddbd79556-qd9nn\" (UID: \"70ff87e0-8bf9-4f4a-b32b-242baf27ceaa\") " pod="opendatahub/maas-controller-6ddbd79556-qd9nn"
Sep 11 17:29:08.131795 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:08.131645    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70ff87e0-8bf9-4f4a-b32b-242baf27ceaa-webhook-cert\") pod \"maas-controller-6ddbd79556-qd9nn\" (UID: \"70ff87e0-8bf9-4f4a-b32b-242baf27ceaa\") " pod="opendatahub/maas-controller-6ddbd79556-qd9nn"
Sep 11 17:29:08.178135 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:08.178093    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d6579744d-92qcp"
Sep 11 17:29:08.327082 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:08.327047    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ddbd79556-qd9nn"
Sep 11 17:29:08.447632 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:08.447605    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d6579744d-92qcp"]
Sep 11 17:29:08.450044 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:29:08.450009    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode110b6ea_4472_4a07_973e_40c599278432.slice/crio-14d5e3fce38654f2210c95bb281cae2633c459cd6b9ef28662ea801af489246e WatchSource:0}: Error finding container 14d5e3fce38654f2210c95bb281cae2633c459cd6b9ef28662ea801af489246e: Status 404 returned error can't find the container with id 14d5e3fce38654f2210c95bb281cae2633c459cd6b9ef28662ea801af489246e
Sep 11 17:29:08.594323 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:08.594267    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6ddbd79556-qd9nn"]
Sep 11 17:29:08.596548 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:29:08.596518    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ff87e0_8bf9_4f4a_b32b_242baf27ceaa.slice/crio-c21cd95163122036f4c5dff3cf7978f1f269b39f7fc4ca87494f0f446f6dc586 WatchSource:0}: Error finding container c21cd95163122036f4c5dff3cf7978f1f269b39f7fc4ca87494f0f446f6dc586: Status 404 returned error can't find the container with id c21cd95163122036f4c5dff3cf7978f1f269b39f7fc4ca87494f0f446f6dc586
Sep 11 17:29:09.092673 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:09.092638    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d6579744d-92qcp" event={"ID":"e110b6ea-4472-4a07-973e-40c599278432","Type":"ContainerStarted","Data":"14d5e3fce38654f2210c95bb281cae2633c459cd6b9ef28662ea801af489246e"}
Sep 11 17:29:09.095889 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:09.093795    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ddbd79556-qd9nn" event={"ID":"70ff87e0-8bf9-4f4a-b32b-242baf27ceaa","Type":"ContainerStarted","Data":"c21cd95163122036f4c5dff3cf7978f1f269b39f7fc4ca87494f0f446f6dc586"}
Sep 11 17:29:12.116085 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:12.116044    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ddbd79556-qd9nn" event={"ID":"70ff87e0-8bf9-4f4a-b32b-242baf27ceaa","Type":"ContainerStarted","Data":"c4ae291c4090858ff53a06d4190170cec6c8fee52819addf85f516e88e3c0df7"}
Sep 11 17:29:12.116542 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:12.116122    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6ddbd79556-qd9nn"
Sep 11 17:29:12.118041 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:12.118008    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d6579744d-92qcp" event={"ID":"e110b6ea-4472-4a07-973e-40c599278432","Type":"ContainerStarted","Data":"cbc40b29bf1f5055aa322e8377b7913b6ac0c75550b6e9e76822325797ce2027"}
Sep 11 17:29:12.118226 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:12.118208    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5d6579744d-92qcp"
Sep 11 17:29:12.144961 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:12.144877    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6ddbd79556-qd9nn" podStartSLOduration=2.297178915 podStartE2EDuration="5.144853624s" totalImagesPullingTime="2.847674709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:29:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:29:12.142281276 +0000 UTC m=+519.946453695" watchObservedRunningTime="2026-09-11 17:29:12.144853624 +0000 UTC m=+519.949026045"
Sep 11 17:29:12.170693 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:12.170641    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5d6579744d-92qcp" podStartSLOduration=2.183092306 podStartE2EDuration="5.170619347s" totalImagesPullingTime="2.987527041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:29:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:29:12.168305351 +0000 UTC m=+519.972477773" watchObservedRunningTime="2026-09-11 17:29:12.170619347 +0000 UTC m=+519.974791768"
Sep 11 17:29:23.125141 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:23.125109    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5d6579744d-92qcp"
Sep 11 17:29:23.125607 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:23.125184    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6ddbd79556-qd9nn"
Sep 11 17:29:23.189331 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:23.189297    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d6579744d-92qcp"]
Sep 11 17:29:23.189569 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:23.189544    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5d6579744d-92qcp" podUID="e110b6ea-4472-4a07-973e-40c599278432" containerName="manager" containerID="cri-o://cbc40b29bf1f5055aa322e8377b7913b6ac0c75550b6e9e76822325797ce2027" gracePeriod=30
Sep 11 17:29:23.435048 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:23.435020    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d6579744d-92qcp"
Sep 11 17:29:23.469812 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:23.469783    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e110b6ea-4472-4a07-973e-40c599278432-metrics-certs\") pod \"e110b6ea-4472-4a07-973e-40c599278432\" (UID: \"e110b6ea-4472-4a07-973e-40c599278432\") "
Sep 11 17:29:23.470010 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:23.469820    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e110b6ea-4472-4a07-973e-40c599278432-webhook-cert\") pod \"e110b6ea-4472-4a07-973e-40c599278432\" (UID: \"e110b6ea-4472-4a07-973e-40c599278432\") "
Sep 11 17:29:23.470010 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:23.469859    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkwtp\" (UniqueName: \"kubernetes.io/projected/e110b6ea-4472-4a07-973e-40c599278432-kube-api-access-hkwtp\") pod \"e110b6ea-4472-4a07-973e-40c599278432\" (UID: \"e110b6ea-4472-4a07-973e-40c599278432\") "
Sep 11 17:29:23.472077 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:23.472044    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e110b6ea-4472-4a07-973e-40c599278432-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e110b6ea-4472-4a07-973e-40c599278432" (UID: "e110b6ea-4472-4a07-973e-40c599278432"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:29:23.472185 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:23.472091    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e110b6ea-4472-4a07-973e-40c599278432-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e110b6ea-4472-4a07-973e-40c599278432" (UID: "e110b6ea-4472-4a07-973e-40c599278432"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:29:23.472185 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:23.472138    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e110b6ea-4472-4a07-973e-40c599278432-kube-api-access-hkwtp" (OuterVolumeSpecName: "kube-api-access-hkwtp") pod "e110b6ea-4472-4a07-973e-40c599278432" (UID: "e110b6ea-4472-4a07-973e-40c599278432"). InnerVolumeSpecName "kube-api-access-hkwtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:29:23.570886 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:23.570854    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e110b6ea-4472-4a07-973e-40c599278432-metrics-certs\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:29:23.570886 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:23.570880    2560 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e110b6ea-4472-4a07-973e-40c599278432-webhook-cert\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:29:23.570886 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:23.570890    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkwtp\" (UniqueName: \"kubernetes.io/projected/e110b6ea-4472-4a07-973e-40c599278432-kube-api-access-hkwtp\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:29:24.174475 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:24.174442    2560 generic.go:356] "Generic (PLEG): container finished" podID="e110b6ea-4472-4a07-973e-40c599278432" containerID="cbc40b29bf1f5055aa322e8377b7913b6ac0c75550b6e9e76822325797ce2027" exitCode=0
Sep 11 17:29:24.174862 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:24.174482    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d6579744d-92qcp"
Sep 11 17:29:24.174862 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:24.174521    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d6579744d-92qcp" event={"ID":"e110b6ea-4472-4a07-973e-40c599278432","Type":"ContainerDied","Data":"cbc40b29bf1f5055aa322e8377b7913b6ac0c75550b6e9e76822325797ce2027"}
Sep 11 17:29:24.174862 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:24.174571    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d6579744d-92qcp" event={"ID":"e110b6ea-4472-4a07-973e-40c599278432","Type":"ContainerDied","Data":"14d5e3fce38654f2210c95bb281cae2633c459cd6b9ef28662ea801af489246e"}
Sep 11 17:29:24.174862 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:24.174592    2560 scope.go:117] "RemoveContainer" containerID="cbc40b29bf1f5055aa322e8377b7913b6ac0c75550b6e9e76822325797ce2027"
Sep 11 17:29:24.183282 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:24.183267    2560 scope.go:117] "RemoveContainer" containerID="cbc40b29bf1f5055aa322e8377b7913b6ac0c75550b6e9e76822325797ce2027"
Sep 11 17:29:24.183547 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:29:24.183530    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbc40b29bf1f5055aa322e8377b7913b6ac0c75550b6e9e76822325797ce2027\": container with ID starting with cbc40b29bf1f5055aa322e8377b7913b6ac0c75550b6e9e76822325797ce2027 not found: ID does not exist" containerID="cbc40b29bf1f5055aa322e8377b7913b6ac0c75550b6e9e76822325797ce2027"
Sep 11 17:29:24.183584 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:24.183556    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbc40b29bf1f5055aa322e8377b7913b6ac0c75550b6e9e76822325797ce2027"} err="failed to get container status \"cbc40b29bf1f5055aa322e8377b7913b6ac0c75550b6e9e76822325797ce2027\": rpc error: code = NotFound desc = could not find container \"cbc40b29bf1f5055aa322e8377b7913b6ac0c75550b6e9e76822325797ce2027\": container with ID starting with cbc40b29bf1f5055aa322e8377b7913b6ac0c75550b6e9e76822325797ce2027 not found: ID does not exist"
Sep 11 17:29:24.195138 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:24.195113    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d6579744d-92qcp"]
Sep 11 17:29:24.203732 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:24.200502    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5d6579744d-92qcp"]
Sep 11 17:29:24.714476 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:24.714442    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e110b6ea-4472-4a07-973e-40c599278432" path="/var/lib/kubelet/pods/e110b6ea-4472-4a07-973e-40c599278432/volumes"
Sep 11 17:29:42.479207 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:42.479167    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl"]
Sep 11 17:29:42.479704 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:42.479688    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e110b6ea-4472-4a07-973e-40c599278432" containerName="manager"
Sep 11 17:29:42.479768 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:42.479706    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e110b6ea-4472-4a07-973e-40c599278432" containerName="manager"
Sep 11 17:29:42.479802 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:42.479792    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e110b6ea-4472-4a07-973e-40c599278432" containerName="manager"
Sep 11 17:29:42.484392 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:42.484374    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl"
Sep 11 17:29:42.486824 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:42.486799    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 17:29:42.487146 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:42.487065    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 17:29:42.487146 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:42.487083    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nwfbn\""
Sep 11 17:29:42.493959 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:42.493925    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl"]
Sep 11 17:29:42.636751 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:42.636718    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/397ddbb9-61bf-4952-8bc8-2763882ba7e2-metrics-certs\") pod \"maas-api-7bf4f9b548-rv2sl\" (UID: \"397ddbb9-61bf-4952-8bc8-2763882ba7e2\") " pod="odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl"
Sep 11 17:29:42.636751 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:42.636753    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4bzd\" (UniqueName: \"kubernetes.io/projected/397ddbb9-61bf-4952-8bc8-2763882ba7e2-kube-api-access-b4bzd\") pod \"maas-api-7bf4f9b548-rv2sl\" (UID: \"397ddbb9-61bf-4952-8bc8-2763882ba7e2\") " pod="odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl"
Sep 11 17:29:42.637022 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:42.636776    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/397ddbb9-61bf-4952-8bc8-2763882ba7e2-maas-api-tls\") pod \"maas-api-7bf4f9b548-rv2sl\" (UID: \"397ddbb9-61bf-4952-8bc8-2763882ba7e2\") " pod="odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl"
Sep 11 17:29:42.637022 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:42.636881    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/397ddbb9-61bf-4952-8bc8-2763882ba7e2-openshift-service-ca\") pod \"maas-api-7bf4f9b548-rv2sl\" (UID: \"397ddbb9-61bf-4952-8bc8-2763882ba7e2\") " pod="odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl"
Sep 11 17:29:42.737460 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:42.737370    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/397ddbb9-61bf-4952-8bc8-2763882ba7e2-metrics-certs\") pod \"maas-api-7bf4f9b548-rv2sl\" (UID: \"397ddbb9-61bf-4952-8bc8-2763882ba7e2\") " pod="odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl"
Sep 11 17:29:42.737677 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:29:42.737533    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 11 17:29:42.737812 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:42.737661    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4bzd\" (UniqueName: \"kubernetes.io/projected/397ddbb9-61bf-4952-8bc8-2763882ba7e2-kube-api-access-b4bzd\") pod \"maas-api-7bf4f9b548-rv2sl\" (UID: \"397ddbb9-61bf-4952-8bc8-2763882ba7e2\") " pod="odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl"
Sep 11 17:29:42.737812 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:29:42.737764    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/397ddbb9-61bf-4952-8bc8-2763882ba7e2-metrics-certs podName:397ddbb9-61bf-4952-8bc8-2763882ba7e2 nodeName:}" failed. No retries permitted until 2026-09-11 17:29:43.237741442 +0000 UTC m=+551.041913842 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/397ddbb9-61bf-4952-8bc8-2763882ba7e2-metrics-certs") pod "maas-api-7bf4f9b548-rv2sl" (UID: "397ddbb9-61bf-4952-8bc8-2763882ba7e2") : secret "maas-api-metrics-tls" not found
Sep 11 17:29:42.737974 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:42.737812    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/397ddbb9-61bf-4952-8bc8-2763882ba7e2-maas-api-tls\") pod \"maas-api-7bf4f9b548-rv2sl\" (UID: \"397ddbb9-61bf-4952-8bc8-2763882ba7e2\") " pod="odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl"
Sep 11 17:29:42.737974 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:42.737888    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/397ddbb9-61bf-4952-8bc8-2763882ba7e2-openshift-service-ca\") pod \"maas-api-7bf4f9b548-rv2sl\" (UID: \"397ddbb9-61bf-4952-8bc8-2763882ba7e2\") " pod="odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl"
Sep 11 17:29:42.739026 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:42.738999    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/397ddbb9-61bf-4952-8bc8-2763882ba7e2-openshift-service-ca\") pod \"maas-api-7bf4f9b548-rv2sl\" (UID: \"397ddbb9-61bf-4952-8bc8-2763882ba7e2\") " pod="odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl"
Sep 11 17:29:42.741310 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:42.741288    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/397ddbb9-61bf-4952-8bc8-2763882ba7e2-maas-api-tls\") pod \"maas-api-7bf4f9b548-rv2sl\" (UID: \"397ddbb9-61bf-4952-8bc8-2763882ba7e2\") " pod="odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl"
Sep 11 17:29:42.749081 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:42.748854    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4bzd\" (UniqueName: \"kubernetes.io/projected/397ddbb9-61bf-4952-8bc8-2763882ba7e2-kube-api-access-b4bzd\") pod \"maas-api-7bf4f9b548-rv2sl\" (UID: \"397ddbb9-61bf-4952-8bc8-2763882ba7e2\") " pod="odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl"
Sep 11 17:29:42.925353 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:42.925310    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-bq6mr"]
Sep 11 17:29:42.929016 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:42.928990    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bq6mr"
Sep 11 17:29:42.931361 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:42.931339    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 11 17:29:42.941119 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:42.941089    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-bq6mr"]
Sep 11 17:29:42.958438 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:42.958407    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-zn9cq"]
Sep 11 17:29:42.961620 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:42.961599    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-zn9cq"
Sep 11 17:29:42.964495 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:42.964470    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-xftq2\""
Sep 11 17:29:42.972840 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:42.972815    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-zn9cq"]
Sep 11 17:29:43.040209 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:43.040064    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhzzc\" (UniqueName: \"kubernetes.io/projected/521643fa-f596-493a-89e3-5b0582f6ed0b-kube-api-access-dhzzc\") pod \"payload-processing-5b54b7fbb7-zn9cq\" (UID: \"521643fa-f596-493a-89e3-5b0582f6ed0b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zn9cq"
Sep 11 17:29:43.040209 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:43.040100    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521643fa-f596-493a-89e3-5b0582f6ed0b-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-zn9cq\" (UID: \"521643fa-f596-493a-89e3-5b0582f6ed0b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zn9cq"
Sep 11 17:29:43.040547 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:43.040230    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xgwn\" (UniqueName: \"kubernetes.io/projected/f6516a4b-de1c-4d62-80d1-7dec6d90eae3-kube-api-access-2xgwn\") pod \"payload-pre-processing-59fbc47bf8-bq6mr\" (UID: \"f6516a4b-de1c-4d62-80d1-7dec6d90eae3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bq6mr"
Sep 11 17:29:43.040547 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:43.040329    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6516a4b-de1c-4d62-80d1-7dec6d90eae3-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-bq6mr\" (UID: \"f6516a4b-de1c-4d62-80d1-7dec6d90eae3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bq6mr"
Sep 11 17:29:43.141665 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:43.141630    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xgwn\" (UniqueName: \"kubernetes.io/projected/f6516a4b-de1c-4d62-80d1-7dec6d90eae3-kube-api-access-2xgwn\") pod \"payload-pre-processing-59fbc47bf8-bq6mr\" (UID: \"f6516a4b-de1c-4d62-80d1-7dec6d90eae3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bq6mr"
Sep 11 17:29:43.141868 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:43.141713    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6516a4b-de1c-4d62-80d1-7dec6d90eae3-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-bq6mr\" (UID: \"f6516a4b-de1c-4d62-80d1-7dec6d90eae3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bq6mr"
Sep 11 17:29:43.141868 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:43.141751    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhzzc\" (UniqueName: \"kubernetes.io/projected/521643fa-f596-493a-89e3-5b0582f6ed0b-kube-api-access-dhzzc\") pod \"payload-processing-5b54b7fbb7-zn9cq\" (UID: \"521643fa-f596-493a-89e3-5b0582f6ed0b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zn9cq"
Sep 11 17:29:43.141868 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:43.141775    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521643fa-f596-493a-89e3-5b0582f6ed0b-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-zn9cq\" (UID: \"521643fa-f596-493a-89e3-5b0582f6ed0b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zn9cq"
Sep 11 17:29:43.142530 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:43.142511    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521643fa-f596-493a-89e3-5b0582f6ed0b-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-zn9cq\" (UID: \"521643fa-f596-493a-89e3-5b0582f6ed0b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zn9cq"
Sep 11 17:29:43.142715 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:43.142692    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6516a4b-de1c-4d62-80d1-7dec6d90eae3-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-bq6mr\" (UID: \"f6516a4b-de1c-4d62-80d1-7dec6d90eae3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bq6mr"
Sep 11 17:29:43.151688 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:43.151662    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xgwn\" (UniqueName: \"kubernetes.io/projected/f6516a4b-de1c-4d62-80d1-7dec6d90eae3-kube-api-access-2xgwn\") pod \"payload-pre-processing-59fbc47bf8-bq6mr\" (UID: \"f6516a4b-de1c-4d62-80d1-7dec6d90eae3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bq6mr"
Sep 11 17:29:43.151846 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:43.151828    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhzzc\" (UniqueName: \"kubernetes.io/projected/521643fa-f596-493a-89e3-5b0582f6ed0b-kube-api-access-dhzzc\") pod \"payload-processing-5b54b7fbb7-zn9cq\" (UID: \"521643fa-f596-493a-89e3-5b0582f6ed0b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zn9cq"
Sep 11 17:29:43.240084 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:43.240044    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bq6mr"
Sep 11 17:29:43.242443 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:43.242419    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/397ddbb9-61bf-4952-8bc8-2763882ba7e2-metrics-certs\") pod \"maas-api-7bf4f9b548-rv2sl\" (UID: \"397ddbb9-61bf-4952-8bc8-2763882ba7e2\") " pod="odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl"
Sep 11 17:29:43.244908 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:43.244879    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/397ddbb9-61bf-4952-8bc8-2763882ba7e2-metrics-certs\") pod \"maas-api-7bf4f9b548-rv2sl\" (UID: \"397ddbb9-61bf-4952-8bc8-2763882ba7e2\") " pod="odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl"
Sep 11 17:29:43.272359 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:43.272332    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-zn9cq"
Sep 11 17:29:43.397766 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:43.397732    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl"
Sep 11 17:29:43.525470 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:43.525447    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-bq6mr"]
Sep 11 17:29:43.527589 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:29:43.527565    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6516a4b_de1c_4d62_80d1_7dec6d90eae3.slice/crio-ca18cf0f27e82caaa94f53b9667e507620ec2025475ce3563848eb84b567c950 WatchSource:0}: Error finding container ca18cf0f27e82caaa94f53b9667e507620ec2025475ce3563848eb84b567c950: Status 404 returned error can't find the container with id ca18cf0f27e82caaa94f53b9667e507620ec2025475ce3563848eb84b567c950
Sep 11 17:29:43.571855 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:43.571829    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-zn9cq"]
Sep 11 17:29:43.668595 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:43.668564    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl"]
Sep 11 17:29:43.671293 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:29:43.671260    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod397ddbb9_61bf_4952_8bc8_2763882ba7e2.slice/crio-2b4c735516613120cb82b242f5e5afb11d9e8755aa00d42e3a1823e31658e258 WatchSource:0}: Error finding container 2b4c735516613120cb82b242f5e5afb11d9e8755aa00d42e3a1823e31658e258: Status 404 returned error can't find the container with id 2b4c735516613120cb82b242f5e5afb11d9e8755aa00d42e3a1823e31658e258
Sep 11 17:29:44.276883 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:44.276825    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bq6mr" event={"ID":"f6516a4b-de1c-4d62-80d1-7dec6d90eae3","Type":"ContainerStarted","Data":"ca18cf0f27e82caaa94f53b9667e507620ec2025475ce3563848eb84b567c950"}
Sep 11 17:29:44.278742 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:44.278707    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-zn9cq" event={"ID":"521643fa-f596-493a-89e3-5b0582f6ed0b","Type":"ContainerStarted","Data":"eec139e9e7985ec28060cd923bbe917ec02b578f44b9f4bfd96910a552f9bfe4"}
Sep 11 17:29:44.280361 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:44.280288    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl" event={"ID":"397ddbb9-61bf-4952-8bc8-2763882ba7e2","Type":"ContainerStarted","Data":"2b4c735516613120cb82b242f5e5afb11d9e8755aa00d42e3a1823e31658e258"}
Sep 11 17:29:47.299324 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:47.299284    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl" event={"ID":"397ddbb9-61bf-4952-8bc8-2763882ba7e2","Type":"ContainerStarted","Data":"0af2473dc5a086b54c5f12584cbcac8e9b6d5b2ecf66a1f3ff112f4b8d01cfd0"}
Sep 11 17:29:47.299819 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:47.299393    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl"
Sep 11 17:29:47.301158 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:47.301130    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bq6mr" event={"ID":"f6516a4b-de1c-4d62-80d1-7dec6d90eae3","Type":"ContainerStarted","Data":"4540d32cd89cc1ede9af575afe8727a4d18c537aadaca24a35cb2562e5726d36"}
Sep 11 17:29:47.301366 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:47.301339    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bq6mr"
Sep 11 17:29:47.302950 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:47.302922    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-zn9cq" event={"ID":"521643fa-f596-493a-89e3-5b0582f6ed0b","Type":"ContainerStarted","Data":"eaa96840d2e770d49c07f34200566b20ab2ab5773628be3ac4c0105962c90d57"}
Sep 11 17:29:47.303052 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:47.302978    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-zn9cq"
Sep 11 17:29:47.317442 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:47.317393    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl" podStartSLOduration=2.442358538 podStartE2EDuration="5.317371224s" totalImagesPullingTime="2.875012686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:29:47.316207201 +0000 UTC m=+555.120379621" watchObservedRunningTime="2026-09-11 17:29:47.317371224 +0000 UTC m=+555.121543644"
Sep 11 17:29:47.334562 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:47.334515    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-zn9cq" podStartSLOduration=2.303354883 podStartE2EDuration="5.334499114s" totalImagesPullingTime="3.031144231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:29:47.333001507 +0000 UTC m=+555.137173927" watchObservedRunningTime="2026-09-11 17:29:47.334499114 +0000 UTC m=+555.138671532"
Sep 11 17:29:47.351672 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:47.351615    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bq6mr" podStartSLOduration=2.263731178 podStartE2EDuration="5.351594525s" totalImagesPullingTime="3.087863347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:29:47.349838034 +0000 UTC m=+555.154010453" watchObservedRunningTime="2026-09-11 17:29:47.351594525 +0000 UTC m=+555.155766946"
Sep 11 17:29:53.313187 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:53.313160    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl"
Sep 11 17:29:58.310976 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:58.310939    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-zn9cq"
Sep 11 17:29:58.311459 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:29:58.311010    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-bq6mr"
Sep 11 17:30:16.124663 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:16.124628    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8"]
Sep 11 17:30:16.130200 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:16.130176    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8"
Sep 11 17:30:16.133175 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:16.133147    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 17:30:16.133329 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:16.133222    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-b865k\""
Sep 11 17:30:16.133941 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:16.133918    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 11 17:30:16.134057 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:16.133960    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 17:30:16.141085 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:16.141035    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8"]
Sep 11 17:30:16.242214 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:16.242174    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21de8bc8-5de8-48a3-9b9e-47cbdff8f669-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdsb8\" (UID: \"21de8bc8-5de8-48a3-9b9e-47cbdff8f669\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8"
Sep 11 17:30:16.242214 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:16.242212    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21de8bc8-5de8-48a3-9b9e-47cbdff8f669-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdsb8\" (UID: \"21de8bc8-5de8-48a3-9b9e-47cbdff8f669\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8"
Sep 11 17:30:16.242419 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:16.242236    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21de8bc8-5de8-48a3-9b9e-47cbdff8f669-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdsb8\" (UID: \"21de8bc8-5de8-48a3-9b9e-47cbdff8f669\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8"
Sep 11 17:30:16.242419 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:16.242282    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21de8bc8-5de8-48a3-9b9e-47cbdff8f669-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdsb8\" (UID: \"21de8bc8-5de8-48a3-9b9e-47cbdff8f669\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8"
Sep 11 17:30:16.242419 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:16.242333    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21de8bc8-5de8-48a3-9b9e-47cbdff8f669-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdsb8\" (UID: \"21de8bc8-5de8-48a3-9b9e-47cbdff8f669\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8"
Sep 11 17:30:16.242419 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:16.242349    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxlp6\" (UniqueName: \"kubernetes.io/projected/21de8bc8-5de8-48a3-9b9e-47cbdff8f669-kube-api-access-dxlp6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdsb8\" (UID: \"21de8bc8-5de8-48a3-9b9e-47cbdff8f669\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8"
Sep 11 17:30:16.343401 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:16.343361    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21de8bc8-5de8-48a3-9b9e-47cbdff8f669-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdsb8\" (UID: \"21de8bc8-5de8-48a3-9b9e-47cbdff8f669\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8"
Sep 11 17:30:16.343401 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:16.343402    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21de8bc8-5de8-48a3-9b9e-47cbdff8f669-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdsb8\" (UID: \"21de8bc8-5de8-48a3-9b9e-47cbdff8f669\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8"
Sep 11 17:30:16.343617 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:16.343428    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21de8bc8-5de8-48a3-9b9e-47cbdff8f669-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdsb8\" (UID: \"21de8bc8-5de8-48a3-9b9e-47cbdff8f669\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8"
Sep 11 17:30:16.343617 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:16.343454    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21de8bc8-5de8-48a3-9b9e-47cbdff8f669-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdsb8\" (UID: \"21de8bc8-5de8-48a3-9b9e-47cbdff8f669\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8"
Sep 11 17:30:16.343617 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:16.343499    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21de8bc8-5de8-48a3-9b9e-47cbdff8f669-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdsb8\" (UID: \"21de8bc8-5de8-48a3-9b9e-47cbdff8f669\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8"
Sep 11 17:30:16.343617 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:16.343517    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxlp6\" (UniqueName: \"kubernetes.io/projected/21de8bc8-5de8-48a3-9b9e-47cbdff8f669-kube-api-access-dxlp6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdsb8\" (UID: \"21de8bc8-5de8-48a3-9b9e-47cbdff8f669\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8"
Sep 11 17:30:16.343840 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:16.343819    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21de8bc8-5de8-48a3-9b9e-47cbdff8f669-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdsb8\" (UID: \"21de8bc8-5de8-48a3-9b9e-47cbdff8f669\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8"
Sep 11 17:30:16.343939 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:16.343885    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21de8bc8-5de8-48a3-9b9e-47cbdff8f669-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdsb8\" (UID: \"21de8bc8-5de8-48a3-9b9e-47cbdff8f669\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8"
Sep 11 17:30:16.344008 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:16.343939    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21de8bc8-5de8-48a3-9b9e-47cbdff8f669-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdsb8\" (UID: \"21de8bc8-5de8-48a3-9b9e-47cbdff8f669\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8"
Sep 11 17:30:16.345794 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:16.345776    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21de8bc8-5de8-48a3-9b9e-47cbdff8f669-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdsb8\" (UID: \"21de8bc8-5de8-48a3-9b9e-47cbdff8f669\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8"
Sep 11 17:30:16.346000 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:16.345984    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21de8bc8-5de8-48a3-9b9e-47cbdff8f669-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdsb8\" (UID: \"21de8bc8-5de8-48a3-9b9e-47cbdff8f669\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8"
Sep 11 17:30:16.355071 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:16.355042    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxlp6\" (UniqueName: \"kubernetes.io/projected/21de8bc8-5de8-48a3-9b9e-47cbdff8f669-kube-api-access-dxlp6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fdsb8\" (UID: \"21de8bc8-5de8-48a3-9b9e-47cbdff8f669\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8"
Sep 11 17:30:16.441734 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:16.441653    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8"
Sep 11 17:30:16.715732 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:30:16.715690    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21de8bc8_5de8_48a3_9b9e_47cbdff8f669.slice/crio-08140dbab283f34106985275006c5dff41f53b263706982029a466429d18c081 WatchSource:0}: Error finding container 08140dbab283f34106985275006c5dff41f53b263706982029a466429d18c081: Status 404 returned error can't find the container with id 08140dbab283f34106985275006c5dff41f53b263706982029a466429d18c081
Sep 11 17:30:16.715944 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:16.715923    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8"]
Sep 11 17:30:17.447062 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:17.447024    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8" event={"ID":"21de8bc8-5de8-48a3-9b9e-47cbdff8f669","Type":"ContainerStarted","Data":"08140dbab283f34106985275006c5dff41f53b263706982029a466429d18c081"}
Sep 11 17:30:18.452681 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:18.452640    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8" event={"ID":"21de8bc8-5de8-48a3-9b9e-47cbdff8f669","Type":"ContainerStarted","Data":"140f936831ec117fa78619ba1e1a9a625f6a35726c09a07c58c39afef70cd67a"}
Sep 11 17:30:18.471680 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:18.471626    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8" podStartSLOduration=1.502528203 podStartE2EDuration="2.471609326s" totalImagesPullingTime="969.081123ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:30:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:30:18.470437168 +0000 UTC m=+586.274609586" watchObservedRunningTime="2026-09-11 17:30:18.471609326 +0000 UTC m=+586.275781745"
Sep 11 17:30:26.442861 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:26.442828    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8"
Sep 11 17:30:26.443482 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:26.442945    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8"
Sep 11 17:30:26.455935 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:26.455908    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8"
Sep 11 17:30:26.500288 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:26.500242    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fdsb8"
Sep 11 17:30:32.595939 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:32.595884    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6fcjs_477cac01-ab9e-4441-b07d-b7e53401c616/console-operator/1.log"
Sep 11 17:30:32.597396 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:32.597374    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6fcjs_477cac01-ab9e-4441-b07d-b7e53401c616/console-operator/1.log"
Sep 11 17:30:38.126935 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:38.126888    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw"]
Sep 11 17:30:38.129936 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:38.129917    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw"
Sep 11 17:30:38.132473 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:38.132446    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 11 17:30:38.140781 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:38.140758    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw"]
Sep 11 17:30:38.248561 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:38.248529    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35fa4045-cbed-45ab-9fee-a4f5a014f7eb-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jbhdw\" (UID: \"35fa4045-cbed-45ab-9fee-a4f5a014f7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw"
Sep 11 17:30:38.248731 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:38.248640    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35fa4045-cbed-45ab-9fee-a4f5a014f7eb-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jbhdw\" (UID: \"35fa4045-cbed-45ab-9fee-a4f5a014f7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw"
Sep 11 17:30:38.248731 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:38.248663    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35fa4045-cbed-45ab-9fee-a4f5a014f7eb-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jbhdw\" (UID: \"35fa4045-cbed-45ab-9fee-a4f5a014f7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw"
Sep 11 17:30:38.248731 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:38.248719    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq4xn\" (UniqueName: \"kubernetes.io/projected/35fa4045-cbed-45ab-9fee-a4f5a014f7eb-kube-api-access-fq4xn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jbhdw\" (UID: \"35fa4045-cbed-45ab-9fee-a4f5a014f7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw"
Sep 11 17:30:38.248852 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:38.248743    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35fa4045-cbed-45ab-9fee-a4f5a014f7eb-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jbhdw\" (UID: \"35fa4045-cbed-45ab-9fee-a4f5a014f7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw"
Sep 11 17:30:38.248852 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:38.248773    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35fa4045-cbed-45ab-9fee-a4f5a014f7eb-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jbhdw\" (UID: \"35fa4045-cbed-45ab-9fee-a4f5a014f7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw"
Sep 11 17:30:38.350264 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:38.350226    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35fa4045-cbed-45ab-9fee-a4f5a014f7eb-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jbhdw\" (UID: \"35fa4045-cbed-45ab-9fee-a4f5a014f7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw"
Sep 11 17:30:38.350264 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:38.350265    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35fa4045-cbed-45ab-9fee-a4f5a014f7eb-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jbhdw\" (UID: \"35fa4045-cbed-45ab-9fee-a4f5a014f7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw"
Sep 11 17:30:38.350465 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:38.350299    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq4xn\" (UniqueName: \"kubernetes.io/projected/35fa4045-cbed-45ab-9fee-a4f5a014f7eb-kube-api-access-fq4xn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jbhdw\" (UID: \"35fa4045-cbed-45ab-9fee-a4f5a014f7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw"
Sep 11 17:30:38.350465 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:38.350319    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35fa4045-cbed-45ab-9fee-a4f5a014f7eb-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jbhdw\" (UID: \"35fa4045-cbed-45ab-9fee-a4f5a014f7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw"
Sep 11 17:30:38.350465 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:38.350346    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35fa4045-cbed-45ab-9fee-a4f5a014f7eb-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jbhdw\" (UID: \"35fa4045-cbed-45ab-9fee-a4f5a014f7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw"
Sep 11 17:30:38.350465 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:38.350376    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35fa4045-cbed-45ab-9fee-a4f5a014f7eb-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jbhdw\" (UID: \"35fa4045-cbed-45ab-9fee-a4f5a014f7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw"
Sep 11 17:30:38.350756 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:38.350730    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35fa4045-cbed-45ab-9fee-a4f5a014f7eb-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jbhdw\" (UID: \"35fa4045-cbed-45ab-9fee-a4f5a014f7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw"
Sep 11 17:30:38.350911 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:38.350786    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35fa4045-cbed-45ab-9fee-a4f5a014f7eb-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jbhdw\" (UID: \"35fa4045-cbed-45ab-9fee-a4f5a014f7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw"
Sep 11 17:30:38.350911 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:38.350792    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35fa4045-cbed-45ab-9fee-a4f5a014f7eb-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jbhdw\" (UID: \"35fa4045-cbed-45ab-9fee-a4f5a014f7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw"
Sep 11 17:30:38.352509 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:38.352489    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35fa4045-cbed-45ab-9fee-a4f5a014f7eb-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jbhdw\" (UID: \"35fa4045-cbed-45ab-9fee-a4f5a014f7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw"
Sep 11 17:30:38.352798 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:38.352780    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35fa4045-cbed-45ab-9fee-a4f5a014f7eb-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jbhdw\" (UID: \"35fa4045-cbed-45ab-9fee-a4f5a014f7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw"
Sep 11 17:30:38.357940 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:38.357921    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq4xn\" (UniqueName: \"kubernetes.io/projected/35fa4045-cbed-45ab-9fee-a4f5a014f7eb-kube-api-access-fq4xn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jbhdw\" (UID: \"35fa4045-cbed-45ab-9fee-a4f5a014f7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw"
Sep 11 17:30:38.440870 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:38.440766    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw"
Sep 11 17:30:38.713913 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:30:38.713863    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35fa4045_cbed_45ab_9fee_a4f5a014f7eb.slice/crio-84454b03c022d3f4acc0440033abd409a6bf175b9d428bffcd2af823c5419936 WatchSource:0}: Error finding container 84454b03c022d3f4acc0440033abd409a6bf175b9d428bffcd2af823c5419936: Status 404 returned error can't find the container with id 84454b03c022d3f4acc0440033abd409a6bf175b9d428bffcd2af823c5419936
Sep 11 17:30:38.715450 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:38.715427    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:30:38.716157 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:38.716133    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw"]
Sep 11 17:30:39.555215 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:39.555172    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw" event={"ID":"35fa4045-cbed-45ab-9fee-a4f5a014f7eb","Type":"ContainerStarted","Data":"f63c4611f7b56ac99e55659fd2f5a86c8d1e37def24574a171ce0d091ce53acd"}
Sep 11 17:30:39.555600 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:39.555224    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw" event={"ID":"35fa4045-cbed-45ab-9fee-a4f5a014f7eb","Type":"ContainerStarted","Data":"84454b03c022d3f4acc0440033abd409a6bf175b9d428bffcd2af823c5419936"}
Sep 11 17:30:39.590270 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:39.590216    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw" podStartSLOduration=1.39488304 podStartE2EDuration="1.590197473s" totalImagesPullingTime="195.314433ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:30:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:30:39.587249017 +0000 UTC m=+607.391421436" watchObservedRunningTime="2026-09-11 17:30:39.590197473 +0000 UTC m=+607.394369891"
Sep 11 17:30:48.441453 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:48.441405    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw"
Sep 11 17:30:48.441931 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:48.441475    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw"
Sep 11 17:30:48.453860 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:48.453834    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw"
Sep 11 17:30:48.609029 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:30:48.608999    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jbhdw"
Sep 11 17:32:32.384771 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:32.384733    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm"]
Sep 11 17:32:32.388943 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:32.388918    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm"
Sep 11 17:32:32.391616 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:32.391590    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d92fb3-serving-cert\""
Sep 11 17:32:32.408997 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:32.408967    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm"]
Sep 11 17:32:32.478687 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:32.478652    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e15398c-1504-47de-a6dc-78a584cd17df-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm\" (UID: \"9e15398c-1504-47de-a6dc-78a584cd17df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm"
Sep 11 17:32:32.479042 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:32.479017    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e15398c-1504-47de-a6dc-78a584cd17df-maas-api-tls\") pod \"maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm\" (UID: \"9e15398c-1504-47de-a6dc-78a584cd17df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm"
Sep 11 17:32:32.479203 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:32.479188    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtz8r\" (UniqueName: \"kubernetes.io/projected/9e15398c-1504-47de-a6dc-78a584cd17df-kube-api-access-qtz8r\") pod \"maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm\" (UID: \"9e15398c-1504-47de-a6dc-78a584cd17df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm"
Sep 11 17:32:32.479361 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:32.479347    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e15398c-1504-47de-a6dc-78a584cd17df-metrics-certs\") pod \"maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm\" (UID: \"9e15398c-1504-47de-a6dc-78a584cd17df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm"
Sep 11 17:32:32.580447 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:32.580417    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e15398c-1504-47de-a6dc-78a584cd17df-metrics-certs\") pod \"maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm\" (UID: \"9e15398c-1504-47de-a6dc-78a584cd17df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm"
Sep 11 17:32:32.580638 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:32.580467    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e15398c-1504-47de-a6dc-78a584cd17df-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm\" (UID: \"9e15398c-1504-47de-a6dc-78a584cd17df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm"
Sep 11 17:32:32.580638 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:32.580496    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e15398c-1504-47de-a6dc-78a584cd17df-maas-api-tls\") pod \"maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm\" (UID: \"9e15398c-1504-47de-a6dc-78a584cd17df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm"
Sep 11 17:32:32.580638 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:32.580523    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtz8r\" (UniqueName: \"kubernetes.io/projected/9e15398c-1504-47de-a6dc-78a584cd17df-kube-api-access-qtz8r\") pod \"maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm\" (UID: \"9e15398c-1504-47de-a6dc-78a584cd17df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm"
Sep 11 17:32:32.581338 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:32.581308    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e15398c-1504-47de-a6dc-78a584cd17df-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm\" (UID: \"9e15398c-1504-47de-a6dc-78a584cd17df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm"
Sep 11 17:32:32.583084 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:32.583055    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e15398c-1504-47de-a6dc-78a584cd17df-metrics-certs\") pod \"maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm\" (UID: \"9e15398c-1504-47de-a6dc-78a584cd17df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm"
Sep 11 17:32:32.583316 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:32.583292    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d92fb3-serving-cert\""
Sep 11 17:32:32.589346 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:32.589323    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtz8r\" (UniqueName: \"kubernetes.io/projected/9e15398c-1504-47de-a6dc-78a584cd17df-kube-api-access-qtz8r\") pod \"maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm\" (UID: \"9e15398c-1504-47de-a6dc-78a584cd17df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm"
Sep 11 17:32:32.591471 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:32:32.591449    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-serving-cert: secret "maas-api-e2e-inf-a-d92fb3-serving-cert" not found
Sep 11 17:32:32.591575 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:32:32.591521    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e15398c-1504-47de-a6dc-78a584cd17df-maas-api-tls podName:9e15398c-1504-47de-a6dc-78a584cd17df nodeName:}" failed. No retries permitted until 2026-09-11 17:32:33.091499461 +0000 UTC m=+720.895671864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9e15398c-1504-47de-a6dc-78a584cd17df-maas-api-tls") pod "maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm" (UID: "9e15398c-1504-47de-a6dc-78a584cd17df") : secret "maas-api-e2e-inf-a-d92fb3-serving-cert" not found
Sep 11 17:32:33.076029 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:33.075931    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74"]
Sep 11 17:32:33.079746 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:33.079719    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74"
Sep 11 17:32:33.083321 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:33.083298    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d92fb3-dockercfg-6ksz9\""
Sep 11 17:32:33.083493 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:33.083475    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d92fb3\""
Sep 11 17:32:33.093606 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:33.093580    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74"]
Sep 11 17:32:33.118478 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:33.118453    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm"]
Sep 11 17:32:33.121690 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:33.121667    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm"
Sep 11 17:32:33.140727 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:33.140702    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm"]
Sep 11 17:32:33.187657 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:33.187616    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s7pj\" (UniqueName: \"kubernetes.io/projected/4f561801-e1cb-4712-88a2-6642d0897ce0-kube-api-access-4s7pj\") pod \"payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm\" (UID: \"4f561801-e1cb-4712-88a2-6642d0897ce0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm"
Sep 11 17:32:33.187834 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:33.187746    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9867\" (UniqueName: \"kubernetes.io/projected/0b5a085e-e930-45c6-a9cb-d2ad581aa47c-kube-api-access-c9867\") pod \"payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74\" (UID: \"0b5a085e-e930-45c6-a9cb-d2ad581aa47c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74"
Sep 11 17:32:33.187920 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:33.187843    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f561801-e1cb-4712-88a2-6642d0897ce0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm\" (UID: \"4f561801-e1cb-4712-88a2-6642d0897ce0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm"
Sep 11 17:32:33.187920 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:33.187872    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e15398c-1504-47de-a6dc-78a584cd17df-maas-api-tls\") pod \"maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm\" (UID: \"9e15398c-1504-47de-a6dc-78a584cd17df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm"
Sep 11 17:32:33.188029 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:33.187994    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5a085e-e930-45c6-a9cb-d2ad581aa47c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74\" (UID: \"0b5a085e-e930-45c6-a9cb-d2ad581aa47c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74"
Sep 11 17:32:33.190454 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:33.190426    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e15398c-1504-47de-a6dc-78a584cd17df-maas-api-tls\") pod \"maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm\" (UID: \"9e15398c-1504-47de-a6dc-78a584cd17df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm"
Sep 11 17:32:33.289083 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:33.289045    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5a085e-e930-45c6-a9cb-d2ad581aa47c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74\" (UID: \"0b5a085e-e930-45c6-a9cb-d2ad581aa47c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74"
Sep 11 17:32:33.289258 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:33.289211    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s7pj\" (UniqueName: \"kubernetes.io/projected/4f561801-e1cb-4712-88a2-6642d0897ce0-kube-api-access-4s7pj\") pod \"payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm\" (UID: \"4f561801-e1cb-4712-88a2-6642d0897ce0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm"
Sep 11 17:32:33.289336 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:33.289289    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9867\" (UniqueName: \"kubernetes.io/projected/0b5a085e-e930-45c6-a9cb-d2ad581aa47c-kube-api-access-c9867\") pod \"payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74\" (UID: \"0b5a085e-e930-45c6-a9cb-d2ad581aa47c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74"
Sep 11 17:32:33.289759 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:33.289411    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f561801-e1cb-4712-88a2-6642d0897ce0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm\" (UID: \"4f561801-e1cb-4712-88a2-6642d0897ce0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm"
Sep 11 17:32:33.290236 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:33.290215    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f561801-e1cb-4712-88a2-6642d0897ce0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm\" (UID: \"4f561801-e1cb-4712-88a2-6642d0897ce0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm"
Sep 11 17:32:33.290319 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:33.290269    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5a085e-e930-45c6-a9cb-d2ad581aa47c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74\" (UID: \"0b5a085e-e930-45c6-a9cb-d2ad581aa47c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74"
Sep 11 17:32:33.300651 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:33.300622    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s7pj\" (UniqueName: \"kubernetes.io/projected/4f561801-e1cb-4712-88a2-6642d0897ce0-kube-api-access-4s7pj\") pod \"payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm\" (UID: \"4f561801-e1cb-4712-88a2-6642d0897ce0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm"
Sep 11 17:32:33.300651 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:33.300633    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9867\" (UniqueName: \"kubernetes.io/projected/0b5a085e-e930-45c6-a9cb-d2ad581aa47c-kube-api-access-c9867\") pod \"payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74\" (UID: \"0b5a085e-e930-45c6-a9cb-d2ad581aa47c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74"
Sep 11 17:32:33.305552 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:33.305480    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm"
Sep 11 17:32:33.391394 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:33.391311    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74"
Sep 11 17:32:33.432420 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:33.432373    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm"
Sep 11 17:32:33.604043 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:33.603962    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm"]
Sep 11 17:32:33.606371 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:32:33.606335    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e15398c_1504_47de_a6dc_78a584cd17df.slice/crio-102f094f5da3e256667b340bf72e8a213b905f4055b2f39a7b27c5b04017e8e7 WatchSource:0}: Error finding container 102f094f5da3e256667b340bf72e8a213b905f4055b2f39a7b27c5b04017e8e7: Status 404 returned error can't find the container with id 102f094f5da3e256667b340bf72e8a213b905f4055b2f39a7b27c5b04017e8e7
Sep 11 17:32:33.667337 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:33.667310    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74"]
Sep 11 17:32:33.669737 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:32:33.669708    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5a085e_e930_45c6_a9cb_d2ad581aa47c.slice/crio-162341601e71dee51f9863587ce176b50061ce6544d4614a85fb8295b25a3b35 WatchSource:0}: Error finding container 162341601e71dee51f9863587ce176b50061ce6544d4614a85fb8295b25a3b35: Status 404 returned error can't find the container with id 162341601e71dee51f9863587ce176b50061ce6544d4614a85fb8295b25a3b35
Sep 11 17:32:33.719684 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:33.719660    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm"]
Sep 11 17:32:33.721357 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:32:33.721318    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f561801_e1cb_4712_88a2_6642d0897ce0.slice/crio-3bb3e4950c1baed489249f7dc614237018b56139290dfe1ef57475ab69f8e77a WatchSource:0}: Error finding container 3bb3e4950c1baed489249f7dc614237018b56139290dfe1ef57475ab69f8e77a: Status 404 returned error can't find the container with id 3bb3e4950c1baed489249f7dc614237018b56139290dfe1ef57475ab69f8e77a
Sep 11 17:32:34.098046 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:34.098015    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm" event={"ID":"4f561801-e1cb-4712-88a2-6642d0897ce0","Type":"ContainerStarted","Data":"6557ae53d8dcd08292362fbab0c8bea7da2fa12ec3d7715c43e4e96664f8dab3"}
Sep 11 17:32:34.098209 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:34.098071    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm" event={"ID":"4f561801-e1cb-4712-88a2-6642d0897ce0","Type":"ContainerStarted","Data":"3bb3e4950c1baed489249f7dc614237018b56139290dfe1ef57475ab69f8e77a"}
Sep 11 17:32:34.098277 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:34.098208    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm"
Sep 11 17:32:34.099639 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:34.099611    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74" event={"ID":"0b5a085e-e930-45c6-a9cb-d2ad581aa47c","Type":"ContainerStarted","Data":"96c6a442763865a88dbd2d2ad27081fc6e1f240b62ae401d5d60e51d9133cef2"}
Sep 11 17:32:34.099741 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:34.099661    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74"
Sep 11 17:32:34.099741 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:34.099692    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74" event={"ID":"0b5a085e-e930-45c6-a9cb-d2ad581aa47c","Type":"ContainerStarted","Data":"162341601e71dee51f9863587ce176b50061ce6544d4614a85fb8295b25a3b35"}
Sep 11 17:32:34.101200 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:34.101172    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm" event={"ID":"9e15398c-1504-47de-a6dc-78a584cd17df","Type":"ContainerStarted","Data":"102f094f5da3e256667b340bf72e8a213b905f4055b2f39a7b27c5b04017e8e7"}
Sep 11 17:32:34.116652 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:34.116611    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm" podStartSLOduration=1.116597111 podStartE2EDuration="1.116597111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:32:34.115383532 +0000 UTC m=+721.919555952" watchObservedRunningTime="2026-09-11 17:32:34.116597111 +0000 UTC m=+721.920769530"
Sep 11 17:32:34.134577 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:34.134530    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74" podStartSLOduration=1.134506725 podStartE2EDuration="1.134506725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:32:34.132383122 +0000 UTC m=+721.936555540" watchObservedRunningTime="2026-09-11 17:32:34.134506725 +0000 UTC m=+721.938679134"
Sep 11 17:32:35.109306 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:35.109261    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm" event={"ID":"9e15398c-1504-47de-a6dc-78a584cd17df","Type":"ContainerStarted","Data":"9da5a2b64e673c3cae7fd3e5ab818fc307fc13a6bc5518ea6c703e66f8b9e79a"}
Sep 11 17:32:35.109735 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:35.109614    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm"
Sep 11 17:32:35.134488 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:35.134442    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm" podStartSLOduration=2.672118908 podStartE2EDuration="3.134426711s" totalImagesPullingTime="462.307803ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:32:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:32:35.132850054 +0000 UTC m=+722.937022474" watchObservedRunningTime="2026-09-11 17:32:35.134426711 +0000 UTC m=+722.938599182"
Sep 11 17:32:36.420386 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:36.420347    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn"]
Sep 11 17:32:36.424498 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:36.424471    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn"
Sep 11 17:32:36.428447 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:36.428409    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-40972c14-serving-cert\""
Sep 11 17:32:36.447334 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:36.447302    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn"]
Sep 11 17:32:36.523663 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:36.523616    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mrm4\" (UniqueName: \"kubernetes.io/projected/47cbd2c1-aba5-47f0-8e6b-6db34de335e1-kube-api-access-5mrm4\") pod \"maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn\" (UID: \"47cbd2c1-aba5-47f0-8e6b-6db34de335e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn"
Sep 11 17:32:36.523663 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:36.523667    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47cbd2c1-aba5-47f0-8e6b-6db34de335e1-metrics-certs\") pod \"maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn\" (UID: \"47cbd2c1-aba5-47f0-8e6b-6db34de335e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn"
Sep 11 17:32:36.524014 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:36.523789    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47cbd2c1-aba5-47f0-8e6b-6db34de335e1-openshift-service-ca\") pod \"maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn\" (UID: \"47cbd2c1-aba5-47f0-8e6b-6db34de335e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn"
Sep 11 17:32:36.524084 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:36.524026    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47cbd2c1-aba5-47f0-8e6b-6db34de335e1-maas-api-tls\") pod \"maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn\" (UID: \"47cbd2c1-aba5-47f0-8e6b-6db34de335e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn"
Sep 11 17:32:36.625239 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:36.625204    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47cbd2c1-aba5-47f0-8e6b-6db34de335e1-openshift-service-ca\") pod \"maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn\" (UID: \"47cbd2c1-aba5-47f0-8e6b-6db34de335e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn"
Sep 11 17:32:36.625439 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:36.625286    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47cbd2c1-aba5-47f0-8e6b-6db34de335e1-maas-api-tls\") pod \"maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn\" (UID: \"47cbd2c1-aba5-47f0-8e6b-6db34de335e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn"
Sep 11 17:32:36.625439 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:36.625325    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mrm4\" (UniqueName: \"kubernetes.io/projected/47cbd2c1-aba5-47f0-8e6b-6db34de335e1-kube-api-access-5mrm4\") pod \"maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn\" (UID: \"47cbd2c1-aba5-47f0-8e6b-6db34de335e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn"
Sep 11 17:32:36.625439 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:36.625357    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47cbd2c1-aba5-47f0-8e6b-6db34de335e1-metrics-certs\") pod \"maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn\" (UID: \"47cbd2c1-aba5-47f0-8e6b-6db34de335e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn"
Sep 11 17:32:36.625629 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:32:36.625437    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-serving-cert: secret "maas-api-e2e-ait-40972c14-serving-cert" not found
Sep 11 17:32:36.625629 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:32:36.625516    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47cbd2c1-aba5-47f0-8e6b-6db34de335e1-maas-api-tls podName:47cbd2c1-aba5-47f0-8e6b-6db34de335e1 nodeName:}" failed. No retries permitted until 2026-09-11 17:32:37.125498015 +0000 UTC m=+724.929670412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/47cbd2c1-aba5-47f0-8e6b-6db34de335e1-maas-api-tls") pod "maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn" (UID: "47cbd2c1-aba5-47f0-8e6b-6db34de335e1") : secret "maas-api-e2e-ait-40972c14-serving-cert" not found
Sep 11 17:32:36.626000 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:36.625981    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47cbd2c1-aba5-47f0-8e6b-6db34de335e1-openshift-service-ca\") pod \"maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn\" (UID: \"47cbd2c1-aba5-47f0-8e6b-6db34de335e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn"
Sep 11 17:32:36.627921 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:36.627877    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47cbd2c1-aba5-47f0-8e6b-6db34de335e1-metrics-certs\") pod \"maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn\" (UID: \"47cbd2c1-aba5-47f0-8e6b-6db34de335e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn"
Sep 11 17:32:36.644663 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:36.644634    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mrm4\" (UniqueName: \"kubernetes.io/projected/47cbd2c1-aba5-47f0-8e6b-6db34de335e1-kube-api-access-5mrm4\") pod \"maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn\" (UID: \"47cbd2c1-aba5-47f0-8e6b-6db34de335e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn"
Sep 11 17:32:37.130735 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:37.130701    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47cbd2c1-aba5-47f0-8e6b-6db34de335e1-maas-api-tls\") pod \"maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn\" (UID: \"47cbd2c1-aba5-47f0-8e6b-6db34de335e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn"
Sep 11 17:32:37.133258 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:37.133226    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47cbd2c1-aba5-47f0-8e6b-6db34de335e1-maas-api-tls\") pod \"maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn\" (UID: \"47cbd2c1-aba5-47f0-8e6b-6db34de335e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn"
Sep 11 17:32:37.339035 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:37.339000    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn"
Sep 11 17:32:37.644518 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:37.644485    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn"]
Sep 11 17:32:37.647196 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:32:37.647162    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47cbd2c1_aba5_47f0_8e6b_6db34de335e1.slice/crio-d95ea1d7ac680841a83f07d241be06bf522fdfa512fcc2967d8319b40ee02e7f WatchSource:0}: Error finding container d95ea1d7ac680841a83f07d241be06bf522fdfa512fcc2967d8319b40ee02e7f: Status 404 returned error can't find the container with id d95ea1d7ac680841a83f07d241be06bf522fdfa512fcc2967d8319b40ee02e7f
Sep 11 17:32:37.725612 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:37.725580    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8"]
Sep 11 17:32:37.730188 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:37.730169    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8"
Sep 11 17:32:37.733489 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:37.733466    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-40972c14\""
Sep 11 17:32:37.733597 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:37.733507    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-40972c14-dockercfg-ngxw6\""
Sep 11 17:32:37.752684 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:37.752649    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8"]
Sep 11 17:32:37.837308 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:37.837271    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgw84\" (UniqueName: \"kubernetes.io/projected/ceb17b4f-57dd-4d45-95b0-8b044df15a39-kube-api-access-zgw84\") pod \"payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8\" (UID: \"ceb17b4f-57dd-4d45-95b0-8b044df15a39\") " pod="openshift-ingress/payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8"
Sep 11 17:32:37.837478 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:37.837319    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceb17b4f-57dd-4d45-95b0-8b044df15a39-plugins-config-volume\") pod \"payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8\" (UID: \"ceb17b4f-57dd-4d45-95b0-8b044df15a39\") " pod="openshift-ingress/payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8"
Sep 11 17:32:37.938214 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:37.938125    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgw84\" (UniqueName: \"kubernetes.io/projected/ceb17b4f-57dd-4d45-95b0-8b044df15a39-kube-api-access-zgw84\") pod \"payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8\" (UID: \"ceb17b4f-57dd-4d45-95b0-8b044df15a39\") " pod="openshift-ingress/payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8"
Sep 11 17:32:37.938214 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:37.938172    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceb17b4f-57dd-4d45-95b0-8b044df15a39-plugins-config-volume\") pod \"payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8\" (UID: \"ceb17b4f-57dd-4d45-95b0-8b044df15a39\") " pod="openshift-ingress/payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8"
Sep 11 17:32:37.939408 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:37.939379    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceb17b4f-57dd-4d45-95b0-8b044df15a39-plugins-config-volume\") pod \"payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8\" (UID: \"ceb17b4f-57dd-4d45-95b0-8b044df15a39\") " pod="openshift-ingress/payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8"
Sep 11 17:32:37.946854 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:37.946818    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgw84\" (UniqueName: \"kubernetes.io/projected/ceb17b4f-57dd-4d45-95b0-8b044df15a39-kube-api-access-zgw84\") pod \"payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8\" (UID: \"ceb17b4f-57dd-4d45-95b0-8b044df15a39\") " pod="openshift-ingress/payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8"
Sep 11 17:32:38.041546 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:38.041504    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8"
Sep 11 17:32:38.129963 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:38.129007    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn" event={"ID":"47cbd2c1-aba5-47f0-8e6b-6db34de335e1","Type":"ContainerStarted","Data":"e99ea7bd2f71604677a556111f3eeeb86eb8f41d220bd045c6c7346f36c7a6b4"}
Sep 11 17:32:38.129963 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:38.129828    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn"
Sep 11 17:32:38.129963 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:38.129943    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn" event={"ID":"47cbd2c1-aba5-47f0-8e6b-6db34de335e1","Type":"ContainerStarted","Data":"d95ea1d7ac680841a83f07d241be06bf522fdfa512fcc2967d8319b40ee02e7f"}
Sep 11 17:32:38.160345 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:38.160281    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn" podStartSLOduration=1.860133214 podStartE2EDuration="2.160261034s" totalImagesPullingTime="300.12782ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:32:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:32:38.157072186 +0000 UTC m=+725.961244604" watchObservedRunningTime="2026-09-11 17:32:38.160261034 +0000 UTC m=+725.964433454"
Sep 11 17:32:38.321389 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:38.321354    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8"]
Sep 11 17:32:38.322467 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:32:38.322441    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceb17b4f_57dd_4d45_95b0_8b044df15a39.slice/crio-f9392b1153bd321102783795dbf54123eb3ae1cc8a376f29ddc8a721edbad7ef WatchSource:0}: Error finding container f9392b1153bd321102783795dbf54123eb3ae1cc8a376f29ddc8a721edbad7ef: Status 404 returned error can't find the container with id f9392b1153bd321102783795dbf54123eb3ae1cc8a376f29ddc8a721edbad7ef
Sep 11 17:32:38.792108 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:38.792077    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk"]
Sep 11 17:32:38.795676 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:38.795659    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk"
Sep 11 17:32:38.798825 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:38.798801    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-3005e5\""
Sep 11 17:32:38.798961 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:38.798940    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-3005e5-dockercfg-9fmrx\""
Sep 11 17:32:38.807598 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:38.807575    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk"]
Sep 11 17:32:38.852526 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:38.852490    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh9zz\" (UniqueName: \"kubernetes.io/projected/51c1731d-e723-422e-8ef8-06860c00f657-kube-api-access-mh9zz\") pod \"payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk\" (UID: \"51c1731d-e723-422e-8ef8-06860c00f657\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk"
Sep 11 17:32:38.852727 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:38.852609    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51c1731d-e723-422e-8ef8-06860c00f657-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk\" (UID: \"51c1731d-e723-422e-8ef8-06860c00f657\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk"
Sep 11 17:32:38.953464 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:38.953429    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh9zz\" (UniqueName: \"kubernetes.io/projected/51c1731d-e723-422e-8ef8-06860c00f657-kube-api-access-mh9zz\") pod \"payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk\" (UID: \"51c1731d-e723-422e-8ef8-06860c00f657\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk"
Sep 11 17:32:38.953660 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:38.953487    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51c1731d-e723-422e-8ef8-06860c00f657-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk\" (UID: \"51c1731d-e723-422e-8ef8-06860c00f657\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk"
Sep 11 17:32:38.954204 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:38.954179    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51c1731d-e723-422e-8ef8-06860c00f657-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk\" (UID: \"51c1731d-e723-422e-8ef8-06860c00f657\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk"
Sep 11 17:32:38.963066 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:38.963036    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh9zz\" (UniqueName: \"kubernetes.io/projected/51c1731d-e723-422e-8ef8-06860c00f657-kube-api-access-mh9zz\") pod \"payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk\" (UID: \"51c1731d-e723-422e-8ef8-06860c00f657\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk"
Sep 11 17:32:39.107987 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:39.107955    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk"
Sep 11 17:32:39.136758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:39.136724    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8" event={"ID":"ceb17b4f-57dd-4d45-95b0-8b044df15a39","Type":"ContainerStarted","Data":"ec2e9b411a6e920066136cef579dbc24a29bbf6822b42033af5c7f4d59a15c16"}
Sep 11 17:32:39.136940 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:39.136784    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8" event={"ID":"ceb17b4f-57dd-4d45-95b0-8b044df15a39","Type":"ContainerStarted","Data":"f9392b1153bd321102783795dbf54123eb3ae1cc8a376f29ddc8a721edbad7ef"}
Sep 11 17:32:39.136940 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:39.136817    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8"
Sep 11 17:32:39.381271 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:39.381161    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8" podStartSLOduration=2.381136188 podStartE2EDuration="2.381136188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:32:39.159088199 +0000 UTC m=+726.963260620" watchObservedRunningTime="2026-09-11 17:32:39.381136188 +0000 UTC m=+727.185308612"
Sep 11 17:32:39.384455 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:39.384427    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk"]
Sep 11 17:32:39.385398 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:32:39.385371    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c1731d_e723_422e_8ef8_06860c00f657.slice/crio-432e4c2c477b8176d3ff59ac2994bc15b6829c78c110e8c677546e22a7f1ec62 WatchSource:0}: Error finding container 432e4c2c477b8176d3ff59ac2994bc15b6829c78c110e8c677546e22a7f1ec62: Status 404 returned error can't find the container with id 432e4c2c477b8176d3ff59ac2994bc15b6829c78c110e8c677546e22a7f1ec62
Sep 11 17:32:40.144136 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:40.144088    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk" event={"ID":"51c1731d-e723-422e-8ef8-06860c00f657","Type":"ContainerStarted","Data":"94289ac0d51fad1dd34979e3b5b556be778230ef0070491210dd0842eb8b97df"}
Sep 11 17:32:40.144584 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:40.144156    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk" event={"ID":"51c1731d-e723-422e-8ef8-06860c00f657","Type":"ContainerStarted","Data":"432e4c2c477b8176d3ff59ac2994bc15b6829c78c110e8c677546e22a7f1ec62"}
Sep 11 17:32:40.144584 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:40.144186    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk"
Sep 11 17:32:40.162859 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:40.162811    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk" podStartSLOduration=2.162794517 podStartE2EDuration="2.162794517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:32:40.160327629 +0000 UTC m=+727.964500048" watchObservedRunningTime="2026-09-11 17:32:40.162794517 +0000 UTC m=+727.966966935"
Sep 11 17:32:41.120871 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:41.120839    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm"
Sep 11 17:32:45.112323 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:45.112289    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm"
Sep 11 17:32:45.112711 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:45.112358    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74"
Sep 11 17:32:45.149867 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:45.149832    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn"
Sep 11 17:32:50.146318 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:50.146287    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8"
Sep 11 17:32:51.153092 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:32:51.153061    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk"
Sep 11 17:33:17.062844 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:17.062803    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn"]
Sep 11 17:33:17.063414 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:17.063082    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn" podUID="47cbd2c1-aba5-47f0-8e6b-6db34de335e1" containerName="maas-api" containerID="cri-o://e99ea7bd2f71604677a556111f3eeeb86eb8f41d220bd045c6c7346f36c7a6b4" gracePeriod=30
Sep 11 17:33:17.132810 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:17.132770    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8"]
Sep 11 17:33:17.133192 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:17.133025    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8" podUID="ceb17b4f-57dd-4d45-95b0-8b044df15a39" containerName="payload-processing" containerID="cri-o://ec2e9b411a6e920066136cef579dbc24a29bbf6822b42033af5c7f4d59a15c16" gracePeriod=30
Sep 11 17:33:17.339753 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:17.339705    2560 generic.go:356] "Generic (PLEG): container finished" podID="47cbd2c1-aba5-47f0-8e6b-6db34de335e1" containerID="e99ea7bd2f71604677a556111f3eeeb86eb8f41d220bd045c6c7346f36c7a6b4" exitCode=0
Sep 11 17:33:17.340031 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:17.339941    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn" event={"ID":"47cbd2c1-aba5-47f0-8e6b-6db34de335e1","Type":"ContainerDied","Data":"e99ea7bd2f71604677a556111f3eeeb86eb8f41d220bd045c6c7346f36c7a6b4"}
Sep 11 17:33:17.344679 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:17.342921    2560 generic.go:356] "Generic (PLEG): container finished" podID="ceb17b4f-57dd-4d45-95b0-8b044df15a39" containerID="ec2e9b411a6e920066136cef579dbc24a29bbf6822b42033af5c7f4d59a15c16" exitCode=0
Sep 11 17:33:17.344679 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:17.342976    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8" event={"ID":"ceb17b4f-57dd-4d45-95b0-8b044df15a39","Type":"ContainerDied","Data":"ec2e9b411a6e920066136cef579dbc24a29bbf6822b42033af5c7f4d59a15c16"}
Sep 11 17:33:17.392865 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:17.392835    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn"
Sep 11 17:33:17.427673 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:17.427642    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8"
Sep 11 17:33:17.525140 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:17.525107    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47cbd2c1-aba5-47f0-8e6b-6db34de335e1-metrics-certs\") pod \"47cbd2c1-aba5-47f0-8e6b-6db34de335e1\" (UID: \"47cbd2c1-aba5-47f0-8e6b-6db34de335e1\") "
Sep 11 17:33:17.525331 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:17.525147    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47cbd2c1-aba5-47f0-8e6b-6db34de335e1-maas-api-tls\") pod \"47cbd2c1-aba5-47f0-8e6b-6db34de335e1\" (UID: \"47cbd2c1-aba5-47f0-8e6b-6db34de335e1\") "
Sep 11 17:33:17.525331 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:17.525180    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceb17b4f-57dd-4d45-95b0-8b044df15a39-plugins-config-volume\") pod \"ceb17b4f-57dd-4d45-95b0-8b044df15a39\" (UID: \"ceb17b4f-57dd-4d45-95b0-8b044df15a39\") "
Sep 11 17:33:17.525331 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:17.525214    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47cbd2c1-aba5-47f0-8e6b-6db34de335e1-openshift-service-ca\") pod \"47cbd2c1-aba5-47f0-8e6b-6db34de335e1\" (UID: \"47cbd2c1-aba5-47f0-8e6b-6db34de335e1\") "
Sep 11 17:33:17.525331 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:17.525317    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgw84\" (UniqueName: \"kubernetes.io/projected/ceb17b4f-57dd-4d45-95b0-8b044df15a39-kube-api-access-zgw84\") pod \"ceb17b4f-57dd-4d45-95b0-8b044df15a39\" (UID: \"ceb17b4f-57dd-4d45-95b0-8b044df15a39\") "
Sep 11 17:33:17.525545 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:17.525375    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mrm4\" (UniqueName: \"kubernetes.io/projected/47cbd2c1-aba5-47f0-8e6b-6db34de335e1-kube-api-access-5mrm4\") pod \"47cbd2c1-aba5-47f0-8e6b-6db34de335e1\" (UID: \"47cbd2c1-aba5-47f0-8e6b-6db34de335e1\") "
Sep 11 17:33:17.525596 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:17.525563    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceb17b4f-57dd-4d45-95b0-8b044df15a39-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ceb17b4f-57dd-4d45-95b0-8b044df15a39" (UID: "ceb17b4f-57dd-4d45-95b0-8b044df15a39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:33:17.525817 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:17.525790    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceb17b4f-57dd-4d45-95b0-8b044df15a39-plugins-config-volume\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:33:17.525985 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:17.525839    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47cbd2c1-aba5-47f0-8e6b-6db34de335e1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "47cbd2c1-aba5-47f0-8e6b-6db34de335e1" (UID: "47cbd2c1-aba5-47f0-8e6b-6db34de335e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:33:17.527796 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:17.527767    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47cbd2c1-aba5-47f0-8e6b-6db34de335e1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "47cbd2c1-aba5-47f0-8e6b-6db34de335e1" (UID: "47cbd2c1-aba5-47f0-8e6b-6db34de335e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:33:17.527882 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:17.527820    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb17b4f-57dd-4d45-95b0-8b044df15a39-kube-api-access-zgw84" (OuterVolumeSpecName: "kube-api-access-zgw84") pod "ceb17b4f-57dd-4d45-95b0-8b044df15a39" (UID: "ceb17b4f-57dd-4d45-95b0-8b044df15a39"). InnerVolumeSpecName "kube-api-access-zgw84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:33:17.527882 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:17.527826    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47cbd2c1-aba5-47f0-8e6b-6db34de335e1-kube-api-access-5mrm4" (OuterVolumeSpecName: "kube-api-access-5mrm4") pod "47cbd2c1-aba5-47f0-8e6b-6db34de335e1" (UID: "47cbd2c1-aba5-47f0-8e6b-6db34de335e1"). InnerVolumeSpecName "kube-api-access-5mrm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:33:17.528084 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:17.528068    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47cbd2c1-aba5-47f0-8e6b-6db34de335e1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "47cbd2c1-aba5-47f0-8e6b-6db34de335e1" (UID: "47cbd2c1-aba5-47f0-8e6b-6db34de335e1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:33:17.627029 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:17.626940    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgw84\" (UniqueName: \"kubernetes.io/projected/ceb17b4f-57dd-4d45-95b0-8b044df15a39-kube-api-access-zgw84\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:33:17.627029 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:17.626971    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mrm4\" (UniqueName: \"kubernetes.io/projected/47cbd2c1-aba5-47f0-8e6b-6db34de335e1-kube-api-access-5mrm4\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:33:17.627029 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:17.626987    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47cbd2c1-aba5-47f0-8e6b-6db34de335e1-metrics-certs\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:33:17.627029 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:17.626999    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47cbd2c1-aba5-47f0-8e6b-6db34de335e1-maas-api-tls\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:33:17.627029 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:17.627008    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47cbd2c1-aba5-47f0-8e6b-6db34de335e1-openshift-service-ca\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:33:18.349588 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:18.349504    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn"
Sep 11 17:33:18.349588 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:18.349537    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn" event={"ID":"47cbd2c1-aba5-47f0-8e6b-6db34de335e1","Type":"ContainerDied","Data":"d95ea1d7ac680841a83f07d241be06bf522fdfa512fcc2967d8319b40ee02e7f"}
Sep 11 17:33:18.350095 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:18.349600    2560 scope.go:117] "RemoveContainer" containerID="e99ea7bd2f71604677a556111f3eeeb86eb8f41d220bd045c6c7346f36c7a6b4"
Sep 11 17:33:18.351203 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:18.351181    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8" event={"ID":"ceb17b4f-57dd-4d45-95b0-8b044df15a39","Type":"ContainerDied","Data":"f9392b1153bd321102783795dbf54123eb3ae1cc8a376f29ddc8a721edbad7ef"}
Sep 11 17:33:18.351305 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:18.351204    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8"
Sep 11 17:33:18.358933 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:18.358863    2560 scope.go:117] "RemoveContainer" containerID="ec2e9b411a6e920066136cef579dbc24a29bbf6822b42033af5c7f4d59a15c16"
Sep 11 17:33:18.393361 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:18.393326    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8"]
Sep 11 17:33:18.397625 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:18.397598    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-40972c14-b8d9c9dc9-k2tf8"]
Sep 11 17:33:18.443860 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:18.443824    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn"]
Sep 11 17:33:18.452376 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:18.452348    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-40972c14-586bb8c6b7-jdxcn"]
Sep 11 17:33:18.716374 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:18.716295    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47cbd2c1-aba5-47f0-8e6b-6db34de335e1" path="/var/lib/kubelet/pods/47cbd2c1-aba5-47f0-8e6b-6db34de335e1/volumes"
Sep 11 17:33:18.716658 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:18.716645    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceb17b4f-57dd-4d45-95b0-8b044df15a39" path="/var/lib/kubelet/pods/ceb17b4f-57dd-4d45-95b0-8b044df15a39/volumes"
Sep 11 17:33:25.273658 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:25.273624    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd"]
Sep 11 17:33:25.274257 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:25.274242    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47cbd2c1-aba5-47f0-8e6b-6db34de335e1" containerName="maas-api"
Sep 11 17:33:25.274300 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:25.274260    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="47cbd2c1-aba5-47f0-8e6b-6db34de335e1" containerName="maas-api"
Sep 11 17:33:25.274300 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:25.274280    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceb17b4f-57dd-4d45-95b0-8b044df15a39" containerName="payload-processing"
Sep 11 17:33:25.274300 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:25.274289    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceb17b4f-57dd-4d45-95b0-8b044df15a39" containerName="payload-processing"
Sep 11 17:33:25.274397 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:25.274370    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceb17b4f-57dd-4d45-95b0-8b044df15a39" containerName="payload-processing"
Sep 11 17:33:25.274397 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:25.274390    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="47cbd2c1-aba5-47f0-8e6b-6db34de335e1" containerName="maas-api"
Sep 11 17:33:25.279093 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:25.279073    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd"
Sep 11 17:33:25.282554 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:25.282536    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-b76706e3-serving-cert\""
Sep 11 17:33:25.299204 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:25.299179    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd"]
Sep 11 17:33:25.399455 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:25.399422    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68e81211-35ae-412d-ac0c-ad8825ab7a2e-maas-api-tls\") pod \"maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd\" (UID: \"68e81211-35ae-412d-ac0c-ad8825ab7a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd"
Sep 11 17:33:25.399455 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:25.399459    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68e81211-35ae-412d-ac0c-ad8825ab7a2e-metrics-certs\") pod \"maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd\" (UID: \"68e81211-35ae-412d-ac0c-ad8825ab7a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd"
Sep 11 17:33:25.399676 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:25.399483    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9bbw\" (UniqueName: \"kubernetes.io/projected/68e81211-35ae-412d-ac0c-ad8825ab7a2e-kube-api-access-x9bbw\") pod \"maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd\" (UID: \"68e81211-35ae-412d-ac0c-ad8825ab7a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd"
Sep 11 17:33:25.399676 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:25.399595    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e81211-35ae-412d-ac0c-ad8825ab7a2e-openshift-service-ca\") pod \"maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd\" (UID: \"68e81211-35ae-412d-ac0c-ad8825ab7a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd"
Sep 11 17:33:25.500312 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:25.500270    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68e81211-35ae-412d-ac0c-ad8825ab7a2e-maas-api-tls\") pod \"maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd\" (UID: \"68e81211-35ae-412d-ac0c-ad8825ab7a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd"
Sep 11 17:33:25.500312 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:25.500314    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68e81211-35ae-412d-ac0c-ad8825ab7a2e-metrics-certs\") pod \"maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd\" (UID: \"68e81211-35ae-412d-ac0c-ad8825ab7a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd"
Sep 11 17:33:25.500533 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:25.500358    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9bbw\" (UniqueName: \"kubernetes.io/projected/68e81211-35ae-412d-ac0c-ad8825ab7a2e-kube-api-access-x9bbw\") pod \"maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd\" (UID: \"68e81211-35ae-412d-ac0c-ad8825ab7a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd"
Sep 11 17:33:25.500533 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:25.500424    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e81211-35ae-412d-ac0c-ad8825ab7a2e-openshift-service-ca\") pod \"maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd\" (UID: \"68e81211-35ae-412d-ac0c-ad8825ab7a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd"
Sep 11 17:33:25.501142 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:25.501116    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e81211-35ae-412d-ac0c-ad8825ab7a2e-openshift-service-ca\") pod \"maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd\" (UID: \"68e81211-35ae-412d-ac0c-ad8825ab7a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd"
Sep 11 17:33:25.502792 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:25.502764    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68e81211-35ae-412d-ac0c-ad8825ab7a2e-maas-api-tls\") pod \"maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd\" (UID: \"68e81211-35ae-412d-ac0c-ad8825ab7a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd"
Sep 11 17:33:25.502874 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:25.502814    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68e81211-35ae-412d-ac0c-ad8825ab7a2e-metrics-certs\") pod \"maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd\" (UID: \"68e81211-35ae-412d-ac0c-ad8825ab7a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd"
Sep 11 17:33:25.513185 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:25.513136    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9bbw\" (UniqueName: \"kubernetes.io/projected/68e81211-35ae-412d-ac0c-ad8825ab7a2e-kube-api-access-x9bbw\") pod \"maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd\" (UID: \"68e81211-35ae-412d-ac0c-ad8825ab7a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd"
Sep 11 17:33:25.590008 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:25.589968    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd"
Sep 11 17:33:25.873940 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:25.873911    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd"]
Sep 11 17:33:25.876420 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:33:25.876371    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e81211_35ae_412d_ac0c_ad8825ab7a2e.slice/crio-fd51b335e440468e3f66b86b768f176891cdcad5444512cc242d7a6234d1fcad WatchSource:0}: Error finding container fd51b335e440468e3f66b86b768f176891cdcad5444512cc242d7a6234d1fcad: Status 404 returned error can't find the container with id fd51b335e440468e3f66b86b768f176891cdcad5444512cc242d7a6234d1fcad
Sep 11 17:33:25.990969 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:25.990925    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl"]
Sep 11 17:33:25.995677 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:25.995657    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl"
Sep 11 17:33:25.998275 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:25.998254    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-b76706e3\""
Sep 11 17:33:25.998504 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:25.998488    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-b76706e3-dockercfg-w6vj5\""
Sep 11 17:33:26.008424 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:26.008402    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl"]
Sep 11 17:33:26.021489 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:26.021460    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl"]
Sep 11 17:33:26.025088 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:26.025065    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl"
Sep 11 17:33:26.042706 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:26.042682    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl"]
Sep 11 17:33:26.107114 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:26.107065    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpnh5\" (UniqueName: \"kubernetes.io/projected/25c778f8-ab42-414f-9c24-f522485ecedb-kube-api-access-qpnh5\") pod \"payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl\" (UID: \"25c778f8-ab42-414f-9c24-f522485ecedb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl"
Sep 11 17:33:26.107270 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:26.107135    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t2sv\" (UniqueName: \"kubernetes.io/projected/a329eceb-a056-4c1f-b043-013c3d45d9d6-kube-api-access-9t2sv\") pod \"payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl\" (UID: \"a329eceb-a056-4c1f-b043-013c3d45d9d6\") " pod="openshift-ingress/payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl"
Sep 11 17:33:26.107270 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:26.107178    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c778f8-ab42-414f-9c24-f522485ecedb-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl\" (UID: \"25c778f8-ab42-414f-9c24-f522485ecedb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl"
Sep 11 17:33:26.107356 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:26.107330    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a329eceb-a056-4c1f-b043-013c3d45d9d6-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl\" (UID: \"a329eceb-a056-4c1f-b043-013c3d45d9d6\") " pod="openshift-ingress/payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl"
Sep 11 17:33:26.208382 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:26.208296    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a329eceb-a056-4c1f-b043-013c3d45d9d6-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl\" (UID: \"a329eceb-a056-4c1f-b043-013c3d45d9d6\") " pod="openshift-ingress/payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl"
Sep 11 17:33:26.208382 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:26.208350    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpnh5\" (UniqueName: \"kubernetes.io/projected/25c778f8-ab42-414f-9c24-f522485ecedb-kube-api-access-qpnh5\") pod \"payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl\" (UID: \"25c778f8-ab42-414f-9c24-f522485ecedb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl"
Sep 11 17:33:26.208382 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:26.208381    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t2sv\" (UniqueName: \"kubernetes.io/projected/a329eceb-a056-4c1f-b043-013c3d45d9d6-kube-api-access-9t2sv\") pod \"payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl\" (UID: \"a329eceb-a056-4c1f-b043-013c3d45d9d6\") " pod="openshift-ingress/payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl"
Sep 11 17:33:26.208612 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:26.208405    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c778f8-ab42-414f-9c24-f522485ecedb-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl\" (UID: \"25c778f8-ab42-414f-9c24-f522485ecedb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl"
Sep 11 17:33:26.209108 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:26.209084    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a329eceb-a056-4c1f-b043-013c3d45d9d6-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl\" (UID: \"a329eceb-a056-4c1f-b043-013c3d45d9d6\") " pod="openshift-ingress/payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl"
Sep 11 17:33:26.209200 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:26.209183    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c778f8-ab42-414f-9c24-f522485ecedb-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl\" (UID: \"25c778f8-ab42-414f-9c24-f522485ecedb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl"
Sep 11 17:33:26.217866 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:26.217840    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t2sv\" (UniqueName: \"kubernetes.io/projected/a329eceb-a056-4c1f-b043-013c3d45d9d6-kube-api-access-9t2sv\") pod \"payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl\" (UID: \"a329eceb-a056-4c1f-b043-013c3d45d9d6\") " pod="openshift-ingress/payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl"
Sep 11 17:33:26.217992 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:26.217850    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpnh5\" (UniqueName: \"kubernetes.io/projected/25c778f8-ab42-414f-9c24-f522485ecedb-kube-api-access-qpnh5\") pod \"payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl\" (UID: \"25c778f8-ab42-414f-9c24-f522485ecedb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl"
Sep 11 17:33:26.307364 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:26.307325    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl"
Sep 11 17:33:26.335085 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:26.335049    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl"
Sep 11 17:33:26.411724 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:26.411653    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd" event={"ID":"68e81211-35ae-412d-ac0c-ad8825ab7a2e","Type":"ContainerStarted","Data":"fd51b335e440468e3f66b86b768f176891cdcad5444512cc242d7a6234d1fcad"}
Sep 11 17:33:26.586767 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:26.586740    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl"]
Sep 11 17:33:26.588934 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:33:26.588888    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda329eceb_a056_4c1f_b043_013c3d45d9d6.slice/crio-1899fb460b4db5ba1a654162f5208494df4dff35f9e912ed260125647d930e83 WatchSource:0}: Error finding container 1899fb460b4db5ba1a654162f5208494df4dff35f9e912ed260125647d930e83: Status 404 returned error can't find the container with id 1899fb460b4db5ba1a654162f5208494df4dff35f9e912ed260125647d930e83
Sep 11 17:33:26.627804 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:26.627777    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl"]
Sep 11 17:33:26.631142 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:33:26.631114    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c778f8_ab42_414f_9c24_f522485ecedb.slice/crio-889c43f700b9f725c10e4b905539a0b8ca3d40073f17a54c024d9ba12fd4a4e6 WatchSource:0}: Error finding container 889c43f700b9f725c10e4b905539a0b8ca3d40073f17a54c024d9ba12fd4a4e6: Status 404 returned error can't find the container with id 889c43f700b9f725c10e4b905539a0b8ca3d40073f17a54c024d9ba12fd4a4e6
Sep 11 17:33:27.419179 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:27.419137    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl" event={"ID":"25c778f8-ab42-414f-9c24-f522485ecedb","Type":"ContainerStarted","Data":"773bbe90544d2b73093c0dbe9dcc0797cbcc5df892dbe244f1e95faeff278a77"}
Sep 11 17:33:27.419644 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:27.419206    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl" event={"ID":"25c778f8-ab42-414f-9c24-f522485ecedb","Type":"ContainerStarted","Data":"889c43f700b9f725c10e4b905539a0b8ca3d40073f17a54c024d9ba12fd4a4e6"}
Sep 11 17:33:27.419644 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:27.419236    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl"
Sep 11 17:33:27.420876 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:27.420849    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd" event={"ID":"68e81211-35ae-412d-ac0c-ad8825ab7a2e","Type":"ContainerStarted","Data":"331bdfd0232f0e295a543bdf080460a3533861c363201d3c699d5732d805f661"}
Sep 11 17:33:27.421011 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:27.420932    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd"
Sep 11 17:33:27.422929 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:27.422886    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl" event={"ID":"a329eceb-a056-4c1f-b043-013c3d45d9d6","Type":"ContainerStarted","Data":"0297ba7ed8cb63d965391b0498672335d8cccc50ef84c4423b2611ae6bc1a980"}
Sep 11 17:33:27.423025 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:27.422952    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl" event={"ID":"a329eceb-a056-4c1f-b043-013c3d45d9d6","Type":"ContainerStarted","Data":"1899fb460b4db5ba1a654162f5208494df4dff35f9e912ed260125647d930e83"}
Sep 11 17:33:27.423132 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:27.423113    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl"
Sep 11 17:33:27.442159 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:27.442110    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl" podStartSLOduration=2.442093802 podStartE2EDuration="2.442093802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:33:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:33:27.440304254 +0000 UTC m=+775.244476674" watchObservedRunningTime="2026-09-11 17:33:27.442093802 +0000 UTC m=+775.246266221"
Sep 11 17:33:27.461552 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:27.461500    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd" podStartSLOduration=1.89779323 podStartE2EDuration="2.461482297s" totalImagesPullingTime="563.689067ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:33:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:33:27.459087338 +0000 UTC m=+775.263259758" watchObservedRunningTime="2026-09-11 17:33:27.461482297 +0000 UTC m=+775.265654715"
Sep 11 17:33:27.483148 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:27.483087    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl" podStartSLOduration=2.483065162 podStartE2EDuration="2.483065162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:33:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:33:27.480731135 +0000 UTC m=+775.284903559" watchObservedRunningTime="2026-09-11 17:33:27.483065162 +0000 UTC m=+775.287237583"
Sep 11 17:33:33.434116 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:33.434088    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd"
Sep 11 17:33:38.430997 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:38.430970    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl"
Sep 11 17:33:39.435993 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:33:39.435961    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl"
Sep 11 17:34:02.381521 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.381436    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd"]
Sep 11 17:34:02.382148 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.381693    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd" podUID="68e81211-35ae-412d-ac0c-ad8825ab7a2e" containerName="maas-api" containerID="cri-o://331bdfd0232f0e295a543bdf080460a3533861c363201d3c699d5732d805f661" gracePeriod=30
Sep 11 17:34:02.443805 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.443760    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl"]
Sep 11 17:34:02.444060 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.444030    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl" podUID="a329eceb-a056-4c1f-b043-013c3d45d9d6" containerName="payload-processing" containerID="cri-o://0297ba7ed8cb63d965391b0498672335d8cccc50ef84c4423b2611ae6bc1a980" gracePeriod=30
Sep 11 17:34:02.450632 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.450604    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl"]
Sep 11 17:34:02.450850 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.450826    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl" podUID="25c778f8-ab42-414f-9c24-f522485ecedb" containerName="payload-pre-processing" containerID="cri-o://773bbe90544d2b73093c0dbe9dcc0797cbcc5df892dbe244f1e95faeff278a77" gracePeriod=30
Sep 11 17:34:02.605407 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.605372    2560 generic.go:356] "Generic (PLEG): container finished" podID="25c778f8-ab42-414f-9c24-f522485ecedb" containerID="773bbe90544d2b73093c0dbe9dcc0797cbcc5df892dbe244f1e95faeff278a77" exitCode=0
Sep 11 17:34:02.605615 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.605488    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl" event={"ID":"25c778f8-ab42-414f-9c24-f522485ecedb","Type":"ContainerDied","Data":"773bbe90544d2b73093c0dbe9dcc0797cbcc5df892dbe244f1e95faeff278a77"}
Sep 11 17:34:02.607655 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.607553    2560 generic.go:356] "Generic (PLEG): container finished" podID="68e81211-35ae-412d-ac0c-ad8825ab7a2e" containerID="331bdfd0232f0e295a543bdf080460a3533861c363201d3c699d5732d805f661" exitCode=0
Sep 11 17:34:02.607655 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.607637    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd" event={"ID":"68e81211-35ae-412d-ac0c-ad8825ab7a2e","Type":"ContainerDied","Data":"331bdfd0232f0e295a543bdf080460a3533861c363201d3c699d5732d805f661"}
Sep 11 17:34:02.609613 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.609588    2560 generic.go:356] "Generic (PLEG): container finished" podID="a329eceb-a056-4c1f-b043-013c3d45d9d6" containerID="0297ba7ed8cb63d965391b0498672335d8cccc50ef84c4423b2611ae6bc1a980" exitCode=0
Sep 11 17:34:02.609748 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.609693    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl" event={"ID":"a329eceb-a056-4c1f-b043-013c3d45d9d6","Type":"ContainerDied","Data":"0297ba7ed8cb63d965391b0498672335d8cccc50ef84c4423b2611ae6bc1a980"}
Sep 11 17:34:02.719188 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.719163    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd"
Sep 11 17:34:02.773759 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.773735    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl"
Sep 11 17:34:02.778052 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.777966    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl"
Sep 11 17:34:02.834990 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.834958    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e81211-35ae-412d-ac0c-ad8825ab7a2e-openshift-service-ca\") pod \"68e81211-35ae-412d-ac0c-ad8825ab7a2e\" (UID: \"68e81211-35ae-412d-ac0c-ad8825ab7a2e\") "
Sep 11 17:34:02.834990 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.834997    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68e81211-35ae-412d-ac0c-ad8825ab7a2e-metrics-certs\") pod \"68e81211-35ae-412d-ac0c-ad8825ab7a2e\" (UID: \"68e81211-35ae-412d-ac0c-ad8825ab7a2e\") "
Sep 11 17:34:02.835237 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.835024    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68e81211-35ae-412d-ac0c-ad8825ab7a2e-maas-api-tls\") pod \"68e81211-35ae-412d-ac0c-ad8825ab7a2e\" (UID: \"68e81211-35ae-412d-ac0c-ad8825ab7a2e\") "
Sep 11 17:34:02.835237 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.835135    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c778f8-ab42-414f-9c24-f522485ecedb-plugins-config-volume\") pod \"25c778f8-ab42-414f-9c24-f522485ecedb\" (UID: \"25c778f8-ab42-414f-9c24-f522485ecedb\") "
Sep 11 17:34:02.835237 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.835197    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9t2sv\" (UniqueName: \"kubernetes.io/projected/a329eceb-a056-4c1f-b043-013c3d45d9d6-kube-api-access-9t2sv\") pod \"a329eceb-a056-4c1f-b043-013c3d45d9d6\" (UID: \"a329eceb-a056-4c1f-b043-013c3d45d9d6\") "
Sep 11 17:34:02.835410 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.835255    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpnh5\" (UniqueName: \"kubernetes.io/projected/25c778f8-ab42-414f-9c24-f522485ecedb-kube-api-access-qpnh5\") pod \"25c778f8-ab42-414f-9c24-f522485ecedb\" (UID: \"25c778f8-ab42-414f-9c24-f522485ecedb\") "
Sep 11 17:34:02.835410 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.835298    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a329eceb-a056-4c1f-b043-013c3d45d9d6-plugins-config-volume\") pod \"a329eceb-a056-4c1f-b043-013c3d45d9d6\" (UID: \"a329eceb-a056-4c1f-b043-013c3d45d9d6\") "
Sep 11 17:34:02.835410 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.835348    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9bbw\" (UniqueName: \"kubernetes.io/projected/68e81211-35ae-412d-ac0c-ad8825ab7a2e-kube-api-access-x9bbw\") pod \"68e81211-35ae-412d-ac0c-ad8825ab7a2e\" (UID: \"68e81211-35ae-412d-ac0c-ad8825ab7a2e\") "
Sep 11 17:34:02.835410 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.835374    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68e81211-35ae-412d-ac0c-ad8825ab7a2e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "68e81211-35ae-412d-ac0c-ad8825ab7a2e" (UID: "68e81211-35ae-412d-ac0c-ad8825ab7a2e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:34:02.835870 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.835714    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a329eceb-a056-4c1f-b043-013c3d45d9d6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a329eceb-a056-4c1f-b043-013c3d45d9d6" (UID: "a329eceb-a056-4c1f-b043-013c3d45d9d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:34:02.835870 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.835813    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e81211-35ae-412d-ac0c-ad8825ab7a2e-openshift-service-ca\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:34:02.836050 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.835956    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25c778f8-ab42-414f-9c24-f522485ecedb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "25c778f8-ab42-414f-9c24-f522485ecedb" (UID: "25c778f8-ab42-414f-9c24-f522485ecedb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:34:02.837643 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.837577    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68e81211-35ae-412d-ac0c-ad8825ab7a2e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "68e81211-35ae-412d-ac0c-ad8825ab7a2e" (UID: "68e81211-35ae-412d-ac0c-ad8825ab7a2e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:34:02.837643 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.837600    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68e81211-35ae-412d-ac0c-ad8825ab7a2e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "68e81211-35ae-412d-ac0c-ad8825ab7a2e" (UID: "68e81211-35ae-412d-ac0c-ad8825ab7a2e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:34:02.837831 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.837649    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a329eceb-a056-4c1f-b043-013c3d45d9d6-kube-api-access-9t2sv" (OuterVolumeSpecName: "kube-api-access-9t2sv") pod "a329eceb-a056-4c1f-b043-013c3d45d9d6" (UID: "a329eceb-a056-4c1f-b043-013c3d45d9d6"). InnerVolumeSpecName "kube-api-access-9t2sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:34:02.838248 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.838223    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c778f8-ab42-414f-9c24-f522485ecedb-kube-api-access-qpnh5" (OuterVolumeSpecName: "kube-api-access-qpnh5") pod "25c778f8-ab42-414f-9c24-f522485ecedb" (UID: "25c778f8-ab42-414f-9c24-f522485ecedb"). InnerVolumeSpecName "kube-api-access-qpnh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:34:02.838343 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.838327    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68e81211-35ae-412d-ac0c-ad8825ab7a2e-kube-api-access-x9bbw" (OuterVolumeSpecName: "kube-api-access-x9bbw") pod "68e81211-35ae-412d-ac0c-ad8825ab7a2e" (UID: "68e81211-35ae-412d-ac0c-ad8825ab7a2e"). InnerVolumeSpecName "kube-api-access-x9bbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:34:02.936873 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.936838    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25c778f8-ab42-414f-9c24-f522485ecedb-plugins-config-volume\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:34:02.936873 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.936865    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t2sv\" (UniqueName: \"kubernetes.io/projected/a329eceb-a056-4c1f-b043-013c3d45d9d6-kube-api-access-9t2sv\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:34:02.936873 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.936874    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpnh5\" (UniqueName: \"kubernetes.io/projected/25c778f8-ab42-414f-9c24-f522485ecedb-kube-api-access-qpnh5\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:34:02.936873 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.936882    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a329eceb-a056-4c1f-b043-013c3d45d9d6-plugins-config-volume\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:34:02.937176 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.936891    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9bbw\" (UniqueName: \"kubernetes.io/projected/68e81211-35ae-412d-ac0c-ad8825ab7a2e-kube-api-access-x9bbw\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:34:02.937176 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.936934    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68e81211-35ae-412d-ac0c-ad8825ab7a2e-metrics-certs\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:34:02.937176 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:02.936944    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68e81211-35ae-412d-ac0c-ad8825ab7a2e-maas-api-tls\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:34:03.618456 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:03.618423    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl"
Sep 11 17:34:03.618924 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:03.618456    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl" event={"ID":"25c778f8-ab42-414f-9c24-f522485ecedb","Type":"ContainerDied","Data":"889c43f700b9f725c10e4b905539a0b8ca3d40073f17a54c024d9ba12fd4a4e6"}
Sep 11 17:34:03.618924 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:03.618522    2560 scope.go:117] "RemoveContainer" containerID="773bbe90544d2b73093c0dbe9dcc0797cbcc5df892dbe244f1e95faeff278a77"
Sep 11 17:34:03.620853 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:03.620815    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd"
Sep 11 17:34:03.621019 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:03.620869    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd" event={"ID":"68e81211-35ae-412d-ac0c-ad8825ab7a2e","Type":"ContainerDied","Data":"fd51b335e440468e3f66b86b768f176891cdcad5444512cc242d7a6234d1fcad"}
Sep 11 17:34:03.625767 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:03.623698    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl" event={"ID":"a329eceb-a056-4c1f-b043-013c3d45d9d6","Type":"ContainerDied","Data":"1899fb460b4db5ba1a654162f5208494df4dff35f9e912ed260125647d930e83"}
Sep 11 17:34:03.625767 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:03.623796    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl"
Sep 11 17:34:03.630761 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:03.630602    2560 scope.go:117] "RemoveContainer" containerID="331bdfd0232f0e295a543bdf080460a3533861c363201d3c699d5732d805f661"
Sep 11 17:34:03.642538 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:03.642520    2560 scope.go:117] "RemoveContainer" containerID="0297ba7ed8cb63d965391b0498672335d8cccc50ef84c4423b2611ae6bc1a980"
Sep 11 17:34:03.651945 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:03.651918    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl"]
Sep 11 17:34:03.664436 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:03.664401    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b76706e3-856c8859c4-p6tpl"]
Sep 11 17:34:03.675093 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:03.675059    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl"]
Sep 11 17:34:03.680168 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:03.680137    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b76706e3-7dcf9554d6-qxdgl"]
Sep 11 17:34:03.694219 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:03.694193    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd"]
Sep 11 17:34:03.696987 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:03.696962    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b76706e3-696fc49f44-fqzmd"]
Sep 11 17:34:04.716003 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:04.715967    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25c778f8-ab42-414f-9c24-f522485ecedb" path="/var/lib/kubelet/pods/25c778f8-ab42-414f-9c24-f522485ecedb/volumes"
Sep 11 17:34:04.716398 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:04.716381    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68e81211-35ae-412d-ac0c-ad8825ab7a2e" path="/var/lib/kubelet/pods/68e81211-35ae-412d-ac0c-ad8825ab7a2e/volumes"
Sep 11 17:34:04.716689 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:04.716676    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a329eceb-a056-4c1f-b043-013c3d45d9d6" path="/var/lib/kubelet/pods/a329eceb-a056-4c1f-b043-013c3d45d9d6/volumes"
Sep 11 17:34:08.904454 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:08.904416    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw"]
Sep 11 17:34:08.909641 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:08.909609    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68e81211-35ae-412d-ac0c-ad8825ab7a2e" containerName="maas-api"
Sep 11 17:34:08.909833 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:08.909809    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="68e81211-35ae-412d-ac0c-ad8825ab7a2e" containerName="maas-api"
Sep 11 17:34:08.909981 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:08.909843    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25c778f8-ab42-414f-9c24-f522485ecedb" containerName="payload-pre-processing"
Sep 11 17:34:08.909981 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:08.909853    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="25c778f8-ab42-414f-9c24-f522485ecedb" containerName="payload-pre-processing"
Sep 11 17:34:08.909981 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:08.909887    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a329eceb-a056-4c1f-b043-013c3d45d9d6" containerName="payload-processing"
Sep 11 17:34:08.909981 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:08.909922    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a329eceb-a056-4c1f-b043-013c3d45d9d6" containerName="payload-processing"
Sep 11 17:34:08.910178 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:08.910031    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a329eceb-a056-4c1f-b043-013c3d45d9d6" containerName="payload-processing"
Sep 11 17:34:08.910178 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:08.910046    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="25c778f8-ab42-414f-9c24-f522485ecedb" containerName="payload-pre-processing"
Sep 11 17:34:08.910178 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:08.910061    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="68e81211-35ae-412d-ac0c-ad8825ab7a2e" containerName="maas-api"
Sep 11 17:34:08.915683 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:08.915658    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw"]
Sep 11 17:34:08.915852 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:08.915807    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw"
Sep 11 17:34:08.918438 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:08.918410    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a1d8713f-serving-cert\""
Sep 11 17:34:08.993740 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:08.993702    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4393699-ff06-4247-a0f2-24e5516c23dd-metrics-certs\") pod \"maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw\" (UID: \"e4393699-ff06-4247-a0f2-24e5516c23dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw"
Sep 11 17:34:08.993947 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:08.993834    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcdp7\" (UniqueName: \"kubernetes.io/projected/e4393699-ff06-4247-a0f2-24e5516c23dd-kube-api-access-xcdp7\") pod \"maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw\" (UID: \"e4393699-ff06-4247-a0f2-24e5516c23dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw"
Sep 11 17:34:08.994010 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:08.993951    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4393699-ff06-4247-a0f2-24e5516c23dd-maas-api-tls\") pod \"maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw\" (UID: \"e4393699-ff06-4247-a0f2-24e5516c23dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw"
Sep 11 17:34:08.994064 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:08.994004    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4393699-ff06-4247-a0f2-24e5516c23dd-openshift-service-ca\") pod \"maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw\" (UID: \"e4393699-ff06-4247-a0f2-24e5516c23dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw"
Sep 11 17:34:09.094720 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.094659    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4393699-ff06-4247-a0f2-24e5516c23dd-maas-api-tls\") pod \"maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw\" (UID: \"e4393699-ff06-4247-a0f2-24e5516c23dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw"
Sep 11 17:34:09.094720 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.094722    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4393699-ff06-4247-a0f2-24e5516c23dd-openshift-service-ca\") pod \"maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw\" (UID: \"e4393699-ff06-4247-a0f2-24e5516c23dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw"
Sep 11 17:34:09.095043 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.094763    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4393699-ff06-4247-a0f2-24e5516c23dd-metrics-certs\") pod \"maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw\" (UID: \"e4393699-ff06-4247-a0f2-24e5516c23dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw"
Sep 11 17:34:09.095043 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:34:09.094804    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-serving-cert: secret "maas-api-e2e-ait-a1d8713f-serving-cert" not found
Sep 11 17:34:09.095043 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.094864    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcdp7\" (UniqueName: \"kubernetes.io/projected/e4393699-ff06-4247-a0f2-24e5516c23dd-kube-api-access-xcdp7\") pod \"maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw\" (UID: \"e4393699-ff06-4247-a0f2-24e5516c23dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw"
Sep 11 17:34:09.095043 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:34:09.094885    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4393699-ff06-4247-a0f2-24e5516c23dd-maas-api-tls podName:e4393699-ff06-4247-a0f2-24e5516c23dd nodeName:}" failed. No retries permitted until 2026-09-11 17:34:09.594862276 +0000 UTC m=+817.399034691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e4393699-ff06-4247-a0f2-24e5516c23dd-maas-api-tls") pod "maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw" (UID: "e4393699-ff06-4247-a0f2-24e5516c23dd") : secret "maas-api-e2e-ait-a1d8713f-serving-cert" not found
Sep 11 17:34:09.095838 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.095765    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4393699-ff06-4247-a0f2-24e5516c23dd-openshift-service-ca\") pod \"maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw\" (UID: \"e4393699-ff06-4247-a0f2-24e5516c23dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw"
Sep 11 17:34:09.098303 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.098278    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4393699-ff06-4247-a0f2-24e5516c23dd-metrics-certs\") pod \"maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw\" (UID: \"e4393699-ff06-4247-a0f2-24e5516c23dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw"
Sep 11 17:34:09.103884 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.103855    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcdp7\" (UniqueName: \"kubernetes.io/projected/e4393699-ff06-4247-a0f2-24e5516c23dd-kube-api-access-xcdp7\") pod \"maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw\" (UID: \"e4393699-ff06-4247-a0f2-24e5516c23dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw"
Sep 11 17:34:09.583850 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.583812    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc"]
Sep 11 17:34:09.587594 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.587576    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc"
Sep 11 17:34:09.590385 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.590363    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a1d8713f\""
Sep 11 17:34:09.597238 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.597213    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc"]
Sep 11 17:34:09.600510 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.600483    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4393699-ff06-4247-a0f2-24e5516c23dd-maas-api-tls\") pod \"maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw\" (UID: \"e4393699-ff06-4247-a0f2-24e5516c23dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw"
Sep 11 17:34:09.603061 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.603035    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4393699-ff06-4247-a0f2-24e5516c23dd-maas-api-tls\") pod \"maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw\" (UID: \"e4393699-ff06-4247-a0f2-24e5516c23dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw"
Sep 11 17:34:09.632042 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.632010    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf"]
Sep 11 17:34:09.635970 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.635934    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf"
Sep 11 17:34:09.638449 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.638428    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a1d8713f-dockercfg-nwzfq\""
Sep 11 17:34:09.649474 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.649435    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf"]
Sep 11 17:34:09.701689 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.701637    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5732e6-da94-41bf-81bd-b6bddcc75857-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc\" (UID: \"fe5732e6-da94-41bf-81bd-b6bddcc75857\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc"
Sep 11 17:34:09.701862 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.701741    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99e6b796-603d-483f-a7e2-49db822614f5-plugins-config-volume\") pod \"payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf\" (UID: \"99e6b796-603d-483f-a7e2-49db822614f5\") " pod="openshift-ingress/payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf"
Sep 11 17:34:09.701862 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.701789    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s96n\" (UniqueName: \"kubernetes.io/projected/99e6b796-603d-483f-a7e2-49db822614f5-kube-api-access-5s96n\") pod \"payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf\" (UID: \"99e6b796-603d-483f-a7e2-49db822614f5\") " pod="openshift-ingress/payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf"
Sep 11 17:34:09.701984 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.701875    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgx7r\" (UniqueName: \"kubernetes.io/projected/fe5732e6-da94-41bf-81bd-b6bddcc75857-kube-api-access-fgx7r\") pod \"payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc\" (UID: \"fe5732e6-da94-41bf-81bd-b6bddcc75857\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc"
Sep 11 17:34:09.802848 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.802803    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99e6b796-603d-483f-a7e2-49db822614f5-plugins-config-volume\") pod \"payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf\" (UID: \"99e6b796-603d-483f-a7e2-49db822614f5\") " pod="openshift-ingress/payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf"
Sep 11 17:34:09.802848 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.802852    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s96n\" (UniqueName: \"kubernetes.io/projected/99e6b796-603d-483f-a7e2-49db822614f5-kube-api-access-5s96n\") pod \"payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf\" (UID: \"99e6b796-603d-483f-a7e2-49db822614f5\") " pod="openshift-ingress/payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf"
Sep 11 17:34:09.803110 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.802953    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgx7r\" (UniqueName: \"kubernetes.io/projected/fe5732e6-da94-41bf-81bd-b6bddcc75857-kube-api-access-fgx7r\") pod \"payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc\" (UID: \"fe5732e6-da94-41bf-81bd-b6bddcc75857\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc"
Sep 11 17:34:09.803110 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.803015    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5732e6-da94-41bf-81bd-b6bddcc75857-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc\" (UID: \"fe5732e6-da94-41bf-81bd-b6bddcc75857\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc"
Sep 11 17:34:09.803650 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.803631    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99e6b796-603d-483f-a7e2-49db822614f5-plugins-config-volume\") pod \"payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf\" (UID: \"99e6b796-603d-483f-a7e2-49db822614f5\") " pod="openshift-ingress/payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf"
Sep 11 17:34:09.803689 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.803645    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5732e6-da94-41bf-81bd-b6bddcc75857-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc\" (UID: \"fe5732e6-da94-41bf-81bd-b6bddcc75857\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc"
Sep 11 17:34:09.811636 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.811599    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s96n\" (UniqueName: \"kubernetes.io/projected/99e6b796-603d-483f-a7e2-49db822614f5-kube-api-access-5s96n\") pod \"payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf\" (UID: \"99e6b796-603d-483f-a7e2-49db822614f5\") " pod="openshift-ingress/payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf"
Sep 11 17:34:09.811636 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.811607    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgx7r\" (UniqueName: \"kubernetes.io/projected/fe5732e6-da94-41bf-81bd-b6bddcc75857-kube-api-access-fgx7r\") pod \"payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc\" (UID: \"fe5732e6-da94-41bf-81bd-b6bddcc75857\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc"
Sep 11 17:34:09.827481 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.827448    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw"
Sep 11 17:34:09.900277 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.900244    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc"
Sep 11 17:34:09.947958 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:09.947886    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf"
Sep 11 17:34:10.102820 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:10.102791    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw"]
Sep 11 17:34:10.104660 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:34:10.104608    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4393699_ff06_4247_a0f2_24e5516c23dd.slice/crio-2af4adda251d8860e72902be0550ba23cb9bc78680682cf8031d069ad4a0ebde WatchSource:0}: Error finding container 2af4adda251d8860e72902be0550ba23cb9bc78680682cf8031d069ad4a0ebde: Status 404 returned error can't find the container with id 2af4adda251d8860e72902be0550ba23cb9bc78680682cf8031d069ad4a0ebde
Sep 11 17:34:10.180649 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:10.180625    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc"]
Sep 11 17:34:10.183532 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:34:10.183504    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5732e6_da94_41bf_81bd_b6bddcc75857.slice/crio-18079a9bf40c9729ab63c8432e7be48c528a320f2d9c748bd1b2449b81d47160 WatchSource:0}: Error finding container 18079a9bf40c9729ab63c8432e7be48c528a320f2d9c748bd1b2449b81d47160: Status 404 returned error can't find the container with id 18079a9bf40c9729ab63c8432e7be48c528a320f2d9c748bd1b2449b81d47160
Sep 11 17:34:10.235470 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:10.235445    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf"]
Sep 11 17:34:10.238126 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:34:10.238104    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e6b796_603d_483f_a7e2_49db822614f5.slice/crio-94da2d4b6a3058c39352455b3a0b5f10bae325cec893f8b4f4aa71508e36aaa5 WatchSource:0}: Error finding container 94da2d4b6a3058c39352455b3a0b5f10bae325cec893f8b4f4aa71508e36aaa5: Status 404 returned error can't find the container with id 94da2d4b6a3058c39352455b3a0b5f10bae325cec893f8b4f4aa71508e36aaa5
Sep 11 17:34:10.675496 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:10.675460    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf" event={"ID":"99e6b796-603d-483f-a7e2-49db822614f5","Type":"ContainerStarted","Data":"507ba2fe90d94cf4dd1ca08adf6638d9daad0556efae7207e15a7139612880d2"}
Sep 11 17:34:10.675804 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:10.675785    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf"
Sep 11 17:34:10.675946 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:10.675887    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf" event={"ID":"99e6b796-603d-483f-a7e2-49db822614f5","Type":"ContainerStarted","Data":"94da2d4b6a3058c39352455b3a0b5f10bae325cec893f8b4f4aa71508e36aaa5"}
Sep 11 17:34:10.677476 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:10.677441    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc" event={"ID":"fe5732e6-da94-41bf-81bd-b6bddcc75857","Type":"ContainerStarted","Data":"2032363722a0e1d503bcacd162178b54a7f8aec36047140dd92aa5f6d4c3e6c1"}
Sep 11 17:34:10.677589 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:10.677487    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc" event={"ID":"fe5732e6-da94-41bf-81bd-b6bddcc75857","Type":"ContainerStarted","Data":"18079a9bf40c9729ab63c8432e7be48c528a320f2d9c748bd1b2449b81d47160"}
Sep 11 17:34:10.677589 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:10.677527    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc"
Sep 11 17:34:10.679186 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:10.679163    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw" event={"ID":"e4393699-ff06-4247-a0f2-24e5516c23dd","Type":"ContainerStarted","Data":"4b9d1d1b9f991de70a4763a725a509910580f20062c0bfff541e765a18b70cc6"}
Sep 11 17:34:10.679281 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:10.679203    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw"
Sep 11 17:34:10.679281 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:10.679220    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw" event={"ID":"e4393699-ff06-4247-a0f2-24e5516c23dd","Type":"ContainerStarted","Data":"2af4adda251d8860e72902be0550ba23cb9bc78680682cf8031d069ad4a0ebde"}
Sep 11 17:34:10.694402 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:10.694341    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf" podStartSLOduration=1.694321566 podStartE2EDuration="1.694321566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:34:10.692591103 +0000 UTC m=+818.496763522" watchObservedRunningTime="2026-09-11 17:34:10.694321566 +0000 UTC m=+818.498494034"
Sep 11 17:34:10.711215 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:10.711115    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw" podStartSLOduration=2.369526208 podStartE2EDuration="2.711098029s" totalImagesPullingTime="341.571821ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:34:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:34:10.708577292 +0000 UTC m=+818.512749724" watchObservedRunningTime="2026-09-11 17:34:10.711098029 +0000 UTC m=+818.515270447"
Sep 11 17:34:10.727410 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:10.727350    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc" podStartSLOduration=1.727327593 podStartE2EDuration="1.727327593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:34:10.72615305 +0000 UTC m=+818.530325470" watchObservedRunningTime="2026-09-11 17:34:10.727327593 +0000 UTC m=+818.531500013"
Sep 11 17:34:16.690993 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:16.690959    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw"
Sep 11 17:34:21.687650 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:21.687613    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf"
Sep 11 17:34:21.688039 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:21.687703    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc"
Sep 11 17:34:45.909241 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:45.909204    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk"]
Sep 11 17:34:45.909751 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:45.909430    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk" podUID="51c1731d-e723-422e-8ef8-06860c00f657" containerName="payload-pre-processing" containerID="cri-o://94289ac0d51fad1dd34979e3b5b556be778230ef0070491210dd0842eb8b97df" gracePeriod=30
Sep 11 17:34:46.177116 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:46.177047    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk"
Sep 11 17:34:46.242519 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:46.242482    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51c1731d-e723-422e-8ef8-06860c00f657-plugins-config-volume\") pod \"51c1731d-e723-422e-8ef8-06860c00f657\" (UID: \"51c1731d-e723-422e-8ef8-06860c00f657\") "
Sep 11 17:34:46.242519 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:46.242518    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mh9zz\" (UniqueName: \"kubernetes.io/projected/51c1731d-e723-422e-8ef8-06860c00f657-kube-api-access-mh9zz\") pod \"51c1731d-e723-422e-8ef8-06860c00f657\" (UID: \"51c1731d-e723-422e-8ef8-06860c00f657\") "
Sep 11 17:34:46.242971 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:46.242912    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51c1731d-e723-422e-8ef8-06860c00f657-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "51c1731d-e723-422e-8ef8-06860c00f657" (UID: "51c1731d-e723-422e-8ef8-06860c00f657"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:34:46.244718 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:46.244687    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c1731d-e723-422e-8ef8-06860c00f657-kube-api-access-mh9zz" (OuterVolumeSpecName: "kube-api-access-mh9zz") pod "51c1731d-e723-422e-8ef8-06860c00f657" (UID: "51c1731d-e723-422e-8ef8-06860c00f657"). InnerVolumeSpecName "kube-api-access-mh9zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:34:46.344056 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:46.344022    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51c1731d-e723-422e-8ef8-06860c00f657-plugins-config-volume\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:34:46.344056 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:46.344049    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh9zz\" (UniqueName: \"kubernetes.io/projected/51c1731d-e723-422e-8ef8-06860c00f657-kube-api-access-mh9zz\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:34:46.860452 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:46.860416    2560 generic.go:356] "Generic (PLEG): container finished" podID="51c1731d-e723-422e-8ef8-06860c00f657" containerID="94289ac0d51fad1dd34979e3b5b556be778230ef0070491210dd0842eb8b97df" exitCode=0
Sep 11 17:34:46.860452 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:46.860456    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk"
Sep 11 17:34:46.860703 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:46.860486    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk" event={"ID":"51c1731d-e723-422e-8ef8-06860c00f657","Type":"ContainerDied","Data":"94289ac0d51fad1dd34979e3b5b556be778230ef0070491210dd0842eb8b97df"}
Sep 11 17:34:46.860703 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:46.860535    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk" event={"ID":"51c1731d-e723-422e-8ef8-06860c00f657","Type":"ContainerDied","Data":"432e4c2c477b8176d3ff59ac2994bc15b6829c78c110e8c677546e22a7f1ec62"}
Sep 11 17:34:46.860703 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:46.860563    2560 scope.go:117] "RemoveContainer" containerID="94289ac0d51fad1dd34979e3b5b556be778230ef0070491210dd0842eb8b97df"
Sep 11 17:34:46.869406 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:46.869384    2560 scope.go:117] "RemoveContainer" containerID="94289ac0d51fad1dd34979e3b5b556be778230ef0070491210dd0842eb8b97df"
Sep 11 17:34:46.869698 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:34:46.869673    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94289ac0d51fad1dd34979e3b5b556be778230ef0070491210dd0842eb8b97df\": container with ID starting with 94289ac0d51fad1dd34979e3b5b556be778230ef0070491210dd0842eb8b97df not found: ID does not exist" containerID="94289ac0d51fad1dd34979e3b5b556be778230ef0070491210dd0842eb8b97df"
Sep 11 17:34:46.869753 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:46.869700    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94289ac0d51fad1dd34979e3b5b556be778230ef0070491210dd0842eb8b97df"} err="failed to get container status \"94289ac0d51fad1dd34979e3b5b556be778230ef0070491210dd0842eb8b97df\": rpc error: code = NotFound desc = could not find container \"94289ac0d51fad1dd34979e3b5b556be778230ef0070491210dd0842eb8b97df\": container with ID starting with 94289ac0d51fad1dd34979e3b5b556be778230ef0070491210dd0842eb8b97df not found: ID does not exist"
Sep 11 17:34:46.876140 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:46.876116    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk"]
Sep 11 17:34:46.879946 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:46.879926    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3005e5-74c4568986-9qpgk"]
Sep 11 17:34:47.995605 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:47.995432    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw"]
Sep 11 17:34:47.996108 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:47.995684    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw" podUID="e4393699-ff06-4247-a0f2-24e5516c23dd" containerName="maas-api" containerID="cri-o://4b9d1d1b9f991de70a4763a725a509910580f20062c0bfff541e765a18b70cc6" gracePeriod=30
Sep 11 17:34:48.064088 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.064054    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf"]
Sep 11 17:34:48.064339 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.064282    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf" podUID="99e6b796-603d-483f-a7e2-49db822614f5" containerName="payload-processing" containerID="cri-o://507ba2fe90d94cf4dd1ca08adf6638d9daad0556efae7207e15a7139612880d2" gracePeriod=30
Sep 11 17:34:48.070220 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.070190    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc"]
Sep 11 17:34:48.070430 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.070407    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc" podUID="fe5732e6-da94-41bf-81bd-b6bddcc75857" containerName="payload-pre-processing" containerID="cri-o://2032363722a0e1d503bcacd162178b54a7f8aec36047140dd92aa5f6d4c3e6c1" gracePeriod=30
Sep 11 17:34:48.375948 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.375918    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw"
Sep 11 17:34:48.381982 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.381921    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf"
Sep 11 17:34:48.386704 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.386682    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc"
Sep 11 17:34:48.465018 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.464974    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99e6b796-603d-483f-a7e2-49db822614f5-plugins-config-volume\") pod \"99e6b796-603d-483f-a7e2-49db822614f5\" (UID: \"99e6b796-603d-483f-a7e2-49db822614f5\") "
Sep 11 17:34:48.465018 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.465022    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgx7r\" (UniqueName: \"kubernetes.io/projected/fe5732e6-da94-41bf-81bd-b6bddcc75857-kube-api-access-fgx7r\") pod \"fe5732e6-da94-41bf-81bd-b6bddcc75857\" (UID: \"fe5732e6-da94-41bf-81bd-b6bddcc75857\") "
Sep 11 17:34:48.465272 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.465068    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4393699-ff06-4247-a0f2-24e5516c23dd-metrics-certs\") pod \"e4393699-ff06-4247-a0f2-24e5516c23dd\" (UID: \"e4393699-ff06-4247-a0f2-24e5516c23dd\") "
Sep 11 17:34:48.465272 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.465106    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcdp7\" (UniqueName: \"kubernetes.io/projected/e4393699-ff06-4247-a0f2-24e5516c23dd-kube-api-access-xcdp7\") pod \"e4393699-ff06-4247-a0f2-24e5516c23dd\" (UID: \"e4393699-ff06-4247-a0f2-24e5516c23dd\") "
Sep 11 17:34:48.465272 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.465137    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5732e6-da94-41bf-81bd-b6bddcc75857-plugins-config-volume\") pod \"fe5732e6-da94-41bf-81bd-b6bddcc75857\" (UID: \"fe5732e6-da94-41bf-81bd-b6bddcc75857\") "
Sep 11 17:34:48.465272 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.465194    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4393699-ff06-4247-a0f2-24e5516c23dd-openshift-service-ca\") pod \"e4393699-ff06-4247-a0f2-24e5516c23dd\" (UID: \"e4393699-ff06-4247-a0f2-24e5516c23dd\") "
Sep 11 17:34:48.465496 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.465272    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4393699-ff06-4247-a0f2-24e5516c23dd-maas-api-tls\") pod \"e4393699-ff06-4247-a0f2-24e5516c23dd\" (UID: \"e4393699-ff06-4247-a0f2-24e5516c23dd\") "
Sep 11 17:34:48.465496 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.465318    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5s96n\" (UniqueName: \"kubernetes.io/projected/99e6b796-603d-483f-a7e2-49db822614f5-kube-api-access-5s96n\") pod \"99e6b796-603d-483f-a7e2-49db822614f5\" (UID: \"99e6b796-603d-483f-a7e2-49db822614f5\") "
Sep 11 17:34:48.465496 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.465411    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99e6b796-603d-483f-a7e2-49db822614f5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "99e6b796-603d-483f-a7e2-49db822614f5" (UID: "99e6b796-603d-483f-a7e2-49db822614f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:34:48.465656 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.465492    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe5732e6-da94-41bf-81bd-b6bddcc75857-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fe5732e6-da94-41bf-81bd-b6bddcc75857" (UID: "fe5732e6-da94-41bf-81bd-b6bddcc75857"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:34:48.465923 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.465799    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99e6b796-603d-483f-a7e2-49db822614f5-plugins-config-volume\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:34:48.465923 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.465837    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5732e6-da94-41bf-81bd-b6bddcc75857-plugins-config-volume\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:34:48.465923 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.465859    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4393699-ff06-4247-a0f2-24e5516c23dd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e4393699-ff06-4247-a0f2-24e5516c23dd" (UID: "e4393699-ff06-4247-a0f2-24e5516c23dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:34:48.467727 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.467697    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4393699-ff06-4247-a0f2-24e5516c23dd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e4393699-ff06-4247-a0f2-24e5516c23dd" (UID: "e4393699-ff06-4247-a0f2-24e5516c23dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:34:48.467838 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.467746    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5732e6-da94-41bf-81bd-b6bddcc75857-kube-api-access-fgx7r" (OuterVolumeSpecName: "kube-api-access-fgx7r") pod "fe5732e6-da94-41bf-81bd-b6bddcc75857" (UID: "fe5732e6-da94-41bf-81bd-b6bddcc75857"). InnerVolumeSpecName "kube-api-access-fgx7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:34:48.467987 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.467958    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4393699-ff06-4247-a0f2-24e5516c23dd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e4393699-ff06-4247-a0f2-24e5516c23dd" (UID: "e4393699-ff06-4247-a0f2-24e5516c23dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:34:48.468062 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.467988    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e6b796-603d-483f-a7e2-49db822614f5-kube-api-access-5s96n" (OuterVolumeSpecName: "kube-api-access-5s96n") pod "99e6b796-603d-483f-a7e2-49db822614f5" (UID: "99e6b796-603d-483f-a7e2-49db822614f5"). InnerVolumeSpecName "kube-api-access-5s96n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:34:48.468062 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.468002    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4393699-ff06-4247-a0f2-24e5516c23dd-kube-api-access-xcdp7" (OuterVolumeSpecName: "kube-api-access-xcdp7") pod "e4393699-ff06-4247-a0f2-24e5516c23dd" (UID: "e4393699-ff06-4247-a0f2-24e5516c23dd"). InnerVolumeSpecName "kube-api-access-xcdp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:34:48.567223 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.567134    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgx7r\" (UniqueName: \"kubernetes.io/projected/fe5732e6-da94-41bf-81bd-b6bddcc75857-kube-api-access-fgx7r\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:34:48.567223 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.567163    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4393699-ff06-4247-a0f2-24e5516c23dd-metrics-certs\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:34:48.567223 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.567172    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcdp7\" (UniqueName: \"kubernetes.io/projected/e4393699-ff06-4247-a0f2-24e5516c23dd-kube-api-access-xcdp7\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:34:48.567223 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.567182    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4393699-ff06-4247-a0f2-24e5516c23dd-openshift-service-ca\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:34:48.567223 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.567191    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4393699-ff06-4247-a0f2-24e5516c23dd-maas-api-tls\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:34:48.567223 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.567199    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s96n\" (UniqueName: \"kubernetes.io/projected/99e6b796-603d-483f-a7e2-49db822614f5-kube-api-access-5s96n\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:34:48.716235 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.716193    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51c1731d-e723-422e-8ef8-06860c00f657" path="/var/lib/kubelet/pods/51c1731d-e723-422e-8ef8-06860c00f657/volumes"
Sep 11 17:34:48.873529 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.873492    2560 generic.go:356] "Generic (PLEG): container finished" podID="99e6b796-603d-483f-a7e2-49db822614f5" containerID="507ba2fe90d94cf4dd1ca08adf6638d9daad0556efae7207e15a7139612880d2" exitCode=0
Sep 11 17:34:48.873695 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.873533    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf"
Sep 11 17:34:48.873695 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.873576    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf" event={"ID":"99e6b796-603d-483f-a7e2-49db822614f5","Type":"ContainerDied","Data":"507ba2fe90d94cf4dd1ca08adf6638d9daad0556efae7207e15a7139612880d2"}
Sep 11 17:34:48.873695 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.873627    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf" event={"ID":"99e6b796-603d-483f-a7e2-49db822614f5","Type":"ContainerDied","Data":"94da2d4b6a3058c39352455b3a0b5f10bae325cec893f8b4f4aa71508e36aaa5"}
Sep 11 17:34:48.873695 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.873655    2560 scope.go:117] "RemoveContainer" containerID="507ba2fe90d94cf4dd1ca08adf6638d9daad0556efae7207e15a7139612880d2"
Sep 11 17:34:48.874988 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.874968    2560 generic.go:356] "Generic (PLEG): container finished" podID="fe5732e6-da94-41bf-81bd-b6bddcc75857" containerID="2032363722a0e1d503bcacd162178b54a7f8aec36047140dd92aa5f6d4c3e6c1" exitCode=0
Sep 11 17:34:48.875118 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.875009    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc"
Sep 11 17:34:48.875118 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.875026    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc" event={"ID":"fe5732e6-da94-41bf-81bd-b6bddcc75857","Type":"ContainerDied","Data":"2032363722a0e1d503bcacd162178b54a7f8aec36047140dd92aa5f6d4c3e6c1"}
Sep 11 17:34:48.875118 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.875074    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc" event={"ID":"fe5732e6-da94-41bf-81bd-b6bddcc75857","Type":"ContainerDied","Data":"18079a9bf40c9729ab63c8432e7be48c528a320f2d9c748bd1b2449b81d47160"}
Sep 11 17:34:48.876571 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.876550    2560 generic.go:356] "Generic (PLEG): container finished" podID="e4393699-ff06-4247-a0f2-24e5516c23dd" containerID="4b9d1d1b9f991de70a4763a725a509910580f20062c0bfff541e765a18b70cc6" exitCode=0
Sep 11 17:34:48.876687 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.876578    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw"
Sep 11 17:34:48.876687 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.876635    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw" event={"ID":"e4393699-ff06-4247-a0f2-24e5516c23dd","Type":"ContainerDied","Data":"4b9d1d1b9f991de70a4763a725a509910580f20062c0bfff541e765a18b70cc6"}
Sep 11 17:34:48.876687 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.876674    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw" event={"ID":"e4393699-ff06-4247-a0f2-24e5516c23dd","Type":"ContainerDied","Data":"2af4adda251d8860e72902be0550ba23cb9bc78680682cf8031d069ad4a0ebde"}
Sep 11 17:34:48.883324 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.883305    2560 scope.go:117] "RemoveContainer" containerID="507ba2fe90d94cf4dd1ca08adf6638d9daad0556efae7207e15a7139612880d2"
Sep 11 17:34:48.883586 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:34:48.883565    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"507ba2fe90d94cf4dd1ca08adf6638d9daad0556efae7207e15a7139612880d2\": container with ID starting with 507ba2fe90d94cf4dd1ca08adf6638d9daad0556efae7207e15a7139612880d2 not found: ID does not exist" containerID="507ba2fe90d94cf4dd1ca08adf6638d9daad0556efae7207e15a7139612880d2"
Sep 11 17:34:48.883629 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.883594    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"507ba2fe90d94cf4dd1ca08adf6638d9daad0556efae7207e15a7139612880d2"} err="failed to get container status \"507ba2fe90d94cf4dd1ca08adf6638d9daad0556efae7207e15a7139612880d2\": rpc error: code = NotFound desc = could not find container \"507ba2fe90d94cf4dd1ca08adf6638d9daad0556efae7207e15a7139612880d2\": container with ID starting with 507ba2fe90d94cf4dd1ca08adf6638d9daad0556efae7207e15a7139612880d2 not found: ID does not exist"
Sep 11 17:34:48.883629 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.883610    2560 scope.go:117] "RemoveContainer" containerID="2032363722a0e1d503bcacd162178b54a7f8aec36047140dd92aa5f6d4c3e6c1"
Sep 11 17:34:48.891813 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.891793    2560 scope.go:117] "RemoveContainer" containerID="2032363722a0e1d503bcacd162178b54a7f8aec36047140dd92aa5f6d4c3e6c1"
Sep 11 17:34:48.892130 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:34:48.892097    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2032363722a0e1d503bcacd162178b54a7f8aec36047140dd92aa5f6d4c3e6c1\": container with ID starting with 2032363722a0e1d503bcacd162178b54a7f8aec36047140dd92aa5f6d4c3e6c1 not found: ID does not exist" containerID="2032363722a0e1d503bcacd162178b54a7f8aec36047140dd92aa5f6d4c3e6c1"
Sep 11 17:34:48.892223 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.892134    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2032363722a0e1d503bcacd162178b54a7f8aec36047140dd92aa5f6d4c3e6c1"} err="failed to get container status \"2032363722a0e1d503bcacd162178b54a7f8aec36047140dd92aa5f6d4c3e6c1\": rpc error: code = NotFound desc = could not find container \"2032363722a0e1d503bcacd162178b54a7f8aec36047140dd92aa5f6d4c3e6c1\": container with ID starting with 2032363722a0e1d503bcacd162178b54a7f8aec36047140dd92aa5f6d4c3e6c1 not found: ID does not exist"
Sep 11 17:34:48.892223 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.892157    2560 scope.go:117] "RemoveContainer" containerID="4b9d1d1b9f991de70a4763a725a509910580f20062c0bfff541e765a18b70cc6"
Sep 11 17:34:48.898302 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.898279    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf"]
Sep 11 17:34:48.900978 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.900960    2560 scope.go:117] "RemoveContainer" containerID="4b9d1d1b9f991de70a4763a725a509910580f20062c0bfff541e765a18b70cc6"
Sep 11 17:34:48.901257 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:34:48.901235    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b9d1d1b9f991de70a4763a725a509910580f20062c0bfff541e765a18b70cc6\": container with ID starting with 4b9d1d1b9f991de70a4763a725a509910580f20062c0bfff541e765a18b70cc6 not found: ID does not exist" containerID="4b9d1d1b9f991de70a4763a725a509910580f20062c0bfff541e765a18b70cc6"
Sep 11 17:34:48.901337 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.901265    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b9d1d1b9f991de70a4763a725a509910580f20062c0bfff541e765a18b70cc6"} err="failed to get container status \"4b9d1d1b9f991de70a4763a725a509910580f20062c0bfff541e765a18b70cc6\": rpc error: code = NotFound desc = could not find container \"4b9d1d1b9f991de70a4763a725a509910580f20062c0bfff541e765a18b70cc6\": container with ID starting with 4b9d1d1b9f991de70a4763a725a509910580f20062c0bfff541e765a18b70cc6 not found: ID does not exist"
Sep 11 17:34:48.902275 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.902255    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1d8713f-664dcf55bf-bxdbf"]
Sep 11 17:34:48.912957 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.912935    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw"]
Sep 11 17:34:48.917319 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.917298    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1d8713f-5fb647c4dd-nq7nw"]
Sep 11 17:34:48.929340 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.929317    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc"]
Sep 11 17:34:48.931982 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:48.931961    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1d8713f-69765d-g7jjc"]
Sep 11 17:34:50.716360 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:50.716326    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99e6b796-603d-483f-a7e2-49db822614f5" path="/var/lib/kubelet/pods/99e6b796-603d-483f-a7e2-49db822614f5/volumes"
Sep 11 17:34:50.716792 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:50.716777    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4393699-ff06-4247-a0f2-24e5516c23dd" path="/var/lib/kubelet/pods/e4393699-ff06-4247-a0f2-24e5516c23dd/volumes"
Sep 11 17:34:50.717115 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:50.717103    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe5732e6-da94-41bf-81bd-b6bddcc75857" path="/var/lib/kubelet/pods/fe5732e6-da94-41bf-81bd-b6bddcc75857/volumes"
Sep 11 17:34:53.997318 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:53.997279    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7"]
Sep 11 17:34:53.997932 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:53.997913    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe5732e6-da94-41bf-81bd-b6bddcc75857" containerName="payload-pre-processing"
Sep 11 17:34:53.998007 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:53.997934    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe5732e6-da94-41bf-81bd-b6bddcc75857" containerName="payload-pre-processing"
Sep 11 17:34:53.998007 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:53.997948    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4393699-ff06-4247-a0f2-24e5516c23dd" containerName="maas-api"
Sep 11 17:34:53.998007 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:53.997958    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4393699-ff06-4247-a0f2-24e5516c23dd" containerName="maas-api"
Sep 11 17:34:53.998007 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:53.997987    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99e6b796-603d-483f-a7e2-49db822614f5" containerName="payload-processing"
Sep 11 17:34:53.998007 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:53.997997    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="99e6b796-603d-483f-a7e2-49db822614f5" containerName="payload-processing"
Sep 11 17:34:53.998007 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:53.998009    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51c1731d-e723-422e-8ef8-06860c00f657" containerName="payload-pre-processing"
Sep 11 17:34:53.998320 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:53.998018    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="51c1731d-e723-422e-8ef8-06860c00f657" containerName="payload-pre-processing"
Sep 11 17:34:53.998320 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:53.998127    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="51c1731d-e723-422e-8ef8-06860c00f657" containerName="payload-pre-processing"
Sep 11 17:34:53.998320 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:53.998138    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4393699-ff06-4247-a0f2-24e5516c23dd" containerName="maas-api"
Sep 11 17:34:53.998320 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:53.998152    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe5732e6-da94-41bf-81bd-b6bddcc75857" containerName="payload-pre-processing"
Sep 11 17:34:53.998320 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:53.998161    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="99e6b796-603d-483f-a7e2-49db822614f5" containerName="payload-processing"
Sep 11 17:34:54.003158 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.003135    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7"
Sep 11 17:34:54.005600 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.005579    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-c4a3e530-serving-cert\""
Sep 11 17:34:54.009511 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.009489    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7"]
Sep 11 17:34:54.122227 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.122194    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-maas-api-tls\") pod \"maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7\" (UID: \"5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7"
Sep 11 17:34:54.122227 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.122228    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-openshift-service-ca\") pod \"maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7\" (UID: \"5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7"
Sep 11 17:34:54.122461 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.122264    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxfqn\" (UniqueName: \"kubernetes.io/projected/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-kube-api-access-lxfqn\") pod \"maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7\" (UID: \"5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7"
Sep 11 17:34:54.122461 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.122338    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-metrics-certs\") pod \"maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7\" (UID: \"5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7"
Sep 11 17:34:54.223307 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.223265    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-metrics-certs\") pod \"maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7\" (UID: \"5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7"
Sep 11 17:34:54.223471 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.223359    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-maas-api-tls\") pod \"maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7\" (UID: \"5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7"
Sep 11 17:34:54.223471 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.223379    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-openshift-service-ca\") pod \"maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7\" (UID: \"5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7"
Sep 11 17:34:54.223471 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.223412    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxfqn\" (UniqueName: \"kubernetes.io/projected/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-kube-api-access-lxfqn\") pod \"maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7\" (UID: \"5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7"
Sep 11 17:34:54.223657 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:34:54.223519    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-serving-cert: secret "maas-api-e2e-derive-c4a3e530-serving-cert" not found
Sep 11 17:34:54.223657 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:34:54.223602    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-maas-api-tls podName:5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3 nodeName:}" failed. No retries permitted until 2026-09-11 17:34:54.723580355 +0000 UTC m=+862.527752757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-maas-api-tls") pod "maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7" (UID: "5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3") : secret "maas-api-e2e-derive-c4a3e530-serving-cert" not found
Sep 11 17:34:54.224088 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.224068    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-openshift-service-ca\") pod \"maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7\" (UID: \"5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7"
Sep 11 17:34:54.225869 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.225845    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-metrics-certs\") pod \"maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7\" (UID: \"5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7"
Sep 11 17:34:54.234303 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.234278    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxfqn\" (UniqueName: \"kubernetes.io/projected/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-kube-api-access-lxfqn\") pod \"maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7\" (UID: \"5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7"
Sep 11 17:34:54.590606 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.590565    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq"]
Sep 11 17:34:54.594581 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.594555    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq"
Sep 11 17:34:54.597010 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.596984    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-c4a3e530\""
Sep 11 17:34:54.606268 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.606240    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq"]
Sep 11 17:34:54.628081 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.628046    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg"]
Sep 11 17:34:54.631769 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.631746    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg"
Sep 11 17:34:54.634305 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.634283    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-c4a3e530-dockercfg-czrzh\""
Sep 11 17:34:54.646071 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.646040    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg"]
Sep 11 17:34:54.728398 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.728362    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0ac508a-8ab1-4225-b4c7-62cc83887d34-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq\" (UID: \"d0ac508a-8ab1-4225-b4c7-62cc83887d34\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq"
Sep 11 17:34:54.728591 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.728436    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml87p\" (UniqueName: \"kubernetes.io/projected/d0ac508a-8ab1-4225-b4c7-62cc83887d34-kube-api-access-ml87p\") pod \"payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq\" (UID: \"d0ac508a-8ab1-4225-b4c7-62cc83887d34\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq"
Sep 11 17:34:54.728591 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.728514    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-maas-api-tls\") pod \"maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7\" (UID: \"5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7"
Sep 11 17:34:54.728591 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.728540    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74bb599a-fa8b-4df1-a21e-ffba1f11e94a-plugins-config-volume\") pod \"payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg\" (UID: \"74bb599a-fa8b-4df1-a21e-ffba1f11e94a\") " pod="openshift-ingress/payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg"
Sep 11 17:34:54.728591 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.728577    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhhrz\" (UniqueName: \"kubernetes.io/projected/74bb599a-fa8b-4df1-a21e-ffba1f11e94a-kube-api-access-vhhrz\") pod \"payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg\" (UID: \"74bb599a-fa8b-4df1-a21e-ffba1f11e94a\") " pod="openshift-ingress/payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg"
Sep 11 17:34:54.728800 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:34:54.728682    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-serving-cert: secret "maas-api-e2e-derive-c4a3e530-serving-cert" not found
Sep 11 17:34:54.728800 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:34:54.728763    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-maas-api-tls podName:5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3 nodeName:}" failed. No retries permitted until 2026-09-11 17:34:55.728742481 +0000 UTC m=+863.532914883 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-maas-api-tls") pod "maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7" (UID: "5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3") : secret "maas-api-e2e-derive-c4a3e530-serving-cert" not found
Sep 11 17:34:54.829168 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.829134    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0ac508a-8ab1-4225-b4c7-62cc83887d34-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq\" (UID: \"d0ac508a-8ab1-4225-b4c7-62cc83887d34\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq"
Sep 11 17:34:54.829339 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.829190    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml87p\" (UniqueName: \"kubernetes.io/projected/d0ac508a-8ab1-4225-b4c7-62cc83887d34-kube-api-access-ml87p\") pod \"payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq\" (UID: \"d0ac508a-8ab1-4225-b4c7-62cc83887d34\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq"
Sep 11 17:34:54.829339 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.829247    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74bb599a-fa8b-4df1-a21e-ffba1f11e94a-plugins-config-volume\") pod \"payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg\" (UID: \"74bb599a-fa8b-4df1-a21e-ffba1f11e94a\") " pod="openshift-ingress/payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg"
Sep 11 17:34:54.829339 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.829270    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhhrz\" (UniqueName: \"kubernetes.io/projected/74bb599a-fa8b-4df1-a21e-ffba1f11e94a-kube-api-access-vhhrz\") pod \"payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg\" (UID: \"74bb599a-fa8b-4df1-a21e-ffba1f11e94a\") " pod="openshift-ingress/payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg"
Sep 11 17:34:54.829829 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.829809    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0ac508a-8ab1-4225-b4c7-62cc83887d34-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq\" (UID: \"d0ac508a-8ab1-4225-b4c7-62cc83887d34\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq"
Sep 11 17:34:54.829965 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.829948    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74bb599a-fa8b-4df1-a21e-ffba1f11e94a-plugins-config-volume\") pod \"payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg\" (UID: \"74bb599a-fa8b-4df1-a21e-ffba1f11e94a\") " pod="openshift-ingress/payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg"
Sep 11 17:34:54.838176 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.838146    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml87p\" (UniqueName: \"kubernetes.io/projected/d0ac508a-8ab1-4225-b4c7-62cc83887d34-kube-api-access-ml87p\") pod \"payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq\" (UID: \"d0ac508a-8ab1-4225-b4c7-62cc83887d34\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq"
Sep 11 17:34:54.838618 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.838595    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhhrz\" (UniqueName: \"kubernetes.io/projected/74bb599a-fa8b-4df1-a21e-ffba1f11e94a-kube-api-access-vhhrz\") pod \"payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg\" (UID: \"74bb599a-fa8b-4df1-a21e-ffba1f11e94a\") " pod="openshift-ingress/payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg"
Sep 11 17:34:54.908747 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.908664    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq"
Sep 11 17:34:54.945854 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:54.945810    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg"
Sep 11 17:34:55.185714 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:55.185686    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq"]
Sep 11 17:34:55.187621 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:34:55.187595    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ac508a_8ab1_4225_b4c7_62cc83887d34.slice/crio-cd44ed0dcd63494fc1f2675650a4d649a20c3ecb17595d41af206a5c4954edc2 WatchSource:0}: Error finding container cd44ed0dcd63494fc1f2675650a4d649a20c3ecb17595d41af206a5c4954edc2: Status 404 returned error can't find the container with id cd44ed0dcd63494fc1f2675650a4d649a20c3ecb17595d41af206a5c4954edc2
Sep 11 17:34:55.228831 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:55.228807    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg"]
Sep 11 17:34:55.231224 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:34:55.231197    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74bb599a_fa8b_4df1_a21e_ffba1f11e94a.slice/crio-9909fc08d8d936abea069ac1e61831fce42cee186c34e5ebf58fa0ee0e656259 WatchSource:0}: Error finding container 9909fc08d8d936abea069ac1e61831fce42cee186c34e5ebf58fa0ee0e656259: Status 404 returned error can't find the container with id 9909fc08d8d936abea069ac1e61831fce42cee186c34e5ebf58fa0ee0e656259
Sep 11 17:34:55.737488 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:55.737432    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-maas-api-tls\") pod \"maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7\" (UID: \"5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7"
Sep 11 17:34:55.739928 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:55.739887    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-maas-api-tls\") pod \"maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7\" (UID: \"5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7"
Sep 11 17:34:55.817724 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:55.817683    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7"
Sep 11 17:34:55.917833 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:55.917797    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq" event={"ID":"d0ac508a-8ab1-4225-b4c7-62cc83887d34","Type":"ContainerStarted","Data":"a98e635e7de7d16d2f35e77b510a7ca2792c807e5e2d25bd515ee25641a271c6"}
Sep 11 17:34:55.918043 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:55.917856    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq" event={"ID":"d0ac508a-8ab1-4225-b4c7-62cc83887d34","Type":"ContainerStarted","Data":"cd44ed0dcd63494fc1f2675650a4d649a20c3ecb17595d41af206a5c4954edc2"}
Sep 11 17:34:55.918043 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:55.917882    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq"
Sep 11 17:34:55.919668 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:55.919632    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg" event={"ID":"74bb599a-fa8b-4df1-a21e-ffba1f11e94a","Type":"ContainerStarted","Data":"ac0742b41155c989906e98118efe85323da05b446e0c355cf61e34fccdac51aa"}
Sep 11 17:34:55.919781 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:55.919685    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg" event={"ID":"74bb599a-fa8b-4df1-a21e-ffba1f11e94a","Type":"ContainerStarted","Data":"9909fc08d8d936abea069ac1e61831fce42cee186c34e5ebf58fa0ee0e656259"}
Sep 11 17:34:55.919781 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:55.919712    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg"
Sep 11 17:34:55.936708 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:55.936663    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq" podStartSLOduration=1.936644933 podStartE2EDuration="1.936644933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:34:55.933875333 +0000 UTC m=+863.738047775" watchObservedRunningTime="2026-09-11 17:34:55.936644933 +0000 UTC m=+863.740817411"
Sep 11 17:34:55.955664 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:55.955432    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg" podStartSLOduration=1.955378279 podStartE2EDuration="1.955378279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:34:55.949967553 +0000 UTC m=+863.754139971" watchObservedRunningTime="2026-09-11 17:34:55.955378279 +0000 UTC m=+863.759550704"
Sep 11 17:34:56.093813 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:56.093782    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7"]
Sep 11 17:34:56.095361 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:34:56.095331    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5147cc2b_5e3e_4a6c_9890_a8ba5618c8c3.slice/crio-66703807d893e4e70ed3f6a74f0e677ba0b666b4c3a329584d22a163ce10f465 WatchSource:0}: Error finding container 66703807d893e4e70ed3f6a74f0e677ba0b666b4c3a329584d22a163ce10f465: Status 404 returned error can't find the container with id 66703807d893e4e70ed3f6a74f0e677ba0b666b4c3a329584d22a163ce10f465
Sep 11 17:34:56.926374 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:56.926269    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7" event={"ID":"5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3","Type":"ContainerStarted","Data":"6ea50031ce0c427aa53b69fc1d566d71bb67552359bf98e11e3ee298774b19a2"}
Sep 11 17:34:56.926810 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:56.926789    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7"
Sep 11 17:34:56.926856 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:56.926830    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7" event={"ID":"5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3","Type":"ContainerStarted","Data":"66703807d893e4e70ed3f6a74f0e677ba0b666b4c3a329584d22a163ce10f465"}
Sep 11 17:34:56.946503 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:34:56.946445    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7" podStartSLOduration=3.459588107 podStartE2EDuration="3.946423417s" totalImagesPullingTime="486.83531ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:34:56.943952368 +0000 UTC m=+864.748124786" watchObservedRunningTime="2026-09-11 17:34:56.946423417 +0000 UTC m=+864.750595838"
Sep 11 17:35:03.941270 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:03.941241    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7"
Sep 11 17:35:06.928996 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:06.928966    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg"
Sep 11 17:35:06.929357 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:06.929032    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq"
Sep 11 17:35:24.485131 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:24.485095    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7"]
Sep 11 17:35:24.485778 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:24.485410    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7" podUID="5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3" containerName="maas-api" containerID="cri-o://6ea50031ce0c427aa53b69fc1d566d71bb67552359bf98e11e3ee298774b19a2" gracePeriod=30
Sep 11 17:35:24.545407 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:24.545368    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg"]
Sep 11 17:35:24.545659 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:24.545632    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg" podUID="74bb599a-fa8b-4df1-a21e-ffba1f11e94a" containerName="payload-processing" containerID="cri-o://ac0742b41155c989906e98118efe85323da05b446e0c355cf61e34fccdac51aa" gracePeriod=30
Sep 11 17:35:24.553911 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:24.553865    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq"]
Sep 11 17:35:24.554188 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:24.554137    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq" podUID="d0ac508a-8ab1-4225-b4c7-62cc83887d34" containerName="payload-pre-processing" containerID="cri-o://a98e635e7de7d16d2f35e77b510a7ca2792c807e5e2d25bd515ee25641a271c6" gracePeriod=30
Sep 11 17:35:24.885586 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:24.885556    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg"
Sep 11 17:35:24.890039 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:24.890015    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq"
Sep 11 17:35:24.897215 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:24.897192    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7"
Sep 11 17:35:24.905098 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:24.905071    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-metrics-certs\") pod \"5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3\" (UID: \"5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3\") "
Sep 11 17:35:24.905233 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:24.905114    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhhrz\" (UniqueName: \"kubernetes.io/projected/74bb599a-fa8b-4df1-a21e-ffba1f11e94a-kube-api-access-vhhrz\") pod \"74bb599a-fa8b-4df1-a21e-ffba1f11e94a\" (UID: \"74bb599a-fa8b-4df1-a21e-ffba1f11e94a\") "
Sep 11 17:35:24.905233 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:24.905164    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ml87p\" (UniqueName: \"kubernetes.io/projected/d0ac508a-8ab1-4225-b4c7-62cc83887d34-kube-api-access-ml87p\") pod \"d0ac508a-8ab1-4225-b4c7-62cc83887d34\" (UID: \"d0ac508a-8ab1-4225-b4c7-62cc83887d34\") "
Sep 11 17:35:24.905233 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:24.905187    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74bb599a-fa8b-4df1-a21e-ffba1f11e94a-plugins-config-volume\") pod \"74bb599a-fa8b-4df1-a21e-ffba1f11e94a\" (UID: \"74bb599a-fa8b-4df1-a21e-ffba1f11e94a\") "
Sep 11 17:35:24.905233 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:24.905229    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-openshift-service-ca\") pod \"5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3\" (UID: \"5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3\") "
Sep 11 17:35:24.905466 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:24.905270    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0ac508a-8ab1-4225-b4c7-62cc83887d34-plugins-config-volume\") pod \"d0ac508a-8ab1-4225-b4c7-62cc83887d34\" (UID: \"d0ac508a-8ab1-4225-b4c7-62cc83887d34\") "
Sep 11 17:35:24.905466 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:24.905310    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-maas-api-tls\") pod \"5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3\" (UID: \"5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3\") "
Sep 11 17:35:24.905466 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:24.905356    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxfqn\" (UniqueName: \"kubernetes.io/projected/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-kube-api-access-lxfqn\") pod \"5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3\" (UID: \"5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3\") "
Sep 11 17:35:24.905620 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:24.905548    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74bb599a-fa8b-4df1-a21e-ffba1f11e94a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "74bb599a-fa8b-4df1-a21e-ffba1f11e94a" (UID: "74bb599a-fa8b-4df1-a21e-ffba1f11e94a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:35:24.905702 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:24.905684    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74bb599a-fa8b-4df1-a21e-ffba1f11e94a-plugins-config-volume\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:35:24.905845 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:24.905815    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0ac508a-8ab1-4225-b4c7-62cc83887d34-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d0ac508a-8ab1-4225-b4c7-62cc83887d34" (UID: "d0ac508a-8ab1-4225-b4c7-62cc83887d34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:35:24.906043 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:24.906016    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3" (UID: "5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:35:24.907836 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:24.907766    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74bb599a-fa8b-4df1-a21e-ffba1f11e94a-kube-api-access-vhhrz" (OuterVolumeSpecName: "kube-api-access-vhhrz") pod "74bb599a-fa8b-4df1-a21e-ffba1f11e94a" (UID: "74bb599a-fa8b-4df1-a21e-ffba1f11e94a"). InnerVolumeSpecName "kube-api-access-vhhrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:35:24.907836 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:24.907801    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0ac508a-8ab1-4225-b4c7-62cc83887d34-kube-api-access-ml87p" (OuterVolumeSpecName: "kube-api-access-ml87p") pod "d0ac508a-8ab1-4225-b4c7-62cc83887d34" (UID: "d0ac508a-8ab1-4225-b4c7-62cc83887d34"). InnerVolumeSpecName "kube-api-access-ml87p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:35:24.907836 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:24.907813    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3" (UID: "5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:35:24.908260 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:24.907960    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3" (UID: "5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:35:24.908260 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:24.908132    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-kube-api-access-lxfqn" (OuterVolumeSpecName: "kube-api-access-lxfqn") pod "5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3" (UID: "5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3"). InnerVolumeSpecName "kube-api-access-lxfqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:35:25.006715 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.006613    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-metrics-certs\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:35:25.006715 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.006648    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhhrz\" (UniqueName: \"kubernetes.io/projected/74bb599a-fa8b-4df1-a21e-ffba1f11e94a-kube-api-access-vhhrz\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:35:25.006715 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.006661    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml87p\" (UniqueName: \"kubernetes.io/projected/d0ac508a-8ab1-4225-b4c7-62cc83887d34-kube-api-access-ml87p\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:35:25.006715 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.006675    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-openshift-service-ca\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:35:25.006715 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.006690    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0ac508a-8ab1-4225-b4c7-62cc83887d34-plugins-config-volume\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:35:25.006715 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.006702    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-maas-api-tls\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:35:25.006715 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.006715    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxfqn\" (UniqueName: \"kubernetes.io/projected/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3-kube-api-access-lxfqn\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:35:25.072583 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.072550    2560 generic.go:356] "Generic (PLEG): container finished" podID="d0ac508a-8ab1-4225-b4c7-62cc83887d34" containerID="a98e635e7de7d16d2f35e77b510a7ca2792c807e5e2d25bd515ee25641a271c6" exitCode=0
Sep 11 17:35:25.072769 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.072593    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq"
Sep 11 17:35:25.072769 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.072630    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq" event={"ID":"d0ac508a-8ab1-4225-b4c7-62cc83887d34","Type":"ContainerDied","Data":"a98e635e7de7d16d2f35e77b510a7ca2792c807e5e2d25bd515ee25641a271c6"}
Sep 11 17:35:25.072769 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.072684    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq" event={"ID":"d0ac508a-8ab1-4225-b4c7-62cc83887d34","Type":"ContainerDied","Data":"cd44ed0dcd63494fc1f2675650a4d649a20c3ecb17595d41af206a5c4954edc2"}
Sep 11 17:35:25.072769 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.072708    2560 scope.go:117] "RemoveContainer" containerID="a98e635e7de7d16d2f35e77b510a7ca2792c807e5e2d25bd515ee25641a271c6"
Sep 11 17:35:25.074170 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.074146    2560 generic.go:356] "Generic (PLEG): container finished" podID="74bb599a-fa8b-4df1-a21e-ffba1f11e94a" containerID="ac0742b41155c989906e98118efe85323da05b446e0c355cf61e34fccdac51aa" exitCode=0
Sep 11 17:35:25.074286 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.074177    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg"
Sep 11 17:35:25.074286 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.074249    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg" event={"ID":"74bb599a-fa8b-4df1-a21e-ffba1f11e94a","Type":"ContainerDied","Data":"ac0742b41155c989906e98118efe85323da05b446e0c355cf61e34fccdac51aa"}
Sep 11 17:35:25.074405 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.074290    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg" event={"ID":"74bb599a-fa8b-4df1-a21e-ffba1f11e94a","Type":"ContainerDied","Data":"9909fc08d8d936abea069ac1e61831fce42cee186c34e5ebf58fa0ee0e656259"}
Sep 11 17:35:25.075738 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.075706    2560 generic.go:356] "Generic (PLEG): container finished" podID="5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3" containerID="6ea50031ce0c427aa53b69fc1d566d71bb67552359bf98e11e3ee298774b19a2" exitCode=0
Sep 11 17:35:25.075861 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.075752    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7"
Sep 11 17:35:25.075861 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.075759    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7" event={"ID":"5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3","Type":"ContainerDied","Data":"6ea50031ce0c427aa53b69fc1d566d71bb67552359bf98e11e3ee298774b19a2"}
Sep 11 17:35:25.075861 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.075798    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7" event={"ID":"5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3","Type":"ContainerDied","Data":"66703807d893e4e70ed3f6a74f0e677ba0b666b4c3a329584d22a163ce10f465"}
Sep 11 17:35:25.084661 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.084634    2560 scope.go:117] "RemoveContainer" containerID="a98e635e7de7d16d2f35e77b510a7ca2792c807e5e2d25bd515ee25641a271c6"
Sep 11 17:35:25.085066 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:35:25.085043    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a98e635e7de7d16d2f35e77b510a7ca2792c807e5e2d25bd515ee25641a271c6\": container with ID starting with a98e635e7de7d16d2f35e77b510a7ca2792c807e5e2d25bd515ee25641a271c6 not found: ID does not exist" containerID="a98e635e7de7d16d2f35e77b510a7ca2792c807e5e2d25bd515ee25641a271c6"
Sep 11 17:35:25.085185 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.085077    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a98e635e7de7d16d2f35e77b510a7ca2792c807e5e2d25bd515ee25641a271c6"} err="failed to get container status \"a98e635e7de7d16d2f35e77b510a7ca2792c807e5e2d25bd515ee25641a271c6\": rpc error: code = NotFound desc = could not find container \"a98e635e7de7d16d2f35e77b510a7ca2792c807e5e2d25bd515ee25641a271c6\": container with ID starting with a98e635e7de7d16d2f35e77b510a7ca2792c807e5e2d25bd515ee25641a271c6 not found: ID does not exist"
Sep 11 17:35:25.085185 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.085101    2560 scope.go:117] "RemoveContainer" containerID="ac0742b41155c989906e98118efe85323da05b446e0c355cf61e34fccdac51aa"
Sep 11 17:35:25.093792 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.093774    2560 scope.go:117] "RemoveContainer" containerID="ac0742b41155c989906e98118efe85323da05b446e0c355cf61e34fccdac51aa"
Sep 11 17:35:25.094122 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:35:25.094092    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac0742b41155c989906e98118efe85323da05b446e0c355cf61e34fccdac51aa\": container with ID starting with ac0742b41155c989906e98118efe85323da05b446e0c355cf61e34fccdac51aa not found: ID does not exist" containerID="ac0742b41155c989906e98118efe85323da05b446e0c355cf61e34fccdac51aa"
Sep 11 17:35:25.094180 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.094132    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac0742b41155c989906e98118efe85323da05b446e0c355cf61e34fccdac51aa"} err="failed to get container status \"ac0742b41155c989906e98118efe85323da05b446e0c355cf61e34fccdac51aa\": rpc error: code = NotFound desc = could not find container \"ac0742b41155c989906e98118efe85323da05b446e0c355cf61e34fccdac51aa\": container with ID starting with ac0742b41155c989906e98118efe85323da05b446e0c355cf61e34fccdac51aa not found: ID does not exist"
Sep 11 17:35:25.094180 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.094151    2560 scope.go:117] "RemoveContainer" containerID="6ea50031ce0c427aa53b69fc1d566d71bb67552359bf98e11e3ee298774b19a2"
Sep 11 17:35:25.102032 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.102009    2560 scope.go:117] "RemoveContainer" containerID="6ea50031ce0c427aa53b69fc1d566d71bb67552359bf98e11e3ee298774b19a2"
Sep 11 17:35:25.102283 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:35:25.102267    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ea50031ce0c427aa53b69fc1d566d71bb67552359bf98e11e3ee298774b19a2\": container with ID starting with 6ea50031ce0c427aa53b69fc1d566d71bb67552359bf98e11e3ee298774b19a2 not found: ID does not exist" containerID="6ea50031ce0c427aa53b69fc1d566d71bb67552359bf98e11e3ee298774b19a2"
Sep 11 17:35:25.102342 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.102291    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea50031ce0c427aa53b69fc1d566d71bb67552359bf98e11e3ee298774b19a2"} err="failed to get container status \"6ea50031ce0c427aa53b69fc1d566d71bb67552359bf98e11e3ee298774b19a2\": rpc error: code = NotFound desc = could not find container \"6ea50031ce0c427aa53b69fc1d566d71bb67552359bf98e11e3ee298774b19a2\": container with ID starting with 6ea50031ce0c427aa53b69fc1d566d71bb67552359bf98e11e3ee298774b19a2 not found: ID does not exist"
Sep 11 17:35:25.106715 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.106693    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg"]
Sep 11 17:35:25.120192 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.120167    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c4a3e530-7c7f784fbf-b49gg"]
Sep 11 17:35:25.133871 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.133839    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq"]
Sep 11 17:35:25.139370 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.139350    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c4a3e530-56847d5798-74gnq"]
Sep 11 17:35:25.156763 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.156735    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7"]
Sep 11 17:35:25.160826 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:25.160799    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c4a3e530-645668dcc4-jlbg7"]
Sep 11 17:35:26.716143 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:26.716108    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3" path="/var/lib/kubelet/pods/5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3/volumes"
Sep 11 17:35:26.716538 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:26.716523    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74bb599a-fa8b-4df1-a21e-ffba1f11e94a" path="/var/lib/kubelet/pods/74bb599a-fa8b-4df1-a21e-ffba1f11e94a/volumes"
Sep 11 17:35:26.716885 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:26.716872    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0ac508a-8ab1-4225-b4c7-62cc83887d34" path="/var/lib/kubelet/pods/d0ac508a-8ab1-4225-b4c7-62cc83887d34/volumes"
Sep 11 17:35:32.634606 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:32.634574    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6fcjs_477cac01-ab9e-4441-b07d-b7e53401c616/console-operator/1.log"
Sep 11 17:35:32.637055 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:32.637032    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6fcjs_477cac01-ab9e-4441-b07d-b7e53401c616/console-operator/1.log"
Sep 11 17:35:49.015961 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.015914    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8"]
Sep 11 17:35:49.018575 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.016475    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0ac508a-8ab1-4225-b4c7-62cc83887d34" containerName="payload-pre-processing"
Sep 11 17:35:49.018575 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.016491    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0ac508a-8ab1-4225-b4c7-62cc83887d34" containerName="payload-pre-processing"
Sep 11 17:35:49.018575 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.016499    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74bb599a-fa8b-4df1-a21e-ffba1f11e94a" containerName="payload-processing"
Sep 11 17:35:49.018575 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.016505    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="74bb599a-fa8b-4df1-a21e-ffba1f11e94a" containerName="payload-processing"
Sep 11 17:35:49.018575 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.016517    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3" containerName="maas-api"
Sep 11 17:35:49.018575 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.016523    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3" containerName="maas-api"
Sep 11 17:35:49.018575 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.016586    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5147cc2b-5e3e-4a6c-9890-a8ba5618c8c3" containerName="maas-api"
Sep 11 17:35:49.018575 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.016596    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0ac508a-8ab1-4225-b4c7-62cc83887d34" containerName="payload-pre-processing"
Sep 11 17:35:49.018575 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.016602    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="74bb599a-fa8b-4df1-a21e-ffba1f11e94a" containerName="payload-processing"
Sep 11 17:35:49.019679 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.019661    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8"
Sep 11 17:35:49.022169 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.022145    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-87b0dc-serving-cert\""
Sep 11 17:35:49.036329 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.036301    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8"]
Sep 11 17:35:49.128368 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.128324    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b910d0fb-ed14-4384-8eb8-08e20409def4-openshift-service-ca\") pod \"maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8\" (UID: \"b910d0fb-ed14-4384-8eb8-08e20409def4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8"
Sep 11 17:35:49.128545 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.128456    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp6sj\" (UniqueName: \"kubernetes.io/projected/b910d0fb-ed14-4384-8eb8-08e20409def4-kube-api-access-pp6sj\") pod \"maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8\" (UID: \"b910d0fb-ed14-4384-8eb8-08e20409def4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8"
Sep 11 17:35:49.128545 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.128486    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b910d0fb-ed14-4384-8eb8-08e20409def4-metrics-certs\") pod \"maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8\" (UID: \"b910d0fb-ed14-4384-8eb8-08e20409def4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8"
Sep 11 17:35:49.128545 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.128535    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b910d0fb-ed14-4384-8eb8-08e20409def4-maas-api-tls\") pod \"maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8\" (UID: \"b910d0fb-ed14-4384-8eb8-08e20409def4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8"
Sep 11 17:35:49.229212 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.229166    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b910d0fb-ed14-4384-8eb8-08e20409def4-maas-api-tls\") pod \"maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8\" (UID: \"b910d0fb-ed14-4384-8eb8-08e20409def4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8"
Sep 11 17:35:49.229212 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.229218    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b910d0fb-ed14-4384-8eb8-08e20409def4-openshift-service-ca\") pod \"maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8\" (UID: \"b910d0fb-ed14-4384-8eb8-08e20409def4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8"
Sep 11 17:35:49.229491 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.229293    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp6sj\" (UniqueName: \"kubernetes.io/projected/b910d0fb-ed14-4384-8eb8-08e20409def4-kube-api-access-pp6sj\") pod \"maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8\" (UID: \"b910d0fb-ed14-4384-8eb8-08e20409def4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8"
Sep 11 17:35:49.229491 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.229313    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b910d0fb-ed14-4384-8eb8-08e20409def4-metrics-certs\") pod \"maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8\" (UID: \"b910d0fb-ed14-4384-8eb8-08e20409def4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8"
Sep 11 17:35:49.229491 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:35:49.229322    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-serving-cert: secret "maas-api-e2e-explicit-87b0dc-serving-cert" not found
Sep 11 17:35:49.229491 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:35:49.229424    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b910d0fb-ed14-4384-8eb8-08e20409def4-maas-api-tls podName:b910d0fb-ed14-4384-8eb8-08e20409def4 nodeName:}" failed. No retries permitted until 2026-09-11 17:35:49.729401513 +0000 UTC m=+917.533573929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b910d0fb-ed14-4384-8eb8-08e20409def4-maas-api-tls") pod "maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8" (UID: "b910d0fb-ed14-4384-8eb8-08e20409def4") : secret "maas-api-e2e-explicit-87b0dc-serving-cert" not found
Sep 11 17:35:49.229962 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.229945    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b910d0fb-ed14-4384-8eb8-08e20409def4-openshift-service-ca\") pod \"maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8\" (UID: \"b910d0fb-ed14-4384-8eb8-08e20409def4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8"
Sep 11 17:35:49.231713 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.231696    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b910d0fb-ed14-4384-8eb8-08e20409def4-metrics-certs\") pod \"maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8\" (UID: \"b910d0fb-ed14-4384-8eb8-08e20409def4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8"
Sep 11 17:35:49.239846 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.239823    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp6sj\" (UniqueName: \"kubernetes.io/projected/b910d0fb-ed14-4384-8eb8-08e20409def4-kube-api-access-pp6sj\") pod \"maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8\" (UID: \"b910d0fb-ed14-4384-8eb8-08e20409def4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8"
Sep 11 17:35:49.681240 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.681202    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc"]
Sep 11 17:35:49.684986 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.684968    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc"
Sep 11 17:35:49.687418 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.687397    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-87b0dc\""
Sep 11 17:35:49.696788 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.696767    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc"]
Sep 11 17:35:49.722294 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.722263    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6"]
Sep 11 17:35:49.726660 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.726632    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6"
Sep 11 17:35:49.729351 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.729328    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-87b0dc-dockercfg-kngdw\""
Sep 11 17:35:49.733570 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.733546    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b910d0fb-ed14-4384-8eb8-08e20409def4-maas-api-tls\") pod \"maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8\" (UID: \"b910d0fb-ed14-4384-8eb8-08e20409def4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8"
Sep 11 17:35:49.736632 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.736607    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b910d0fb-ed14-4384-8eb8-08e20409def4-maas-api-tls\") pod \"maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8\" (UID: \"b910d0fb-ed14-4384-8eb8-08e20409def4\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8"
Sep 11 17:35:49.738527 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.738505    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6"]
Sep 11 17:35:49.834421 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.834381    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj7q8\" (UniqueName: \"kubernetes.io/projected/d340cc33-f8fe-4853-8751-184156ad373f-kube-api-access-pj7q8\") pod \"payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6\" (UID: \"d340cc33-f8fe-4853-8751-184156ad373f\") " pod="openshift-ingress/payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6"
Sep 11 17:35:49.834636 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.834460    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d340cc33-f8fe-4853-8751-184156ad373f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6\" (UID: \"d340cc33-f8fe-4853-8751-184156ad373f\") " pod="openshift-ingress/payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6"
Sep 11 17:35:49.834636 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.834485    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwz4j\" (UniqueName: \"kubernetes.io/projected/745a7d85-cab0-48c3-815b-0f0b14e5f1c1-kube-api-access-hwz4j\") pod \"payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc\" (UID: \"745a7d85-cab0-48c3-815b-0f0b14e5f1c1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc"
Sep 11 17:35:49.834636 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.834594    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/745a7d85-cab0-48c3-815b-0f0b14e5f1c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc\" (UID: \"745a7d85-cab0-48c3-815b-0f0b14e5f1c1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc"
Sep 11 17:35:49.931430 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.931335    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8"
Sep 11 17:35:49.935480 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.935457    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d340cc33-f8fe-4853-8751-184156ad373f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6\" (UID: \"d340cc33-f8fe-4853-8751-184156ad373f\") " pod="openshift-ingress/payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6"
Sep 11 17:35:49.935548 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.935489    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwz4j\" (UniqueName: \"kubernetes.io/projected/745a7d85-cab0-48c3-815b-0f0b14e5f1c1-kube-api-access-hwz4j\") pod \"payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc\" (UID: \"745a7d85-cab0-48c3-815b-0f0b14e5f1c1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc"
Sep 11 17:35:49.935602 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.935553    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/745a7d85-cab0-48c3-815b-0f0b14e5f1c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc\" (UID: \"745a7d85-cab0-48c3-815b-0f0b14e5f1c1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc"
Sep 11 17:35:49.935705 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.935680    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj7q8\" (UniqueName: \"kubernetes.io/projected/d340cc33-f8fe-4853-8751-184156ad373f-kube-api-access-pj7q8\") pod \"payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6\" (UID: \"d340cc33-f8fe-4853-8751-184156ad373f\") " pod="openshift-ingress/payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6"
Sep 11 17:35:49.936299 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.936274    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d340cc33-f8fe-4853-8751-184156ad373f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6\" (UID: \"d340cc33-f8fe-4853-8751-184156ad373f\") " pod="openshift-ingress/payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6"
Sep 11 17:35:49.936387 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.936283    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/745a7d85-cab0-48c3-815b-0f0b14e5f1c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc\" (UID: \"745a7d85-cab0-48c3-815b-0f0b14e5f1c1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc"
Sep 11 17:35:49.947775 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.947740    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwz4j\" (UniqueName: \"kubernetes.io/projected/745a7d85-cab0-48c3-815b-0f0b14e5f1c1-kube-api-access-hwz4j\") pod \"payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc\" (UID: \"745a7d85-cab0-48c3-815b-0f0b14e5f1c1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc"
Sep 11 17:35:49.948204 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.948185    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj7q8\" (UniqueName: \"kubernetes.io/projected/d340cc33-f8fe-4853-8751-184156ad373f-kube-api-access-pj7q8\") pod \"payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6\" (UID: \"d340cc33-f8fe-4853-8751-184156ad373f\") " pod="openshift-ingress/payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6"
Sep 11 17:35:49.999310 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:49.998922    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc"
Sep 11 17:35:50.037217 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:50.037184    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6"
Sep 11 17:35:50.208921 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:50.208877    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8"]
Sep 11 17:35:50.210395 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:35:50.210369    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb910d0fb_ed14_4384_8eb8_08e20409def4.slice/crio-4e16930beb06c76cfa76ba24e55459d8420085f7f527b880c21e98f1689dde2c WatchSource:0}: Error finding container 4e16930beb06c76cfa76ba24e55459d8420085f7f527b880c21e98f1689dde2c: Status 404 returned error can't find the container with id 4e16930beb06c76cfa76ba24e55459d8420085f7f527b880c21e98f1689dde2c
Sep 11 17:35:50.211817 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:50.211792    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:35:50.283271 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:50.283133    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc"]
Sep 11 17:35:50.285629 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:35:50.285588    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod745a7d85_cab0_48c3_815b_0f0b14e5f1c1.slice/crio-5f388e9dc0397cff5a98b7bb801e8c358c77a1680159b2248786ce8493e38dd8 WatchSource:0}: Error finding container 5f388e9dc0397cff5a98b7bb801e8c358c77a1680159b2248786ce8493e38dd8: Status 404 returned error can't find the container with id 5f388e9dc0397cff5a98b7bb801e8c358c77a1680159b2248786ce8493e38dd8
Sep 11 17:35:50.323064 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:50.323035    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6"]
Sep 11 17:35:50.324467 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:35:50.324434    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd340cc33_f8fe_4853_8751_184156ad373f.slice/crio-a190aeb677867bbe9a3942e429fc97cbe1552deeb2c1ac6748614fdb08dd7203 WatchSource:0}: Error finding container a190aeb677867bbe9a3942e429fc97cbe1552deeb2c1ac6748614fdb08dd7203: Status 404 returned error can't find the container with id a190aeb677867bbe9a3942e429fc97cbe1552deeb2c1ac6748614fdb08dd7203
Sep 11 17:35:51.208266 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:51.208227    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6" event={"ID":"d340cc33-f8fe-4853-8751-184156ad373f","Type":"ContainerStarted","Data":"18bd4dae0fe25824d6a56be571b0aa04258605687f1f81b918a2a774d71e35d4"}
Sep 11 17:35:51.208745 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:51.208371    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6"
Sep 11 17:35:51.208745 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:51.208402    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6" event={"ID":"d340cc33-f8fe-4853-8751-184156ad373f","Type":"ContainerStarted","Data":"a190aeb677867bbe9a3942e429fc97cbe1552deeb2c1ac6748614fdb08dd7203"}
Sep 11 17:35:51.209870 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:51.209840    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc" event={"ID":"745a7d85-cab0-48c3-815b-0f0b14e5f1c1","Type":"ContainerStarted","Data":"415e423ed5c53910349dba99494a82fc48c055ac702ec876e282faa94543573c"}
Sep 11 17:35:51.210027 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:51.209888    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc" event={"ID":"745a7d85-cab0-48c3-815b-0f0b14e5f1c1","Type":"ContainerStarted","Data":"5f388e9dc0397cff5a98b7bb801e8c358c77a1680159b2248786ce8493e38dd8"}
Sep 11 17:35:51.210027 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:51.209942    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc"
Sep 11 17:35:51.211356 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:51.211324    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8" event={"ID":"b910d0fb-ed14-4384-8eb8-08e20409def4","Type":"ContainerStarted","Data":"6eb77828db2f9fab3c084cbdd93f88f890d9b942d155928bbf15df172ac39ab6"}
Sep 11 17:35:51.211460 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:51.211367    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8" event={"ID":"b910d0fb-ed14-4384-8eb8-08e20409def4","Type":"ContainerStarted","Data":"4e16930beb06c76cfa76ba24e55459d8420085f7f527b880c21e98f1689dde2c"}
Sep 11 17:35:51.211460 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:51.211401    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8"
Sep 11 17:35:51.234313 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:51.234264    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6" podStartSLOduration=2.234247404 podStartE2EDuration="2.234247404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:35:51.232854924 +0000 UTC m=+919.037027343" watchObservedRunningTime="2026-09-11 17:35:51.234247404 +0000 UTC m=+919.038419814"
Sep 11 17:35:51.262849 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:51.262794    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc" podStartSLOduration=2.262779817 podStartE2EDuration="2.262779817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:35:51.262244741 +0000 UTC m=+919.066417159" watchObservedRunningTime="2026-09-11 17:35:51.262779817 +0000 UTC m=+919.066952237"
Sep 11 17:35:51.300767 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:51.300709    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8" podStartSLOduration=2.762028112 podStartE2EDuration="3.300691254s" totalImagesPullingTime="538.663142ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:35:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:35:51.299122392 +0000 UTC m=+919.103294812" watchObservedRunningTime="2026-09-11 17:35:51.300691254 +0000 UTC m=+919.104863672"
Sep 11 17:35:57.221689 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:35:57.221657    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8"
Sep 11 17:36:02.218814 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:36:02.218779    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc"
Sep 11 17:36:02.219229 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:36:02.218941    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6"
Sep 11 17:37:08.093639 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:08.093555    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2"]
Sep 11 17:37:08.097688 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:08.097663    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2"
Sep 11 17:37:08.100908 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:08.100864    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2340fb02-serving-cert\""
Sep 11 17:37:08.111386 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:08.111356    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2"]
Sep 11 17:37:08.161496 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:08.161442    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f9d576-b8cf-4d91-833b-9399fa6dd7d9-openshift-service-ca\") pod \"maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2\" (UID: \"24f9d576-b8cf-4d91-833b-9399fa6dd7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2"
Sep 11 17:37:08.161668 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:08.161525    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24f9d576-b8cf-4d91-833b-9399fa6dd7d9-metrics-certs\") pod \"maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2\" (UID: \"24f9d576-b8cf-4d91-833b-9399fa6dd7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2"
Sep 11 17:37:08.161668 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:08.161618    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24f9d576-b8cf-4d91-833b-9399fa6dd7d9-maas-api-tls\") pod \"maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2\" (UID: \"24f9d576-b8cf-4d91-833b-9399fa6dd7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2"
Sep 11 17:37:08.161668 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:08.161660    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-648sz\" (UniqueName: \"kubernetes.io/projected/24f9d576-b8cf-4d91-833b-9399fa6dd7d9-kube-api-access-648sz\") pod \"maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2\" (UID: \"24f9d576-b8cf-4d91-833b-9399fa6dd7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2"
Sep 11 17:37:08.262362 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:08.262325    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-648sz\" (UniqueName: \"kubernetes.io/projected/24f9d576-b8cf-4d91-833b-9399fa6dd7d9-kube-api-access-648sz\") pod \"maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2\" (UID: \"24f9d576-b8cf-4d91-833b-9399fa6dd7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2"
Sep 11 17:37:08.262578 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:08.262398    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f9d576-b8cf-4d91-833b-9399fa6dd7d9-openshift-service-ca\") pod \"maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2\" (UID: \"24f9d576-b8cf-4d91-833b-9399fa6dd7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2"
Sep 11 17:37:08.262578 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:08.262426    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24f9d576-b8cf-4d91-833b-9399fa6dd7d9-metrics-certs\") pod \"maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2\" (UID: \"24f9d576-b8cf-4d91-833b-9399fa6dd7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2"
Sep 11 17:37:08.262578 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:08.262461    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24f9d576-b8cf-4d91-833b-9399fa6dd7d9-maas-api-tls\") pod \"maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2\" (UID: \"24f9d576-b8cf-4d91-833b-9399fa6dd7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2"
Sep 11 17:37:08.262578 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:37:08.262553    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-serving-cert: secret "maas-api-e2e-mt-2340fb02-serving-cert" not found
Sep 11 17:37:08.262803 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:37:08.262620    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24f9d576-b8cf-4d91-833b-9399fa6dd7d9-maas-api-tls podName:24f9d576-b8cf-4d91-833b-9399fa6dd7d9 nodeName:}" failed. No retries permitted until 2026-09-11 17:37:08.762603944 +0000 UTC m=+996.566776343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/24f9d576-b8cf-4d91-833b-9399fa6dd7d9-maas-api-tls") pod "maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2" (UID: "24f9d576-b8cf-4d91-833b-9399fa6dd7d9") : secret "maas-api-e2e-mt-2340fb02-serving-cert" not found
Sep 11 17:37:08.263098 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:08.263077    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f9d576-b8cf-4d91-833b-9399fa6dd7d9-openshift-service-ca\") pod \"maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2\" (UID: \"24f9d576-b8cf-4d91-833b-9399fa6dd7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2"
Sep 11 17:37:08.264831 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:08.264813    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24f9d576-b8cf-4d91-833b-9399fa6dd7d9-metrics-certs\") pod \"maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2\" (UID: \"24f9d576-b8cf-4d91-833b-9399fa6dd7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2"
Sep 11 17:37:08.273007 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:08.272981    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-648sz\" (UniqueName: \"kubernetes.io/projected/24f9d576-b8cf-4d91-833b-9399fa6dd7d9-kube-api-access-648sz\") pod \"maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2\" (UID: \"24f9d576-b8cf-4d91-833b-9399fa6dd7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2"
Sep 11 17:37:08.768263 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:08.768231    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24f9d576-b8cf-4d91-833b-9399fa6dd7d9-maas-api-tls\") pod \"maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2\" (UID: \"24f9d576-b8cf-4d91-833b-9399fa6dd7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2"
Sep 11 17:37:08.770756 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:08.770728    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24f9d576-b8cf-4d91-833b-9399fa6dd7d9-maas-api-tls\") pod \"maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2\" (UID: \"24f9d576-b8cf-4d91-833b-9399fa6dd7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2"
Sep 11 17:37:09.016940 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.016871    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2"
Sep 11 17:37:09.290577 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.288352    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2"]
Sep 11 17:37:09.291336 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:37:09.291278    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f9d576_b8cf_4d91_833b_9399fa6dd7d9.slice/crio-4bfd5daf5cc3ddce94e7c5fb8353b528761701c7ae1750da4c1fc29654829216 WatchSource:0}: Error finding container 4bfd5daf5cc3ddce94e7c5fb8353b528761701c7ae1750da4c1fc29654829216: Status 404 returned error can't find the container with id 4bfd5daf5cc3ddce94e7c5fb8353b528761701c7ae1750da4c1fc29654829216
Sep 11 17:37:09.440960 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.440927    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8"]
Sep 11 17:37:09.441288 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.441168    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8" podUID="b910d0fb-ed14-4384-8eb8-08e20409def4" containerName="maas-api" containerID="cri-o://6eb77828db2f9fab3c084cbdd93f88f890d9b942d155928bbf15df172ac39ab6" gracePeriod=30
Sep 11 17:37:09.495322 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.495284    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6"]
Sep 11 17:37:09.495512 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.495490    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6" podUID="d340cc33-f8fe-4853-8751-184156ad373f" containerName="payload-processing" containerID="cri-o://18bd4dae0fe25824d6a56be571b0aa04258605687f1f81b918a2a774d71e35d4" gracePeriod=30
Sep 11 17:37:09.506067 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.506035    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc"]
Sep 11 17:37:09.506953 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.506847    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc" podUID="745a7d85-cab0-48c3-815b-0f0b14e5f1c1" containerName="payload-pre-processing" containerID="cri-o://415e423ed5c53910349dba99494a82fc48c055ac702ec876e282faa94543573c" gracePeriod=30
Sep 11 17:37:09.607053 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.607016    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2" event={"ID":"24f9d576-b8cf-4d91-833b-9399fa6dd7d9","Type":"ContainerStarted","Data":"4bfd5daf5cc3ddce94e7c5fb8353b528761701c7ae1750da4c1fc29654829216"}
Sep 11 17:37:09.608864 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.608835    2560 generic.go:356] "Generic (PLEG): container finished" podID="b910d0fb-ed14-4384-8eb8-08e20409def4" containerID="6eb77828db2f9fab3c084cbdd93f88f890d9b942d155928bbf15df172ac39ab6" exitCode=0
Sep 11 17:37:09.609010 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.608876    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8" event={"ID":"b910d0fb-ed14-4384-8eb8-08e20409def4","Type":"ContainerDied","Data":"6eb77828db2f9fab3c084cbdd93f88f890d9b942d155928bbf15df172ac39ab6"}
Sep 11 17:37:09.783858 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.783828    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8"
Sep 11 17:37:09.806060 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.806006    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6"
Sep 11 17:37:09.835333 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.835259    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc"
Sep 11 17:37:09.880377 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.880337    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b910d0fb-ed14-4384-8eb8-08e20409def4-maas-api-tls\") pod \"b910d0fb-ed14-4384-8eb8-08e20409def4\" (UID: \"b910d0fb-ed14-4384-8eb8-08e20409def4\") "
Sep 11 17:37:09.880541 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.880446    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b910d0fb-ed14-4384-8eb8-08e20409def4-metrics-certs\") pod \"b910d0fb-ed14-4384-8eb8-08e20409def4\" (UID: \"b910d0fb-ed14-4384-8eb8-08e20409def4\") "
Sep 11 17:37:09.880541 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.880479    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b910d0fb-ed14-4384-8eb8-08e20409def4-openshift-service-ca\") pod \"b910d0fb-ed14-4384-8eb8-08e20409def4\" (UID: \"b910d0fb-ed14-4384-8eb8-08e20409def4\") "
Sep 11 17:37:09.880541 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.880495    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pp6sj\" (UniqueName: \"kubernetes.io/projected/b910d0fb-ed14-4384-8eb8-08e20409def4-kube-api-access-pp6sj\") pod \"b910d0fb-ed14-4384-8eb8-08e20409def4\" (UID: \"b910d0fb-ed14-4384-8eb8-08e20409def4\") "
Sep 11 17:37:09.880541 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.880515    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pj7q8\" (UniqueName: \"kubernetes.io/projected/d340cc33-f8fe-4853-8751-184156ad373f-kube-api-access-pj7q8\") pod \"d340cc33-f8fe-4853-8751-184156ad373f\" (UID: \"d340cc33-f8fe-4853-8751-184156ad373f\") "
Sep 11 17:37:09.880757 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.880544    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d340cc33-f8fe-4853-8751-184156ad373f-plugins-config-volume\") pod \"d340cc33-f8fe-4853-8751-184156ad373f\" (UID: \"d340cc33-f8fe-4853-8751-184156ad373f\") "
Sep 11 17:37:09.880757 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.880570    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwz4j\" (UniqueName: \"kubernetes.io/projected/745a7d85-cab0-48c3-815b-0f0b14e5f1c1-kube-api-access-hwz4j\") pod \"745a7d85-cab0-48c3-815b-0f0b14e5f1c1\" (UID: \"745a7d85-cab0-48c3-815b-0f0b14e5f1c1\") "
Sep 11 17:37:09.880757 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.880594    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/745a7d85-cab0-48c3-815b-0f0b14e5f1c1-plugins-config-volume\") pod \"745a7d85-cab0-48c3-815b-0f0b14e5f1c1\" (UID: \"745a7d85-cab0-48c3-815b-0f0b14e5f1c1\") "
Sep 11 17:37:09.880951 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.880852    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b910d0fb-ed14-4384-8eb8-08e20409def4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b910d0fb-ed14-4384-8eb8-08e20409def4" (UID: "b910d0fb-ed14-4384-8eb8-08e20409def4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:37:09.881038 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.881011    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b910d0fb-ed14-4384-8eb8-08e20409def4-openshift-service-ca\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:37:09.881424 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.881397    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d340cc33-f8fe-4853-8751-184156ad373f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d340cc33-f8fe-4853-8751-184156ad373f" (UID: "d340cc33-f8fe-4853-8751-184156ad373f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:37:09.881849 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.881823    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/745a7d85-cab0-48c3-815b-0f0b14e5f1c1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "745a7d85-cab0-48c3-815b-0f0b14e5f1c1" (UID: "745a7d85-cab0-48c3-815b-0f0b14e5f1c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:37:09.884076 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.884035    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d340cc33-f8fe-4853-8751-184156ad373f-kube-api-access-pj7q8" (OuterVolumeSpecName: "kube-api-access-pj7q8") pod "d340cc33-f8fe-4853-8751-184156ad373f" (UID: "d340cc33-f8fe-4853-8751-184156ad373f"). InnerVolumeSpecName "kube-api-access-pj7q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:37:09.884785 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.884275    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b910d0fb-ed14-4384-8eb8-08e20409def4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b910d0fb-ed14-4384-8eb8-08e20409def4" (UID: "b910d0fb-ed14-4384-8eb8-08e20409def4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:37:09.884785 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.884424    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b910d0fb-ed14-4384-8eb8-08e20409def4-kube-api-access-pp6sj" (OuterVolumeSpecName: "kube-api-access-pp6sj") pod "b910d0fb-ed14-4384-8eb8-08e20409def4" (UID: "b910d0fb-ed14-4384-8eb8-08e20409def4"). InnerVolumeSpecName "kube-api-access-pp6sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:37:09.884785 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.884522    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/745a7d85-cab0-48c3-815b-0f0b14e5f1c1-kube-api-access-hwz4j" (OuterVolumeSpecName: "kube-api-access-hwz4j") pod "745a7d85-cab0-48c3-815b-0f0b14e5f1c1" (UID: "745a7d85-cab0-48c3-815b-0f0b14e5f1c1"). InnerVolumeSpecName "kube-api-access-hwz4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:37:09.884785 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.884567    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b910d0fb-ed14-4384-8eb8-08e20409def4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b910d0fb-ed14-4384-8eb8-08e20409def4" (UID: "b910d0fb-ed14-4384-8eb8-08e20409def4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:37:09.981533 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.981495    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b910d0fb-ed14-4384-8eb8-08e20409def4-metrics-certs\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:37:09.981533 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.981526    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp6sj\" (UniqueName: \"kubernetes.io/projected/b910d0fb-ed14-4384-8eb8-08e20409def4-kube-api-access-pp6sj\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:37:09.981533 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.981539    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj7q8\" (UniqueName: \"kubernetes.io/projected/d340cc33-f8fe-4853-8751-184156ad373f-kube-api-access-pj7q8\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:37:09.981769 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.981548    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d340cc33-f8fe-4853-8751-184156ad373f-plugins-config-volume\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:37:09.981769 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.981557    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwz4j\" (UniqueName: \"kubernetes.io/projected/745a7d85-cab0-48c3-815b-0f0b14e5f1c1-kube-api-access-hwz4j\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:37:09.981769 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.981566    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/745a7d85-cab0-48c3-815b-0f0b14e5f1c1-plugins-config-volume\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:37:09.981769 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:09.981574    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b910d0fb-ed14-4384-8eb8-08e20409def4-maas-api-tls\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:37:10.615196 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.615160    2560 generic.go:356] "Generic (PLEG): container finished" podID="d340cc33-f8fe-4853-8751-184156ad373f" containerID="18bd4dae0fe25824d6a56be571b0aa04258605687f1f81b918a2a774d71e35d4" exitCode=0
Sep 11 17:37:10.615641 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.615204    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6"
Sep 11 17:37:10.615641 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.615249    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6" event={"ID":"d340cc33-f8fe-4853-8751-184156ad373f","Type":"ContainerDied","Data":"18bd4dae0fe25824d6a56be571b0aa04258605687f1f81b918a2a774d71e35d4"}
Sep 11 17:37:10.615641 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.615296    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6" event={"ID":"d340cc33-f8fe-4853-8751-184156ad373f","Type":"ContainerDied","Data":"a190aeb677867bbe9a3942e429fc97cbe1552deeb2c1ac6748614fdb08dd7203"}
Sep 11 17:37:10.615641 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.615322    2560 scope.go:117] "RemoveContainer" containerID="18bd4dae0fe25824d6a56be571b0aa04258605687f1f81b918a2a774d71e35d4"
Sep 11 17:37:10.616695 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.616649    2560 generic.go:356] "Generic (PLEG): container finished" podID="745a7d85-cab0-48c3-815b-0f0b14e5f1c1" containerID="415e423ed5c53910349dba99494a82fc48c055ac702ec876e282faa94543573c" exitCode=0
Sep 11 17:37:10.616695 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.616689    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc"
Sep 11 17:37:10.616820 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.616729    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc" event={"ID":"745a7d85-cab0-48c3-815b-0f0b14e5f1c1","Type":"ContainerDied","Data":"415e423ed5c53910349dba99494a82fc48c055ac702ec876e282faa94543573c"}
Sep 11 17:37:10.616820 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.616777    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc" event={"ID":"745a7d85-cab0-48c3-815b-0f0b14e5f1c1","Type":"ContainerDied","Data":"5f388e9dc0397cff5a98b7bb801e8c358c77a1680159b2248786ce8493e38dd8"}
Sep 11 17:37:10.618712 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.618654    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2" event={"ID":"24f9d576-b8cf-4d91-833b-9399fa6dd7d9","Type":"ContainerStarted","Data":"45be04261eb792832a06854c7349071fc63d8c454b17be55b610f15819a89ef0"}
Sep 11 17:37:10.618841 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.618767    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2"
Sep 11 17:37:10.620251 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.620228    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8" event={"ID":"b910d0fb-ed14-4384-8eb8-08e20409def4","Type":"ContainerDied","Data":"4e16930beb06c76cfa76ba24e55459d8420085f7f527b880c21e98f1689dde2c"}
Sep 11 17:37:10.620345 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.620233    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8"
Sep 11 17:37:10.625421 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.625400    2560 scope.go:117] "RemoveContainer" containerID="18bd4dae0fe25824d6a56be571b0aa04258605687f1f81b918a2a774d71e35d4"
Sep 11 17:37:10.625952 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:37:10.625726    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18bd4dae0fe25824d6a56be571b0aa04258605687f1f81b918a2a774d71e35d4\": container with ID starting with 18bd4dae0fe25824d6a56be571b0aa04258605687f1f81b918a2a774d71e35d4 not found: ID does not exist" containerID="18bd4dae0fe25824d6a56be571b0aa04258605687f1f81b918a2a774d71e35d4"
Sep 11 17:37:10.625952 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.625762    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18bd4dae0fe25824d6a56be571b0aa04258605687f1f81b918a2a774d71e35d4"} err="failed to get container status \"18bd4dae0fe25824d6a56be571b0aa04258605687f1f81b918a2a774d71e35d4\": rpc error: code = NotFound desc = could not find container \"18bd4dae0fe25824d6a56be571b0aa04258605687f1f81b918a2a774d71e35d4\": container with ID starting with 18bd4dae0fe25824d6a56be571b0aa04258605687f1f81b918a2a774d71e35d4 not found: ID does not exist"
Sep 11 17:37:10.625952 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.625785    2560 scope.go:117] "RemoveContainer" containerID="415e423ed5c53910349dba99494a82fc48c055ac702ec876e282faa94543573c"
Sep 11 17:37:10.634451 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.634431    2560 scope.go:117] "RemoveContainer" containerID="415e423ed5c53910349dba99494a82fc48c055ac702ec876e282faa94543573c"
Sep 11 17:37:10.634795 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:37:10.634776    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"415e423ed5c53910349dba99494a82fc48c055ac702ec876e282faa94543573c\": container with ID starting with 415e423ed5c53910349dba99494a82fc48c055ac702ec876e282faa94543573c not found: ID does not exist" containerID="415e423ed5c53910349dba99494a82fc48c055ac702ec876e282faa94543573c"
Sep 11 17:37:10.634839 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.634802    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"415e423ed5c53910349dba99494a82fc48c055ac702ec876e282faa94543573c"} err="failed to get container status \"415e423ed5c53910349dba99494a82fc48c055ac702ec876e282faa94543573c\": rpc error: code = NotFound desc = could not find container \"415e423ed5c53910349dba99494a82fc48c055ac702ec876e282faa94543573c\": container with ID starting with 415e423ed5c53910349dba99494a82fc48c055ac702ec876e282faa94543573c not found: ID does not exist"
Sep 11 17:37:10.634839 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.634819    2560 scope.go:117] "RemoveContainer" containerID="6eb77828db2f9fab3c084cbdd93f88f890d9b942d155928bbf15df172ac39ab6"
Sep 11 17:37:10.659517 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.659418    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2" podStartSLOduration=2.163675246 podStartE2EDuration="2.659397746s" totalImagesPullingTime="495.7225ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:37:10.65774315 +0000 UTC m=+998.461915569" watchObservedRunningTime="2026-09-11 17:37:10.659397746 +0000 UTC m=+998.463570167"
Sep 11 17:37:10.677963 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.677926    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc"]
Sep 11 17:37:10.681757 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.681731    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-87b0dc-56df9665fd-9jtqc"]
Sep 11 17:37:10.693941 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.693912    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6"]
Sep 11 17:37:10.695640 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.695616    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-87b0dc-8799f7bf-mllg6"]
Sep 11 17:37:10.704767 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.704744    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8"]
Sep 11 17:37:10.708797 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.708775    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-87b0dc-7c76f5dfcc-mhbh8"]
Sep 11 17:37:10.718912 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.717370    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="745a7d85-cab0-48c3-815b-0f0b14e5f1c1" path="/var/lib/kubelet/pods/745a7d85-cab0-48c3-815b-0f0b14e5f1c1/volumes"
Sep 11 17:37:10.718912 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.718014    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b910d0fb-ed14-4384-8eb8-08e20409def4" path="/var/lib/kubelet/pods/b910d0fb-ed14-4384-8eb8-08e20409def4/volumes"
Sep 11 17:37:10.718912 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:10.718521    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d340cc33-f8fe-4853-8751-184156ad373f" path="/var/lib/kubelet/pods/d340cc33-f8fe-4853-8751-184156ad373f/volumes"
Sep 11 17:37:16.634393 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:16.634365    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2"
Sep 11 17:37:52.461741 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:52.461702    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2"]
Sep 11 17:37:52.462308 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:52.461977    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2" podUID="24f9d576-b8cf-4d91-833b-9399fa6dd7d9" containerName="maas-api" containerID="cri-o://45be04261eb792832a06854c7349071fc63d8c454b17be55b610f15819a89ef0" gracePeriod=30
Sep 11 17:37:52.714122 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:52.714058    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2"
Sep 11 17:37:52.776748 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:52.776705    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-648sz\" (UniqueName: \"kubernetes.io/projected/24f9d576-b8cf-4d91-833b-9399fa6dd7d9-kube-api-access-648sz\") pod \"24f9d576-b8cf-4d91-833b-9399fa6dd7d9\" (UID: \"24f9d576-b8cf-4d91-833b-9399fa6dd7d9\") "
Sep 11 17:37:52.776965 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:52.776805    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24f9d576-b8cf-4d91-833b-9399fa6dd7d9-maas-api-tls\") pod \"24f9d576-b8cf-4d91-833b-9399fa6dd7d9\" (UID: \"24f9d576-b8cf-4d91-833b-9399fa6dd7d9\") "
Sep 11 17:37:52.776965 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:52.776887    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f9d576-b8cf-4d91-833b-9399fa6dd7d9-openshift-service-ca\") pod \"24f9d576-b8cf-4d91-833b-9399fa6dd7d9\" (UID: \"24f9d576-b8cf-4d91-833b-9399fa6dd7d9\") "
Sep 11 17:37:52.776965 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:52.776930    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24f9d576-b8cf-4d91-833b-9399fa6dd7d9-metrics-certs\") pod \"24f9d576-b8cf-4d91-833b-9399fa6dd7d9\" (UID: \"24f9d576-b8cf-4d91-833b-9399fa6dd7d9\") "
Sep 11 17:37:52.777334 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:52.777299    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24f9d576-b8cf-4d91-833b-9399fa6dd7d9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "24f9d576-b8cf-4d91-833b-9399fa6dd7d9" (UID: "24f9d576-b8cf-4d91-833b-9399fa6dd7d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:37:52.779248 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:52.779213    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24f9d576-b8cf-4d91-833b-9399fa6dd7d9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "24f9d576-b8cf-4d91-833b-9399fa6dd7d9" (UID: "24f9d576-b8cf-4d91-833b-9399fa6dd7d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:37:52.779361 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:52.779300    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f9d576-b8cf-4d91-833b-9399fa6dd7d9-kube-api-access-648sz" (OuterVolumeSpecName: "kube-api-access-648sz") pod "24f9d576-b8cf-4d91-833b-9399fa6dd7d9" (UID: "24f9d576-b8cf-4d91-833b-9399fa6dd7d9"). InnerVolumeSpecName "kube-api-access-648sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:37:52.779361 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:52.779332    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24f9d576-b8cf-4d91-833b-9399fa6dd7d9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "24f9d576-b8cf-4d91-833b-9399fa6dd7d9" (UID: "24f9d576-b8cf-4d91-833b-9399fa6dd7d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:37:52.846885 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:52.842858    2560 generic.go:356] "Generic (PLEG): container finished" podID="24f9d576-b8cf-4d91-833b-9399fa6dd7d9" containerID="45be04261eb792832a06854c7349071fc63d8c454b17be55b610f15819a89ef0" exitCode=0
Sep 11 17:37:52.846885 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:52.843046    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2" event={"ID":"24f9d576-b8cf-4d91-833b-9399fa6dd7d9","Type":"ContainerDied","Data":"45be04261eb792832a06854c7349071fc63d8c454b17be55b610f15819a89ef0"}
Sep 11 17:37:52.846885 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:52.843100    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2" event={"ID":"24f9d576-b8cf-4d91-833b-9399fa6dd7d9","Type":"ContainerDied","Data":"4bfd5daf5cc3ddce94e7c5fb8353b528761701c7ae1750da4c1fc29654829216"}
Sep 11 17:37:52.846885 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:52.843157    2560 scope.go:117] "RemoveContainer" containerID="45be04261eb792832a06854c7349071fc63d8c454b17be55b610f15819a89ef0"
Sep 11 17:37:52.846885 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:52.843405    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2"
Sep 11 17:37:52.858075 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:52.858025    2560 scope.go:117] "RemoveContainer" containerID="45be04261eb792832a06854c7349071fc63d8c454b17be55b610f15819a89ef0"
Sep 11 17:37:52.858767 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:37:52.858731    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45be04261eb792832a06854c7349071fc63d8c454b17be55b610f15819a89ef0\": container with ID starting with 45be04261eb792832a06854c7349071fc63d8c454b17be55b610f15819a89ef0 not found: ID does not exist" containerID="45be04261eb792832a06854c7349071fc63d8c454b17be55b610f15819a89ef0"
Sep 11 17:37:52.858864 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:52.858788    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45be04261eb792832a06854c7349071fc63d8c454b17be55b610f15819a89ef0"} err="failed to get container status \"45be04261eb792832a06854c7349071fc63d8c454b17be55b610f15819a89ef0\": rpc error: code = NotFound desc = could not find container \"45be04261eb792832a06854c7349071fc63d8c454b17be55b610f15819a89ef0\": container with ID starting with 45be04261eb792832a06854c7349071fc63d8c454b17be55b610f15819a89ef0 not found: ID does not exist"
Sep 11 17:37:52.870757 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:52.870514    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2"]
Sep 11 17:37:52.876185 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:52.876149    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2340fb02-7cdf84b477-n6ds2"]
Sep 11 17:37:52.877878 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:52.877849    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f9d576-b8cf-4d91-833b-9399fa6dd7d9-openshift-service-ca\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:37:52.877878 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:52.877880    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24f9d576-b8cf-4d91-833b-9399fa6dd7d9-metrics-certs\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:37:52.878372 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:52.877927    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-648sz\" (UniqueName: \"kubernetes.io/projected/24f9d576-b8cf-4d91-833b-9399fa6dd7d9-kube-api-access-648sz\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:37:52.878372 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:52.877941    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24f9d576-b8cf-4d91-833b-9399fa6dd7d9-maas-api-tls\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:37:54.715329 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:37:54.715293    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24f9d576-b8cf-4d91-833b-9399fa6dd7d9" path="/var/lib/kubelet/pods/24f9d576-b8cf-4d91-833b-9399fa6dd7d9/volumes"
Sep 11 17:38:06.761487 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:06.761440    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm"]
Sep 11 17:38:06.762078 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:06.761677    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm" podUID="9e15398c-1504-47de-a6dc-78a584cd17df" containerName="maas-api" containerID="cri-o://9da5a2b64e673c3cae7fd3e5ab818fc307fc13a6bc5518ea6c703e66f8b9e79a" gracePeriod=30
Sep 11 17:38:06.829221 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:06.829179    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm"]
Sep 11 17:38:06.829459 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:06.829430    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm" podUID="4f561801-e1cb-4712-88a2-6642d0897ce0" containerName="payload-processing" containerID="cri-o://6557ae53d8dcd08292362fbab0c8bea7da2fa12ec3d7715c43e4e96664f8dab3" gracePeriod=30
Sep 11 17:38:06.836384 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:06.836354    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74"]
Sep 11 17:38:06.836784 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:06.836757    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74" podUID="0b5a085e-e930-45c6-a9cb-d2ad581aa47c" containerName="payload-pre-processing" containerID="cri-o://96c6a442763865a88dbd2d2ad27081fc6e1f240b62ae401d5d60e51d9133cef2" gracePeriod=30
Sep 11 17:38:06.913686 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:06.913643    2560 generic.go:356] "Generic (PLEG): container finished" podID="9e15398c-1504-47de-a6dc-78a584cd17df" containerID="9da5a2b64e673c3cae7fd3e5ab818fc307fc13a6bc5518ea6c703e66f8b9e79a" exitCode=0
Sep 11 17:38:06.913831 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:06.913710    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm" event={"ID":"9e15398c-1504-47de-a6dc-78a584cd17df","Type":"ContainerDied","Data":"9da5a2b64e673c3cae7fd3e5ab818fc307fc13a6bc5518ea6c703e66f8b9e79a"}
Sep 11 17:38:07.112471 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.112430    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm"
Sep 11 17:38:07.115206 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.115170    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm"
Sep 11 17:38:07.138324 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.138295    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74"
Sep 11 17:38:07.213663 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.213627    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e15398c-1504-47de-a6dc-78a584cd17df-metrics-certs\") pod \"9e15398c-1504-47de-a6dc-78a584cd17df\" (UID: \"9e15398c-1504-47de-a6dc-78a584cd17df\") "
Sep 11 17:38:07.213860 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.213693    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e15398c-1504-47de-a6dc-78a584cd17df-maas-api-tls\") pod \"9e15398c-1504-47de-a6dc-78a584cd17df\" (UID: \"9e15398c-1504-47de-a6dc-78a584cd17df\") "
Sep 11 17:38:07.213860 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.213723    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f561801-e1cb-4712-88a2-6642d0897ce0-plugins-config-volume\") pod \"4f561801-e1cb-4712-88a2-6642d0897ce0\" (UID: \"4f561801-e1cb-4712-88a2-6642d0897ce0\") "
Sep 11 17:38:07.213860 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.213765    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtz8r\" (UniqueName: \"kubernetes.io/projected/9e15398c-1504-47de-a6dc-78a584cd17df-kube-api-access-qtz8r\") pod \"9e15398c-1504-47de-a6dc-78a584cd17df\" (UID: \"9e15398c-1504-47de-a6dc-78a584cd17df\") "
Sep 11 17:38:07.213860 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.213787    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s7pj\" (UniqueName: \"kubernetes.io/projected/4f561801-e1cb-4712-88a2-6642d0897ce0-kube-api-access-4s7pj\") pod \"4f561801-e1cb-4712-88a2-6642d0897ce0\" (UID: \"4f561801-e1cb-4712-88a2-6642d0897ce0\") "
Sep 11 17:38:07.213860 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.213809    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5a085e-e930-45c6-a9cb-d2ad581aa47c-plugins-config-volume\") pod \"0b5a085e-e930-45c6-a9cb-d2ad581aa47c\" (UID: \"0b5a085e-e930-45c6-a9cb-d2ad581aa47c\") "
Sep 11 17:38:07.213860 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.213860    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9867\" (UniqueName: \"kubernetes.io/projected/0b5a085e-e930-45c6-a9cb-d2ad581aa47c-kube-api-access-c9867\") pod \"0b5a085e-e930-45c6-a9cb-d2ad581aa47c\" (UID: \"0b5a085e-e930-45c6-a9cb-d2ad581aa47c\") "
Sep 11 17:38:07.214230 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.213920    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e15398c-1504-47de-a6dc-78a584cd17df-openshift-service-ca\") pod \"9e15398c-1504-47de-a6dc-78a584cd17df\" (UID: \"9e15398c-1504-47de-a6dc-78a584cd17df\") "
Sep 11 17:38:07.214230 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.214162    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f561801-e1cb-4712-88a2-6642d0897ce0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4f561801-e1cb-4712-88a2-6642d0897ce0" (UID: "4f561801-e1cb-4712-88a2-6642d0897ce0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:38:07.214337 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.214229    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f561801-e1cb-4712-88a2-6642d0897ce0-plugins-config-volume\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:38:07.214500 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.214472    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e15398c-1504-47de-a6dc-78a584cd17df-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9e15398c-1504-47de-a6dc-78a584cd17df" (UID: "9e15398c-1504-47de-a6dc-78a584cd17df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:38:07.214754 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.214709    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b5a085e-e930-45c6-a9cb-d2ad581aa47c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0b5a085e-e930-45c6-a9cb-d2ad581aa47c" (UID: "0b5a085e-e930-45c6-a9cb-d2ad581aa47c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:38:07.216117 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.216094    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e15398c-1504-47de-a6dc-78a584cd17df-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9e15398c-1504-47de-a6dc-78a584cd17df" (UID: "9e15398c-1504-47de-a6dc-78a584cd17df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:38:07.216202 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.216118    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5a085e-e930-45c6-a9cb-d2ad581aa47c-kube-api-access-c9867" (OuterVolumeSpecName: "kube-api-access-c9867") pod "0b5a085e-e930-45c6-a9cb-d2ad581aa47c" (UID: "0b5a085e-e930-45c6-a9cb-d2ad581aa47c"). InnerVolumeSpecName "kube-api-access-c9867". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:38:07.216202 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.216166    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e15398c-1504-47de-a6dc-78a584cd17df-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9e15398c-1504-47de-a6dc-78a584cd17df" (UID: "9e15398c-1504-47de-a6dc-78a584cd17df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:38:07.216351 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.216331    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e15398c-1504-47de-a6dc-78a584cd17df-kube-api-access-qtz8r" (OuterVolumeSpecName: "kube-api-access-qtz8r") pod "9e15398c-1504-47de-a6dc-78a584cd17df" (UID: "9e15398c-1504-47de-a6dc-78a584cd17df"). InnerVolumeSpecName "kube-api-access-qtz8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:38:07.216431 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.216411    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f561801-e1cb-4712-88a2-6642d0897ce0-kube-api-access-4s7pj" (OuterVolumeSpecName: "kube-api-access-4s7pj") pod "4f561801-e1cb-4712-88a2-6642d0897ce0" (UID: "4f561801-e1cb-4712-88a2-6642d0897ce0"). InnerVolumeSpecName "kube-api-access-4s7pj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:38:07.315018 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.314925    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e15398c-1504-47de-a6dc-78a584cd17df-maas-api-tls\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:38:07.315018 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.314957    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtz8r\" (UniqueName: \"kubernetes.io/projected/9e15398c-1504-47de-a6dc-78a584cd17df-kube-api-access-qtz8r\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:38:07.315018 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.314969    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s7pj\" (UniqueName: \"kubernetes.io/projected/4f561801-e1cb-4712-88a2-6642d0897ce0-kube-api-access-4s7pj\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:38:07.315018 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.314978    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5a085e-e930-45c6-a9cb-d2ad581aa47c-plugins-config-volume\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:38:07.315018 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.314987    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9867\" (UniqueName: \"kubernetes.io/projected/0b5a085e-e930-45c6-a9cb-d2ad581aa47c-kube-api-access-c9867\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:38:07.315018 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.314997    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e15398c-1504-47de-a6dc-78a584cd17df-openshift-service-ca\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:38:07.315018 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.315005    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e15398c-1504-47de-a6dc-78a584cd17df-metrics-certs\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:38:07.919928 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.919887    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm"
Sep 11 17:38:07.920380 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.919927    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm" event={"ID":"9e15398c-1504-47de-a6dc-78a584cd17df","Type":"ContainerDied","Data":"102f094f5da3e256667b340bf72e8a213b905f4055b2f39a7b27c5b04017e8e7"}
Sep 11 17:38:07.920380 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.920001    2560 scope.go:117] "RemoveContainer" containerID="9da5a2b64e673c3cae7fd3e5ab818fc307fc13a6bc5518ea6c703e66f8b9e79a"
Sep 11 17:38:07.921369 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.921348    2560 generic.go:356] "Generic (PLEG): container finished" podID="4f561801-e1cb-4712-88a2-6642d0897ce0" containerID="6557ae53d8dcd08292362fbab0c8bea7da2fa12ec3d7715c43e4e96664f8dab3" exitCode=0
Sep 11 17:38:07.921434 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.921371    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm" event={"ID":"4f561801-e1cb-4712-88a2-6642d0897ce0","Type":"ContainerDied","Data":"6557ae53d8dcd08292362fbab0c8bea7da2fa12ec3d7715c43e4e96664f8dab3"}
Sep 11 17:38:07.921434 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.921400    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm"
Sep 11 17:38:07.921434 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.921415    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm" event={"ID":"4f561801-e1cb-4712-88a2-6642d0897ce0","Type":"ContainerDied","Data":"3bb3e4950c1baed489249f7dc614237018b56139290dfe1ef57475ab69f8e77a"}
Sep 11 17:38:07.922946 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.922873    2560 generic.go:356] "Generic (PLEG): container finished" podID="0b5a085e-e930-45c6-a9cb-d2ad581aa47c" containerID="96c6a442763865a88dbd2d2ad27081fc6e1f240b62ae401d5d60e51d9133cef2" exitCode=0
Sep 11 17:38:07.923051 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.922953    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74" event={"ID":"0b5a085e-e930-45c6-a9cb-d2ad581aa47c","Type":"ContainerDied","Data":"96c6a442763865a88dbd2d2ad27081fc6e1f240b62ae401d5d60e51d9133cef2"}
Sep 11 17:38:07.923051 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.922991    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74" event={"ID":"0b5a085e-e930-45c6-a9cb-d2ad581aa47c","Type":"ContainerDied","Data":"162341601e71dee51f9863587ce176b50061ce6544d4614a85fb8295b25a3b35"}
Sep 11 17:38:07.923051 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.922996    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74"
Sep 11 17:38:07.929763 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.929738    2560 scope.go:117] "RemoveContainer" containerID="6557ae53d8dcd08292362fbab0c8bea7da2fa12ec3d7715c43e4e96664f8dab3"
Sep 11 17:38:07.938732 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.938714    2560 scope.go:117] "RemoveContainer" containerID="6557ae53d8dcd08292362fbab0c8bea7da2fa12ec3d7715c43e4e96664f8dab3"
Sep 11 17:38:07.939078 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:38:07.939057    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6557ae53d8dcd08292362fbab0c8bea7da2fa12ec3d7715c43e4e96664f8dab3\": container with ID starting with 6557ae53d8dcd08292362fbab0c8bea7da2fa12ec3d7715c43e4e96664f8dab3 not found: ID does not exist" containerID="6557ae53d8dcd08292362fbab0c8bea7da2fa12ec3d7715c43e4e96664f8dab3"
Sep 11 17:38:07.939162 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.939087    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6557ae53d8dcd08292362fbab0c8bea7da2fa12ec3d7715c43e4e96664f8dab3"} err="failed to get container status \"6557ae53d8dcd08292362fbab0c8bea7da2fa12ec3d7715c43e4e96664f8dab3\": rpc error: code = NotFound desc = could not find container \"6557ae53d8dcd08292362fbab0c8bea7da2fa12ec3d7715c43e4e96664f8dab3\": container with ID starting with 6557ae53d8dcd08292362fbab0c8bea7da2fa12ec3d7715c43e4e96664f8dab3 not found: ID does not exist"
Sep 11 17:38:07.939162 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.939106    2560 scope.go:117] "RemoveContainer" containerID="96c6a442763865a88dbd2d2ad27081fc6e1f240b62ae401d5d60e51d9133cef2"
Sep 11 17:38:07.949511 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.949472    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm"]
Sep 11 17:38:07.951023 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.951003    2560 scope.go:117] "RemoveContainer" containerID="96c6a442763865a88dbd2d2ad27081fc6e1f240b62ae401d5d60e51d9133cef2"
Sep 11 17:38:07.951353 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:38:07.951333    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96c6a442763865a88dbd2d2ad27081fc6e1f240b62ae401d5d60e51d9133cef2\": container with ID starting with 96c6a442763865a88dbd2d2ad27081fc6e1f240b62ae401d5d60e51d9133cef2 not found: ID does not exist" containerID="96c6a442763865a88dbd2d2ad27081fc6e1f240b62ae401d5d60e51d9133cef2"
Sep 11 17:38:07.951436 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.951359    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96c6a442763865a88dbd2d2ad27081fc6e1f240b62ae401d5d60e51d9133cef2"} err="failed to get container status \"96c6a442763865a88dbd2d2ad27081fc6e1f240b62ae401d5d60e51d9133cef2\": rpc error: code = NotFound desc = could not find container \"96c6a442763865a88dbd2d2ad27081fc6e1f240b62ae401d5d60e51d9133cef2\": container with ID starting with 96c6a442763865a88dbd2d2ad27081fc6e1f240b62ae401d5d60e51d9133cef2 not found: ID does not exist"
Sep 11 17:38:07.952125 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.952105    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d92fb3-7554f4565d-bjxjm"]
Sep 11 17:38:07.967168 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.967138    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74"]
Sep 11 17:38:07.972941 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.972917    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d92fb3-658bfc5bf8-d4h74"]
Sep 11 17:38:07.982537 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.982511    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm"]
Sep 11 17:38:07.986527 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:07.986507    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d92fb3-554d7df677-dndnm"]
Sep 11 17:38:08.715564 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:08.715535    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b5a085e-e930-45c6-a9cb-d2ad581aa47c" path="/var/lib/kubelet/pods/0b5a085e-e930-45c6-a9cb-d2ad581aa47c/volumes"
Sep 11 17:38:08.715880 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:08.715867    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f561801-e1cb-4712-88a2-6642d0897ce0" path="/var/lib/kubelet/pods/4f561801-e1cb-4712-88a2-6642d0897ce0/volumes"
Sep 11 17:38:08.716239 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:08.716227    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e15398c-1504-47de-a6dc-78a584cd17df" path="/var/lib/kubelet/pods/9e15398c-1504-47de-a6dc-78a584cd17df/volumes"
Sep 11 17:38:17.809212 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.809175    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm"]
Sep 11 17:38:17.809752 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.809728    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f561801-e1cb-4712-88a2-6642d0897ce0" containerName="payload-processing"
Sep 11 17:38:17.810257 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.810238    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f561801-e1cb-4712-88a2-6642d0897ce0" containerName="payload-processing"
Sep 11 17:38:17.810379 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.810367    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d340cc33-f8fe-4853-8751-184156ad373f" containerName="payload-processing"
Sep 11 17:38:17.810447 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.810423    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d340cc33-f8fe-4853-8751-184156ad373f" containerName="payload-processing"
Sep 11 17:38:17.810505 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.810465    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24f9d576-b8cf-4d91-833b-9399fa6dd7d9" containerName="maas-api"
Sep 11 17:38:17.810505 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.810477    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="24f9d576-b8cf-4d91-833b-9399fa6dd7d9" containerName="maas-api"
Sep 11 17:38:17.810505 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.810499    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b5a085e-e930-45c6-a9cb-d2ad581aa47c" containerName="payload-pre-processing"
Sep 11 17:38:17.810655 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.810508    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b5a085e-e930-45c6-a9cb-d2ad581aa47c" containerName="payload-pre-processing"
Sep 11 17:38:17.810655 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.810521    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e15398c-1504-47de-a6dc-78a584cd17df" containerName="maas-api"
Sep 11 17:38:17.810655 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.810530    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e15398c-1504-47de-a6dc-78a584cd17df" containerName="maas-api"
Sep 11 17:38:17.810655 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.810542    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b910d0fb-ed14-4384-8eb8-08e20409def4" containerName="maas-api"
Sep 11 17:38:17.810655 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.810551    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b910d0fb-ed14-4384-8eb8-08e20409def4" containerName="maas-api"
Sep 11 17:38:17.810655 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.810562    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="745a7d85-cab0-48c3-815b-0f0b14e5f1c1" containerName="payload-pre-processing"
Sep 11 17:38:17.810655 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.810571    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="745a7d85-cab0-48c3-815b-0f0b14e5f1c1" containerName="payload-pre-processing"
Sep 11 17:38:17.811007 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.810677    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e15398c-1504-47de-a6dc-78a584cd17df" containerName="maas-api"
Sep 11 17:38:17.811007 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.810691    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b5a085e-e930-45c6-a9cb-d2ad581aa47c" containerName="payload-pre-processing"
Sep 11 17:38:17.811007 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.810704    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="745a7d85-cab0-48c3-815b-0f0b14e5f1c1" containerName="payload-pre-processing"
Sep 11 17:38:17.811007 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.810715    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d340cc33-f8fe-4853-8751-184156ad373f" containerName="payload-processing"
Sep 11 17:38:17.811007 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.810725    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b910d0fb-ed14-4384-8eb8-08e20409def4" containerName="maas-api"
Sep 11 17:38:17.811007 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.810734    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f561801-e1cb-4712-88a2-6642d0897ce0" containerName="payload-processing"
Sep 11 17:38:17.811007 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.810742    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="24f9d576-b8cf-4d91-833b-9399fa6dd7d9" containerName="maas-api"
Sep 11 17:38:17.815657 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.815623    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm"
Sep 11 17:38:17.819073 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.819050    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-470521-serving-cert\""
Sep 11 17:38:17.826576 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.826547    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm"]
Sep 11 17:38:17.910482 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.910416    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94e3e2a7-dcb6-4d61-b62c-37da2674c99a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm\" (UID: \"94e3e2a7-dcb6-4d61-b62c-37da2674c99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm"
Sep 11 17:38:17.910482 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.910487    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsmlm\" (UniqueName: \"kubernetes.io/projected/94e3e2a7-dcb6-4d61-b62c-37da2674c99a-kube-api-access-nsmlm\") pod \"maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm\" (UID: \"94e3e2a7-dcb6-4d61-b62c-37da2674c99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm"
Sep 11 17:38:17.910663 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.910548    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94e3e2a7-dcb6-4d61-b62c-37da2674c99a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm\" (UID: \"94e3e2a7-dcb6-4d61-b62c-37da2674c99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm"
Sep 11 17:38:17.910663 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:17.910631    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94e3e2a7-dcb6-4d61-b62c-37da2674c99a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm\" (UID: \"94e3e2a7-dcb6-4d61-b62c-37da2674c99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm"
Sep 11 17:38:18.011552 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.011517    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94e3e2a7-dcb6-4d61-b62c-37da2674c99a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm\" (UID: \"94e3e2a7-dcb6-4d61-b62c-37da2674c99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm"
Sep 11 17:38:18.011721 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.011593    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94e3e2a7-dcb6-4d61-b62c-37da2674c99a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm\" (UID: \"94e3e2a7-dcb6-4d61-b62c-37da2674c99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm"
Sep 11 17:38:18.011721 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.011620    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsmlm\" (UniqueName: \"kubernetes.io/projected/94e3e2a7-dcb6-4d61-b62c-37da2674c99a-kube-api-access-nsmlm\") pod \"maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm\" (UID: \"94e3e2a7-dcb6-4d61-b62c-37da2674c99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm"
Sep 11 17:38:18.011721 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.011639    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94e3e2a7-dcb6-4d61-b62c-37da2674c99a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm\" (UID: \"94e3e2a7-dcb6-4d61-b62c-37da2674c99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm"
Sep 11 17:38:18.011828 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:38:18.011763    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-serving-cert: secret "maas-api-e2e-shared-a-w2-470521-serving-cert" not found
Sep 11 17:38:18.011865 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:38:18.011837    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94e3e2a7-dcb6-4d61-b62c-37da2674c99a-maas-api-tls podName:94e3e2a7-dcb6-4d61-b62c-37da2674c99a nodeName:}" failed. No retries permitted until 2026-09-11 17:38:18.511820389 +0000 UTC m=+1066.315992785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/94e3e2a7-dcb6-4d61-b62c-37da2674c99a-maas-api-tls") pod "maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm" (UID: "94e3e2a7-dcb6-4d61-b62c-37da2674c99a") : secret "maas-api-e2e-shared-a-w2-470521-serving-cert" not found
Sep 11 17:38:18.012327 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.012303    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94e3e2a7-dcb6-4d61-b62c-37da2674c99a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm\" (UID: \"94e3e2a7-dcb6-4d61-b62c-37da2674c99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm"
Sep 11 17:38:18.014029 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.014008    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94e3e2a7-dcb6-4d61-b62c-37da2674c99a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm\" (UID: \"94e3e2a7-dcb6-4d61-b62c-37da2674c99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm"
Sep 11 17:38:18.024045 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.024023    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsmlm\" (UniqueName: \"kubernetes.io/projected/94e3e2a7-dcb6-4d61-b62c-37da2674c99a-kube-api-access-nsmlm\") pod \"maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm\" (UID: \"94e3e2a7-dcb6-4d61-b62c-37da2674c99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm"
Sep 11 17:38:18.508357 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.508323    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw"]
Sep 11 17:38:18.512436 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.512415    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw"
Sep 11 17:38:18.515271 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.515248    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-470521\""
Sep 11 17:38:18.516060 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.516037    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94e3e2a7-dcb6-4d61-b62c-37da2674c99a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm\" (UID: \"94e3e2a7-dcb6-4d61-b62c-37da2674c99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm"
Sep 11 17:38:18.518421 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.518399    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94e3e2a7-dcb6-4d61-b62c-37da2674c99a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm\" (UID: \"94e3e2a7-dcb6-4d61-b62c-37da2674c99a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm"
Sep 11 17:38:18.528211 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.528083    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw"]
Sep 11 17:38:18.537544 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.537518    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn"]
Sep 11 17:38:18.541245 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.541225    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn"
Sep 11 17:38:18.544026 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.544001    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-470521-dockercfg-wpwvz\""
Sep 11 17:38:18.551402 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.551375    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn"]
Sep 11 17:38:18.616649 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.616607    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c80eed9-b8a3-409a-b85f-d1c7fea05e94-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw\" (UID: \"4c80eed9-b8a3-409a-b85f-d1c7fea05e94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw"
Sep 11 17:38:18.616818 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.616747    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wzwg\" (UniqueName: \"kubernetes.io/projected/4c80eed9-b8a3-409a-b85f-d1c7fea05e94-kube-api-access-4wzwg\") pod \"payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw\" (UID: \"4c80eed9-b8a3-409a-b85f-d1c7fea05e94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw"
Sep 11 17:38:18.718047 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.718016    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-666xq\" (UniqueName: \"kubernetes.io/projected/a3454246-63c2-4f32-b368-f12af2cab60d-kube-api-access-666xq\") pod \"payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn\" (UID: \"a3454246-63c2-4f32-b368-f12af2cab60d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn"
Sep 11 17:38:18.718208 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.718055    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c80eed9-b8a3-409a-b85f-d1c7fea05e94-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw\" (UID: \"4c80eed9-b8a3-409a-b85f-d1c7fea05e94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw"
Sep 11 17:38:18.718208 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.718116    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wzwg\" (UniqueName: \"kubernetes.io/projected/4c80eed9-b8a3-409a-b85f-d1c7fea05e94-kube-api-access-4wzwg\") pod \"payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw\" (UID: \"4c80eed9-b8a3-409a-b85f-d1c7fea05e94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw"
Sep 11 17:38:18.718208 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.718153    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3454246-63c2-4f32-b368-f12af2cab60d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn\" (UID: \"a3454246-63c2-4f32-b368-f12af2cab60d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn"
Sep 11 17:38:18.718670 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.718653    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c80eed9-b8a3-409a-b85f-d1c7fea05e94-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw\" (UID: \"4c80eed9-b8a3-409a-b85f-d1c7fea05e94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw"
Sep 11 17:38:18.726836 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.726803    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wzwg\" (UniqueName: \"kubernetes.io/projected/4c80eed9-b8a3-409a-b85f-d1c7fea05e94-kube-api-access-4wzwg\") pod \"payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw\" (UID: \"4c80eed9-b8a3-409a-b85f-d1c7fea05e94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw"
Sep 11 17:38:18.727426 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.727411    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm"
Sep 11 17:38:18.818940 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.818873    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3454246-63c2-4f32-b368-f12af2cab60d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn\" (UID: \"a3454246-63c2-4f32-b368-f12af2cab60d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn"
Sep 11 17:38:18.819330 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.819014    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-666xq\" (UniqueName: \"kubernetes.io/projected/a3454246-63c2-4f32-b368-f12af2cab60d-kube-api-access-666xq\") pod \"payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn\" (UID: \"a3454246-63c2-4f32-b368-f12af2cab60d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn"
Sep 11 17:38:18.819591 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.819572    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3454246-63c2-4f32-b368-f12af2cab60d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn\" (UID: \"a3454246-63c2-4f32-b368-f12af2cab60d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn"
Sep 11 17:38:18.827570 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.827542    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-666xq\" (UniqueName: \"kubernetes.io/projected/a3454246-63c2-4f32-b368-f12af2cab60d-kube-api-access-666xq\") pod \"payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn\" (UID: \"a3454246-63c2-4f32-b368-f12af2cab60d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn"
Sep 11 17:38:18.832364 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.832343    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw"
Sep 11 17:38:18.852723 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:18.852692    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn"
Sep 11 17:38:19.005965 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:19.005941    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm"]
Sep 11 17:38:19.007175 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:38:19.007152    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e3e2a7_dcb6_4d61_b62c_37da2674c99a.slice/crio-a277aff9fa9c7ffa38ad5afa50cebdfa8838c5f874f180e9c93a00f0cbaa0b68 WatchSource:0}: Error finding container a277aff9fa9c7ffa38ad5afa50cebdfa8838c5f874f180e9c93a00f0cbaa0b68: Status 404 returned error can't find the container with id a277aff9fa9c7ffa38ad5afa50cebdfa8838c5f874f180e9c93a00f0cbaa0b68
Sep 11 17:38:19.116803 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:19.116774    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw"]
Sep 11 17:38:19.118700 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:38:19.118670    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c80eed9_b8a3_409a_b85f_d1c7fea05e94.slice/crio-ddbb5d92a72c7e0148dff21f1d1a3fe924aebd8839f13d303e3ccfb92b170709 WatchSource:0}: Error finding container ddbb5d92a72c7e0148dff21f1d1a3fe924aebd8839f13d303e3ccfb92b170709: Status 404 returned error can't find the container with id ddbb5d92a72c7e0148dff21f1d1a3fe924aebd8839f13d303e3ccfb92b170709
Sep 11 17:38:19.134022 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:19.133871    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn"]
Sep 11 17:38:19.136906 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:38:19.136873    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3454246_63c2_4f32_b368_f12af2cab60d.slice/crio-8ae37d72ee5b1d193d173a8513fe57abfa957d686e69188d6953d5f1a25b160a WatchSource:0}: Error finding container 8ae37d72ee5b1d193d173a8513fe57abfa957d686e69188d6953d5f1a25b160a: Status 404 returned error can't find the container with id 8ae37d72ee5b1d193d173a8513fe57abfa957d686e69188d6953d5f1a25b160a
Sep 11 17:38:19.985938 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:19.985856    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn" event={"ID":"a3454246-63c2-4f32-b368-f12af2cab60d","Type":"ContainerStarted","Data":"a0f3bf54cec9281088828ef4bc4673bb29983e67abf1ee7f9923cdbbfaeaac14"}
Sep 11 17:38:19.985938 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:19.985952    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn" event={"ID":"a3454246-63c2-4f32-b368-f12af2cab60d","Type":"ContainerStarted","Data":"8ae37d72ee5b1d193d173a8513fe57abfa957d686e69188d6953d5f1a25b160a"}
Sep 11 17:38:19.986477 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:19.986003    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn"
Sep 11 17:38:19.987646 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:19.987623    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw" event={"ID":"4c80eed9-b8a3-409a-b85f-d1c7fea05e94","Type":"ContainerStarted","Data":"10b4a8e4f92f81312ad7a5a7238062393be686463cabd00e6c1926c9249c50a9"}
Sep 11 17:38:19.987772 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:19.987675    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw"
Sep 11 17:38:19.987772 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:19.987703    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw" event={"ID":"4c80eed9-b8a3-409a-b85f-d1c7fea05e94","Type":"ContainerStarted","Data":"ddbb5d92a72c7e0148dff21f1d1a3fe924aebd8839f13d303e3ccfb92b170709"}
Sep 11 17:38:19.989453 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:19.989427    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm" event={"ID":"94e3e2a7-dcb6-4d61-b62c-37da2674c99a","Type":"ContainerStarted","Data":"5796f3d7547ebc5c58b45633b5641ccdfe791861f2111a7af519c1d47a8255e3"}
Sep 11 17:38:19.989570 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:19.989476    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm"
Sep 11 17:38:19.989570 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:19.989506    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm" event={"ID":"94e3e2a7-dcb6-4d61-b62c-37da2674c99a","Type":"ContainerStarted","Data":"a277aff9fa9c7ffa38ad5afa50cebdfa8838c5f874f180e9c93a00f0cbaa0b68"}
Sep 11 17:38:20.002280 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:20.002223    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn" podStartSLOduration=2.002201151 podStartE2EDuration="2.002201151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:38:20.00164269 +0000 UTC m=+1067.805815105" watchObservedRunningTime="2026-09-11 17:38:20.002201151 +0000 UTC m=+1067.806373573"
Sep 11 17:38:20.021111 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:20.021057    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm" podStartSLOduration=2.618756511 podStartE2EDuration="3.021038021s" totalImagesPullingTime="402.28151ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:38:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:38:20.018373179 +0000 UTC m=+1067.822545599" watchObservedRunningTime="2026-09-11 17:38:20.021038021 +0000 UTC m=+1067.825210442"
Sep 11 17:38:20.036448 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:20.036399    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw" podStartSLOduration=2.03638427 podStartE2EDuration="2.03638427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:38:20.034671278 +0000 UTC m=+1067.838843696" watchObservedRunningTime="2026-09-11 17:38:20.03638427 +0000 UTC m=+1067.840556693"
Sep 11 17:38:26.000545 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:26.000466    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm"
Sep 11 17:38:30.997882 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:30.997735    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw"
Sep 11 17:38:30.997882 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:30.997876    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn"
Sep 11 17:38:38.176779 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:38.176739    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv"]
Sep 11 17:38:38.181481 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:38.181460    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv"
Sep 11 17:38:38.183930 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:38.183890    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-b6b979\""
Sep 11 17:38:38.184050 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:38.184018    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-b6b979-dockercfg-7nblc\""
Sep 11 17:38:38.193431 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:38.193405    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv"]
Sep 11 17:38:38.311948 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:38.311907    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aeb1857-85d7-4fd0-b267-68d0e60738dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv\" (UID: \"9aeb1857-85d7-4fd0-b267-68d0e60738dc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv"
Sep 11 17:38:38.312124 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:38.312043    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w7gj\" (UniqueName: \"kubernetes.io/projected/9aeb1857-85d7-4fd0-b267-68d0e60738dc-kube-api-access-4w7gj\") pod \"payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv\" (UID: \"9aeb1857-85d7-4fd0-b267-68d0e60738dc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv"
Sep 11 17:38:38.413307 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:38.413268    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aeb1857-85d7-4fd0-b267-68d0e60738dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv\" (UID: \"9aeb1857-85d7-4fd0-b267-68d0e60738dc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv"
Sep 11 17:38:38.413498 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:38.413379    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w7gj\" (UniqueName: \"kubernetes.io/projected/9aeb1857-85d7-4fd0-b267-68d0e60738dc-kube-api-access-4w7gj\") pod \"payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv\" (UID: \"9aeb1857-85d7-4fd0-b267-68d0e60738dc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv"
Sep 11 17:38:38.414051 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:38.414029    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aeb1857-85d7-4fd0-b267-68d0e60738dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv\" (UID: \"9aeb1857-85d7-4fd0-b267-68d0e60738dc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv"
Sep 11 17:38:38.422852 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:38.422823    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w7gj\" (UniqueName: \"kubernetes.io/projected/9aeb1857-85d7-4fd0-b267-68d0e60738dc-kube-api-access-4w7gj\") pod \"payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv\" (UID: \"9aeb1857-85d7-4fd0-b267-68d0e60738dc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv"
Sep 11 17:38:38.491612 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:38.491523    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv"
Sep 11 17:38:38.779224 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:38.779198    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv"]
Sep 11 17:38:38.780957 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:38:38.780931    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aeb1857_85d7_4fd0_b267_68d0e60738dc.slice/crio-c28e8555720cdd63259bc585af753fe4f5ac6cd9236fed4ca6be81c66d5ee2ef WatchSource:0}: Error finding container c28e8555720cdd63259bc585af753fe4f5ac6cd9236fed4ca6be81c66d5ee2ef: Status 404 returned error can't find the container with id c28e8555720cdd63259bc585af753fe4f5ac6cd9236fed4ca6be81c66d5ee2ef
Sep 11 17:38:39.089877 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:39.089833    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv" event={"ID":"9aeb1857-85d7-4fd0-b267-68d0e60738dc","Type":"ContainerStarted","Data":"2f3f32526b5ec68851be1b2d85510d18088c47ac5b6f996e9308da12ad75f42d"}
Sep 11 17:38:39.090139 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:39.089926    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv" event={"ID":"9aeb1857-85d7-4fd0-b267-68d0e60738dc","Type":"ContainerStarted","Data":"c28e8555720cdd63259bc585af753fe4f5ac6cd9236fed4ca6be81c66d5ee2ef"}
Sep 11 17:38:39.090139 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:39.089978    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv"
Sep 11 17:38:39.114150 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:39.113741    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv" podStartSLOduration=1.113717794 podStartE2EDuration="1.113717794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:38:39.113297613 +0000 UTC m=+1086.917470033" watchObservedRunningTime="2026-09-11 17:38:39.113717794 +0000 UTC m=+1086.917890214"
Sep 11 17:38:39.178476 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:39.178433    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd"]
Sep 11 17:38:39.182286 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:39.182261    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd"
Sep 11 17:38:39.186742 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:39.186717    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-4c36f4\""
Sep 11 17:38:39.196488 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:39.196457    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd"]
Sep 11 17:38:39.210618 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:39.210588    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct"]
Sep 11 17:38:39.214961 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:39.214935    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct"
Sep 11 17:38:39.217555 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:39.217530    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-4c36f4-dockercfg-cwh7x\""
Sep 11 17:38:39.221157 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:39.221133    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhmj4\" (UniqueName: \"kubernetes.io/projected/16b9fd20-840b-47a2-a977-2b67f535c831-kube-api-access-rhmj4\") pod \"payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd\" (UID: \"16b9fd20-840b-47a2-a977-2b67f535c831\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd"
Sep 11 17:38:39.221252 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:39.221191    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b9fd20-840b-47a2-a977-2b67f535c831-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd\" (UID: \"16b9fd20-840b-47a2-a977-2b67f535c831\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd"
Sep 11 17:38:39.227463 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:39.227440    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct"]
Sep 11 17:38:39.321908 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:39.321864    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d27723b-8c92-45d2-ab77-80ecb372d5ef-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct\" (UID: \"8d27723b-8c92-45d2-ab77-80ecb372d5ef\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct"
Sep 11 17:38:39.322099 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:39.321995    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhmj4\" (UniqueName: \"kubernetes.io/projected/16b9fd20-840b-47a2-a977-2b67f535c831-kube-api-access-rhmj4\") pod \"payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd\" (UID: \"16b9fd20-840b-47a2-a977-2b67f535c831\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd"
Sep 11 17:38:39.322099 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:39.322051    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp4z7\" (UniqueName: \"kubernetes.io/projected/8d27723b-8c92-45d2-ab77-80ecb372d5ef-kube-api-access-dp4z7\") pod \"payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct\" (UID: \"8d27723b-8c92-45d2-ab77-80ecb372d5ef\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct"
Sep 11 17:38:39.322214 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:39.322111    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b9fd20-840b-47a2-a977-2b67f535c831-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd\" (UID: \"16b9fd20-840b-47a2-a977-2b67f535c831\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd"
Sep 11 17:38:39.323023 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:39.323000    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b9fd20-840b-47a2-a977-2b67f535c831-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd\" (UID: \"16b9fd20-840b-47a2-a977-2b67f535c831\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd"
Sep 11 17:38:39.334030 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:39.334007    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhmj4\" (UniqueName: \"kubernetes.io/projected/16b9fd20-840b-47a2-a977-2b67f535c831-kube-api-access-rhmj4\") pod \"payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd\" (UID: \"16b9fd20-840b-47a2-a977-2b67f535c831\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd"
Sep 11 17:38:39.423165 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:39.423074    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d27723b-8c92-45d2-ab77-80ecb372d5ef-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct\" (UID: \"8d27723b-8c92-45d2-ab77-80ecb372d5ef\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct"
Sep 11 17:38:39.423165 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:39.423147    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp4z7\" (UniqueName: \"kubernetes.io/projected/8d27723b-8c92-45d2-ab77-80ecb372d5ef-kube-api-access-dp4z7\") pod \"payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct\" (UID: \"8d27723b-8c92-45d2-ab77-80ecb372d5ef\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct"
Sep 11 17:38:39.423797 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:39.423777    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d27723b-8c92-45d2-ab77-80ecb372d5ef-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct\" (UID: \"8d27723b-8c92-45d2-ab77-80ecb372d5ef\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct"
Sep 11 17:38:39.433849 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:39.433822    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp4z7\" (UniqueName: \"kubernetes.io/projected/8d27723b-8c92-45d2-ab77-80ecb372d5ef-kube-api-access-dp4z7\") pod \"payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct\" (UID: \"8d27723b-8c92-45d2-ab77-80ecb372d5ef\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct"
Sep 11 17:38:39.492859 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:39.492819    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd"
Sep 11 17:38:39.526282 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:39.526241    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct"
Sep 11 17:38:39.764891 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:39.764664    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd"]
Sep 11 17:38:39.766515 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:38:39.766490    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b9fd20_840b_47a2_a977_2b67f535c831.slice/crio-caa034113ca4a64c4c3b5eef59b608a1f4326bb94d8c5e4149263530042edc77 WatchSource:0}: Error finding container caa034113ca4a64c4c3b5eef59b608a1f4326bb94d8c5e4149263530042edc77: Status 404 returned error can't find the container with id caa034113ca4a64c4c3b5eef59b608a1f4326bb94d8c5e4149263530042edc77
Sep 11 17:38:39.817380 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:39.817352    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct"]
Sep 11 17:38:39.824112 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:38:39.824084    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d27723b_8c92_45d2_ab77_80ecb372d5ef.slice/crio-973a6ac62fad0649ee58d7b26a0ab3a8fdfc2e37b5ec04438c48b1b418a8760b WatchSource:0}: Error finding container 973a6ac62fad0649ee58d7b26a0ab3a8fdfc2e37b5ec04438c48b1b418a8760b: Status 404 returned error can't find the container with id 973a6ac62fad0649ee58d7b26a0ab3a8fdfc2e37b5ec04438c48b1b418a8760b
Sep 11 17:38:40.099041 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:40.098995    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct" event={"ID":"8d27723b-8c92-45d2-ab77-80ecb372d5ef","Type":"ContainerStarted","Data":"96c8a74b412bc828e0856a745e1d4e80a43a041e365f1fe2f298f6ef2bd35eb7"}
Sep 11 17:38:40.099246 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:40.099219    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct"
Sep 11 17:38:40.099310 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:40.099252    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct" event={"ID":"8d27723b-8c92-45d2-ab77-80ecb372d5ef","Type":"ContainerStarted","Data":"973a6ac62fad0649ee58d7b26a0ab3a8fdfc2e37b5ec04438c48b1b418a8760b"}
Sep 11 17:38:40.101614 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:40.101584    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd" event={"ID":"16b9fd20-840b-47a2-a977-2b67f535c831","Type":"ContainerStarted","Data":"35e4e37aab1e606acbae1ce93f61a84f0f5c72563ec91e05f45e193fa47a275b"}
Sep 11 17:38:40.101754 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:40.101635    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd" event={"ID":"16b9fd20-840b-47a2-a977-2b67f535c831","Type":"ContainerStarted","Data":"caa034113ca4a64c4c3b5eef59b608a1f4326bb94d8c5e4149263530042edc77"}
Sep 11 17:38:40.101754 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:40.101663    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd"
Sep 11 17:38:40.117258 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:40.117187    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct" podStartSLOduration=1.117173626 podStartE2EDuration="1.117173626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:38:40.114550608 +0000 UTC m=+1087.918723026" watchObservedRunningTime="2026-09-11 17:38:40.117173626 +0000 UTC m=+1087.921346045"
Sep 11 17:38:40.130203 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:40.130153    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd" podStartSLOduration=1.130137626 podStartE2EDuration="1.130137626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:38:40.129619954 +0000 UTC m=+1087.933792378" watchObservedRunningTime="2026-09-11 17:38:40.130137626 +0000 UTC m=+1087.934310045"
Sep 11 17:38:50.103762 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:50.103727    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv"
Sep 11 17:38:51.110499 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:51.110467    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct"
Sep 11 17:38:51.111025 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:38:51.111006    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd"
Sep 11 17:39:11.701908 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:11.701855    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2"]
Sep 11 17:39:11.707400 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:11.707372    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2"
Sep 11 17:39:11.710091 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:11.710065    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-db95fd-serving-cert\""
Sep 11 17:39:11.719143 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:11.719113    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2"]
Sep 11 17:39:11.726502 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:11.726463    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e9776f8-d9e6-4fda-a3ab-df684c35a71e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2\" (UID: \"8e9776f8-d9e6-4fda-a3ab-df684c35a71e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2"
Sep 11 17:39:11.726672 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:11.726513    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e9776f8-d9e6-4fda-a3ab-df684c35a71e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2\" (UID: \"8e9776f8-d9e6-4fda-a3ab-df684c35a71e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2"
Sep 11 17:39:11.726672 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:11.726589    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e9776f8-d9e6-4fda-a3ab-df684c35a71e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2\" (UID: \"8e9776f8-d9e6-4fda-a3ab-df684c35a71e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2"
Sep 11 17:39:11.726794 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:11.726734    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rg25\" (UniqueName: \"kubernetes.io/projected/8e9776f8-d9e6-4fda-a3ab-df684c35a71e-kube-api-access-2rg25\") pod \"maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2\" (UID: \"8e9776f8-d9e6-4fda-a3ab-df684c35a71e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2"
Sep 11 17:39:11.827582 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:11.827541    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e9776f8-d9e6-4fda-a3ab-df684c35a71e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2\" (UID: \"8e9776f8-d9e6-4fda-a3ab-df684c35a71e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2"
Sep 11 17:39:11.827582 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:11.827582    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e9776f8-d9e6-4fda-a3ab-df684c35a71e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2\" (UID: \"8e9776f8-d9e6-4fda-a3ab-df684c35a71e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2"
Sep 11 17:39:11.827854 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:11.827625    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e9776f8-d9e6-4fda-a3ab-df684c35a71e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2\" (UID: \"8e9776f8-d9e6-4fda-a3ab-df684c35a71e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2"
Sep 11 17:39:11.827854 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:11.827699    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rg25\" (UniqueName: \"kubernetes.io/projected/8e9776f8-d9e6-4fda-a3ab-df684c35a71e-kube-api-access-2rg25\") pod \"maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2\" (UID: \"8e9776f8-d9e6-4fda-a3ab-df684c35a71e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2"
Sep 11 17:39:11.827854 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:39:11.827837    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-serving-cert: secret "maas-api-e2e-shared-b-w3-db95fd-serving-cert" not found
Sep 11 17:39:11.828053 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:39:11.827953    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e9776f8-d9e6-4fda-a3ab-df684c35a71e-maas-api-tls podName:8e9776f8-d9e6-4fda-a3ab-df684c35a71e nodeName:}" failed. No retries permitted until 2026-09-11 17:39:12.327932194 +0000 UTC m=+1120.132104598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8e9776f8-d9e6-4fda-a3ab-df684c35a71e-maas-api-tls") pod "maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2" (UID: "8e9776f8-d9e6-4fda-a3ab-df684c35a71e") : secret "maas-api-e2e-shared-b-w3-db95fd-serving-cert" not found
Sep 11 17:39:11.828339 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:11.828300    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e9776f8-d9e6-4fda-a3ab-df684c35a71e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2\" (UID: \"8e9776f8-d9e6-4fda-a3ab-df684c35a71e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2"
Sep 11 17:39:11.830201 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:11.830182    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e9776f8-d9e6-4fda-a3ab-df684c35a71e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2\" (UID: \"8e9776f8-d9e6-4fda-a3ab-df684c35a71e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2"
Sep 11 17:39:11.846806 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:11.846780    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rg25\" (UniqueName: \"kubernetes.io/projected/8e9776f8-d9e6-4fda-a3ab-df684c35a71e-kube-api-access-2rg25\") pod \"maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2\" (UID: \"8e9776f8-d9e6-4fda-a3ab-df684c35a71e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2"
Sep 11 17:39:12.333258 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:12.333218    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e9776f8-d9e6-4fda-a3ab-df684c35a71e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2\" (UID: \"8e9776f8-d9e6-4fda-a3ab-df684c35a71e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2"
Sep 11 17:39:12.335480 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:12.335454    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e9776f8-d9e6-4fda-a3ab-df684c35a71e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2\" (UID: \"8e9776f8-d9e6-4fda-a3ab-df684c35a71e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2"
Sep 11 17:39:12.486608 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:12.486574    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq"]
Sep 11 17:39:12.490550 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:12.490530    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq"
Sep 11 17:39:12.493014 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:12.492987    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-db95fd\""
Sep 11 17:39:12.493136 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:12.493037    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-db95fd-dockercfg-6jdl2\""
Sep 11 17:39:12.501325 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:12.501299    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq"]
Sep 11 17:39:12.535521 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:12.535482    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c72tf\" (UniqueName: \"kubernetes.io/projected/1cc5f681-b4a2-438e-9460-71f081ba5c73-kube-api-access-c72tf\") pod \"payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq\" (UID: \"1cc5f681-b4a2-438e-9460-71f081ba5c73\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq"
Sep 11 17:39:12.535521 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:12.535528    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc5f681-b4a2-438e-9460-71f081ba5c73-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq\" (UID: \"1cc5f681-b4a2-438e-9460-71f081ba5c73\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq"
Sep 11 17:39:12.621218 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:12.621137    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2"
Sep 11 17:39:12.636684 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:12.636649    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc5f681-b4a2-438e-9460-71f081ba5c73-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq\" (UID: \"1cc5f681-b4a2-438e-9460-71f081ba5c73\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq"
Sep 11 17:39:12.636849 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:12.636828    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c72tf\" (UniqueName: \"kubernetes.io/projected/1cc5f681-b4a2-438e-9460-71f081ba5c73-kube-api-access-c72tf\") pod \"payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq\" (UID: \"1cc5f681-b4a2-438e-9460-71f081ba5c73\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq"
Sep 11 17:39:12.637412 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:12.637388    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc5f681-b4a2-438e-9460-71f081ba5c73-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq\" (UID: \"1cc5f681-b4a2-438e-9460-71f081ba5c73\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq"
Sep 11 17:39:12.644969 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:12.644941    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c72tf\" (UniqueName: \"kubernetes.io/projected/1cc5f681-b4a2-438e-9460-71f081ba5c73-kube-api-access-c72tf\") pod \"payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq\" (UID: \"1cc5f681-b4a2-438e-9460-71f081ba5c73\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq"
Sep 11 17:39:12.802591 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:12.802552    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq"
Sep 11 17:39:12.896209 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:12.896046    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2"]
Sep 11 17:39:12.900373 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:39:12.900343    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e9776f8_d9e6_4fda_a3ab_df684c35a71e.slice/crio-d78114ddea4f9b73971e3b3687831c28cf97c08da5a04d01a02000983f674d67 WatchSource:0}: Error finding container d78114ddea4f9b73971e3b3687831c28cf97c08da5a04d01a02000983f674d67: Status 404 returned error can't find the container with id d78114ddea4f9b73971e3b3687831c28cf97c08da5a04d01a02000983f674d67
Sep 11 17:39:13.075219 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:13.075195    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq"]
Sep 11 17:39:13.077115 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:39:13.077084    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc5f681_b4a2_438e_9460_71f081ba5c73.slice/crio-71405a34792dc98e38f870dfc6b570ad6e4592ac3f870dcb6ce3da593a7ca6da WatchSource:0}: Error finding container 71405a34792dc98e38f870dfc6b570ad6e4592ac3f870dcb6ce3da593a7ca6da: Status 404 returned error can't find the container with id 71405a34792dc98e38f870dfc6b570ad6e4592ac3f870dcb6ce3da593a7ca6da
Sep 11 17:39:13.275699 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:13.275665    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq" event={"ID":"1cc5f681-b4a2-438e-9460-71f081ba5c73","Type":"ContainerStarted","Data":"d0379ae05b21843938729073b178b46b97d0990f70901376637baa498eb28a63"}
Sep 11 17:39:13.275882 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:13.275728    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq"
Sep 11 17:39:13.275882 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:13.275756    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq" event={"ID":"1cc5f681-b4a2-438e-9460-71f081ba5c73","Type":"ContainerStarted","Data":"71405a34792dc98e38f870dfc6b570ad6e4592ac3f870dcb6ce3da593a7ca6da"}
Sep 11 17:39:13.276953 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:13.276928    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2" event={"ID":"8e9776f8-d9e6-4fda-a3ab-df684c35a71e","Type":"ContainerStarted","Data":"d78114ddea4f9b73971e3b3687831c28cf97c08da5a04d01a02000983f674d67"}
Sep 11 17:39:13.294658 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:13.294601    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq" podStartSLOduration=1.294582594 podStartE2EDuration="1.294582594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:13.293553159 +0000 UTC m=+1121.097725583" watchObservedRunningTime="2026-09-11 17:39:13.294582594 +0000 UTC m=+1121.098755017"
Sep 11 17:39:14.283237 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:14.283202    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2" event={"ID":"8e9776f8-d9e6-4fda-a3ab-df684c35a71e","Type":"ContainerStarted","Data":"db603b63f6b32c0f2003cd1b7dff037ef6726be302dbcbb725cac7d0d7cd3a09"}
Sep 11 17:39:14.302983 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:14.302929    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2" podStartSLOduration=2.838825442 podStartE2EDuration="3.302913801s" totalImagesPullingTime="464.088359ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:14.300707582 +0000 UTC m=+1122.104880002" watchObservedRunningTime="2026-09-11 17:39:14.302913801 +0000 UTC m=+1122.107086213"
Sep 11 17:39:15.288053 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:15.288026    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2"
Sep 11 17:39:20.399053 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:20.399012    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5"]
Sep 11 17:39:20.404410 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:20.404380    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5"
Sep 11 17:39:20.406815 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:20.406792    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3d144889-serving-cert\""
Sep 11 17:39:20.413373 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:20.413341    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5"]
Sep 11 17:39:20.518405 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:20.518365    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b6e9e7e-6399-4b10-9c44-cbe775fe2f56-openshift-service-ca\") pod \"maas-api-e2e-mt-3d144889-776cc67f77-2jqr5\" (UID: \"6b6e9e7e-6399-4b10-9c44-cbe775fe2f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5"
Sep 11 17:39:20.518578 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:20.518513    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lckjq\" (UniqueName: \"kubernetes.io/projected/6b6e9e7e-6399-4b10-9c44-cbe775fe2f56-kube-api-access-lckjq\") pod \"maas-api-e2e-mt-3d144889-776cc67f77-2jqr5\" (UID: \"6b6e9e7e-6399-4b10-9c44-cbe775fe2f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5"
Sep 11 17:39:20.518578 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:20.518561    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6e9e7e-6399-4b10-9c44-cbe775fe2f56-metrics-certs\") pod \"maas-api-e2e-mt-3d144889-776cc67f77-2jqr5\" (UID: \"6b6e9e7e-6399-4b10-9c44-cbe775fe2f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5"
Sep 11 17:39:20.518663 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:20.518602    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6e9e7e-6399-4b10-9c44-cbe775fe2f56-maas-api-tls\") pod \"maas-api-e2e-mt-3d144889-776cc67f77-2jqr5\" (UID: \"6b6e9e7e-6399-4b10-9c44-cbe775fe2f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5"
Sep 11 17:39:20.619726 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:20.619689    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lckjq\" (UniqueName: \"kubernetes.io/projected/6b6e9e7e-6399-4b10-9c44-cbe775fe2f56-kube-api-access-lckjq\") pod \"maas-api-e2e-mt-3d144889-776cc67f77-2jqr5\" (UID: \"6b6e9e7e-6399-4b10-9c44-cbe775fe2f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5"
Sep 11 17:39:20.619726 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:20.619728    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6e9e7e-6399-4b10-9c44-cbe775fe2f56-metrics-certs\") pod \"maas-api-e2e-mt-3d144889-776cc67f77-2jqr5\" (UID: \"6b6e9e7e-6399-4b10-9c44-cbe775fe2f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5"
Sep 11 17:39:20.619973 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:20.619755    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6e9e7e-6399-4b10-9c44-cbe775fe2f56-maas-api-tls\") pod \"maas-api-e2e-mt-3d144889-776cc67f77-2jqr5\" (UID: \"6b6e9e7e-6399-4b10-9c44-cbe775fe2f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5"
Sep 11 17:39:20.619973 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:20.619819    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b6e9e7e-6399-4b10-9c44-cbe775fe2f56-openshift-service-ca\") pod \"maas-api-e2e-mt-3d144889-776cc67f77-2jqr5\" (UID: \"6b6e9e7e-6399-4b10-9c44-cbe775fe2f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5"
Sep 11 17:39:20.619973 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:39:20.619936    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-serving-cert: secret "maas-api-e2e-mt-3d144889-serving-cert" not found
Sep 11 17:39:20.620115 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:39:20.620003    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b6e9e7e-6399-4b10-9c44-cbe775fe2f56-maas-api-tls podName:6b6e9e7e-6399-4b10-9c44-cbe775fe2f56 nodeName:}" failed. No retries permitted until 2026-09-11 17:39:21.119985918 +0000 UTC m=+1128.924158321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6b6e9e7e-6399-4b10-9c44-cbe775fe2f56-maas-api-tls") pod "maas-api-e2e-mt-3d144889-776cc67f77-2jqr5" (UID: "6b6e9e7e-6399-4b10-9c44-cbe775fe2f56") : secret "maas-api-e2e-mt-3d144889-serving-cert" not found
Sep 11 17:39:20.620520 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:20.620497    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b6e9e7e-6399-4b10-9c44-cbe775fe2f56-openshift-service-ca\") pod \"maas-api-e2e-mt-3d144889-776cc67f77-2jqr5\" (UID: \"6b6e9e7e-6399-4b10-9c44-cbe775fe2f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5"
Sep 11 17:39:20.622288 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:20.622263    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6e9e7e-6399-4b10-9c44-cbe775fe2f56-metrics-certs\") pod \"maas-api-e2e-mt-3d144889-776cc67f77-2jqr5\" (UID: \"6b6e9e7e-6399-4b10-9c44-cbe775fe2f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5"
Sep 11 17:39:20.629257 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:20.629235    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lckjq\" (UniqueName: \"kubernetes.io/projected/6b6e9e7e-6399-4b10-9c44-cbe775fe2f56-kube-api-access-lckjq\") pod \"maas-api-e2e-mt-3d144889-776cc67f77-2jqr5\" (UID: \"6b6e9e7e-6399-4b10-9c44-cbe775fe2f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5"
Sep 11 17:39:21.125889 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:21.125854    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6e9e7e-6399-4b10-9c44-cbe775fe2f56-maas-api-tls\") pod \"maas-api-e2e-mt-3d144889-776cc67f77-2jqr5\" (UID: \"6b6e9e7e-6399-4b10-9c44-cbe775fe2f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5"
Sep 11 17:39:21.128308 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:21.128280    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6e9e7e-6399-4b10-9c44-cbe775fe2f56-maas-api-tls\") pod \"maas-api-e2e-mt-3d144889-776cc67f77-2jqr5\" (UID: \"6b6e9e7e-6399-4b10-9c44-cbe775fe2f56\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5"
Sep 11 17:39:21.298888 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:21.298856    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2"
Sep 11 17:39:21.318873 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:21.318843    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5"
Sep 11 17:39:21.603470 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:21.603408    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5"]
Sep 11 17:39:21.605873 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:39:21.605843    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6e9e7e_6399_4b10_9c44_cbe775fe2f56.slice/crio-e47c28ddd47275055d567cb654b450d02ab5ad774cc2afc0e1ef3a6dd830cd3f WatchSource:0}: Error finding container e47c28ddd47275055d567cb654b450d02ab5ad774cc2afc0e1ef3a6dd830cd3f: Status 404 returned error can't find the container with id e47c28ddd47275055d567cb654b450d02ab5ad774cc2afc0e1ef3a6dd830cd3f
Sep 11 17:39:22.324048 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:22.324017    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5" event={"ID":"6b6e9e7e-6399-4b10-9c44-cbe775fe2f56","Type":"ContainerStarted","Data":"49cf361cc8d95df14510909a6784cd79cb03e7c4913c80aaeb5275c2efe492bb"}
Sep 11 17:39:22.324201 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:22.324067    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5" event={"ID":"6b6e9e7e-6399-4b10-9c44-cbe775fe2f56","Type":"ContainerStarted","Data":"e47c28ddd47275055d567cb654b450d02ab5ad774cc2afc0e1ef3a6dd830cd3f"}
Sep 11 17:39:22.324201 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:22.324100    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5"
Sep 11 17:39:22.346174 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:22.346119    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5" podStartSLOduration=1.9948744710000001 podStartE2EDuration="2.346102916s" totalImagesPullingTime="351.228445ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:39:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:22.342516922 +0000 UTC m=+1130.146689338" watchObservedRunningTime="2026-09-11 17:39:22.346102916 +0000 UTC m=+1130.150275331"
Sep 11 17:39:24.286202 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:24.286170    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq"
Sep 11 17:39:28.334442 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:39:28.334411    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5"
Sep 11 17:40:26.641082 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:40:26.641048    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5"]
Sep 11 17:40:26.641641 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:40:26.641241    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5" podUID="6b6e9e7e-6399-4b10-9c44-cbe775fe2f56" containerName="maas-api" containerID="cri-o://49cf361cc8d95df14510909a6784cd79cb03e7c4913c80aaeb5275c2efe492bb" gracePeriod=30
Sep 11 17:40:26.894068 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:40:26.894010    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5"
Sep 11 17:40:26.965784 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:40:26.965750    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b6e9e7e-6399-4b10-9c44-cbe775fe2f56-openshift-service-ca\") pod \"6b6e9e7e-6399-4b10-9c44-cbe775fe2f56\" (UID: \"6b6e9e7e-6399-4b10-9c44-cbe775fe2f56\") "
Sep 11 17:40:26.965967 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:40:26.965816    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6e9e7e-6399-4b10-9c44-cbe775fe2f56-metrics-certs\") pod \"6b6e9e7e-6399-4b10-9c44-cbe775fe2f56\" (UID: \"6b6e9e7e-6399-4b10-9c44-cbe775fe2f56\") "
Sep 11 17:40:26.965967 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:40:26.965839    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lckjq\" (UniqueName: \"kubernetes.io/projected/6b6e9e7e-6399-4b10-9c44-cbe775fe2f56-kube-api-access-lckjq\") pod \"6b6e9e7e-6399-4b10-9c44-cbe775fe2f56\" (UID: \"6b6e9e7e-6399-4b10-9c44-cbe775fe2f56\") "
Sep 11 17:40:26.965967 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:40:26.965924    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6e9e7e-6399-4b10-9c44-cbe775fe2f56-maas-api-tls\") pod \"6b6e9e7e-6399-4b10-9c44-cbe775fe2f56\" (UID: \"6b6e9e7e-6399-4b10-9c44-cbe775fe2f56\") "
Sep 11 17:40:26.966222 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:40:26.966190    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b6e9e7e-6399-4b10-9c44-cbe775fe2f56-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6b6e9e7e-6399-4b10-9c44-cbe775fe2f56" (UID: "6b6e9e7e-6399-4b10-9c44-cbe775fe2f56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:40:26.968163 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:40:26.968138    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b6e9e7e-6399-4b10-9c44-cbe775fe2f56-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6b6e9e7e-6399-4b10-9c44-cbe775fe2f56" (UID: "6b6e9e7e-6399-4b10-9c44-cbe775fe2f56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:40:26.968267 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:40:26.968202    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6e9e7e-6399-4b10-9c44-cbe775fe2f56-kube-api-access-lckjq" (OuterVolumeSpecName: "kube-api-access-lckjq") pod "6b6e9e7e-6399-4b10-9c44-cbe775fe2f56" (UID: "6b6e9e7e-6399-4b10-9c44-cbe775fe2f56"). InnerVolumeSpecName "kube-api-access-lckjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:40:26.968267 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:40:26.968226    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b6e9e7e-6399-4b10-9c44-cbe775fe2f56-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6b6e9e7e-6399-4b10-9c44-cbe775fe2f56" (UID: "6b6e9e7e-6399-4b10-9c44-cbe775fe2f56"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:40:27.067784 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:40:27.067747    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b6e9e7e-6399-4b10-9c44-cbe775fe2f56-openshift-service-ca\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:40:27.067784 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:40:27.067778    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6e9e7e-6399-4b10-9c44-cbe775fe2f56-metrics-certs\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:40:27.067784 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:40:27.067788    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lckjq\" (UniqueName: \"kubernetes.io/projected/6b6e9e7e-6399-4b10-9c44-cbe775fe2f56-kube-api-access-lckjq\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:40:27.068048 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:40:27.067796    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6e9e7e-6399-4b10-9c44-cbe775fe2f56-maas-api-tls\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:40:27.667418 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:40:27.667384    2560 generic.go:356] "Generic (PLEG): container finished" podID="6b6e9e7e-6399-4b10-9c44-cbe775fe2f56" containerID="49cf361cc8d95df14510909a6784cd79cb03e7c4913c80aaeb5275c2efe492bb" exitCode=0
Sep 11 17:40:27.667934 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:40:27.667439    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5"
Sep 11 17:40:27.667934 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:40:27.667461    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5" event={"ID":"6b6e9e7e-6399-4b10-9c44-cbe775fe2f56","Type":"ContainerDied","Data":"49cf361cc8d95df14510909a6784cd79cb03e7c4913c80aaeb5275c2efe492bb"}
Sep 11 17:40:27.667934 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:40:27.667508    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5" event={"ID":"6b6e9e7e-6399-4b10-9c44-cbe775fe2f56","Type":"ContainerDied","Data":"e47c28ddd47275055d567cb654b450d02ab5ad774cc2afc0e1ef3a6dd830cd3f"}
Sep 11 17:40:27.667934 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:40:27.667535    2560 scope.go:117] "RemoveContainer" containerID="49cf361cc8d95df14510909a6784cd79cb03e7c4913c80aaeb5275c2efe492bb"
Sep 11 17:40:27.677198 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:40:27.677177    2560 scope.go:117] "RemoveContainer" containerID="49cf361cc8d95df14510909a6784cd79cb03e7c4913c80aaeb5275c2efe492bb"
Sep 11 17:40:27.677499 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:40:27.677476    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49cf361cc8d95df14510909a6784cd79cb03e7c4913c80aaeb5275c2efe492bb\": container with ID starting with 49cf361cc8d95df14510909a6784cd79cb03e7c4913c80aaeb5275c2efe492bb not found: ID does not exist" containerID="49cf361cc8d95df14510909a6784cd79cb03e7c4913c80aaeb5275c2efe492bb"
Sep 11 17:40:27.677586 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:40:27.677508    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49cf361cc8d95df14510909a6784cd79cb03e7c4913c80aaeb5275c2efe492bb"} err="failed to get container status \"49cf361cc8d95df14510909a6784cd79cb03e7c4913c80aaeb5275c2efe492bb\": rpc error: code = NotFound desc = could not find container \"49cf361cc8d95df14510909a6784cd79cb03e7c4913c80aaeb5275c2efe492bb\": container with ID starting with 49cf361cc8d95df14510909a6784cd79cb03e7c4913c80aaeb5275c2efe492bb not found: ID does not exist"
Sep 11 17:40:27.693961 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:40:27.693926    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5"]
Sep 11 17:40:27.698162 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:40:27.698137    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d144889-776cc67f77-2jqr5"]
Sep 11 17:40:28.715757 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:40:28.715722    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b6e9e7e-6399-4b10-9c44-cbe775fe2f56" path="/var/lib/kubelet/pods/6b6e9e7e-6399-4b10-9c44-cbe775fe2f56/volumes"
Sep 11 17:40:32.669667 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:40:32.669636    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6fcjs_477cac01-ab9e-4441-b07d-b7e53401c616/console-operator/1.log"
Sep 11 17:40:32.673758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:40:32.673733    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6fcjs_477cac01-ab9e-4441-b07d-b7e53401c616/console-operator/1.log"
Sep 11 17:42:20.814290 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:20.814252    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v"]
Sep 11 17:42:20.814957 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:20.814940    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b6e9e7e-6399-4b10-9c44-cbe775fe2f56" containerName="maas-api"
Sep 11 17:42:20.815029 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:20.814960    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b6e9e7e-6399-4b10-9c44-cbe775fe2f56" containerName="maas-api"
Sep 11 17:42:20.815098 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:20.815086    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b6e9e7e-6399-4b10-9c44-cbe775fe2f56" containerName="maas-api"
Sep 11 17:42:20.818765 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:20.818744    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v"
Sep 11 17:42:20.821370 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:20.821347    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-40f1fc-serving-cert\""
Sep 11 17:42:20.828283 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:20.828256    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v"]
Sep 11 17:42:20.980187 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:20.980146    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be6adcf5-c27f-4d20-a164-170a1d12f060-openshift-service-ca\") pod \"maas-api-e2e-api-a-40f1fc-694886b94c-9r29v\" (UID: \"be6adcf5-c27f-4d20-a164-170a1d12f060\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v"
Sep 11 17:42:20.980389 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:20.980239    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be6adcf5-c27f-4d20-a164-170a1d12f060-metrics-certs\") pod \"maas-api-e2e-api-a-40f1fc-694886b94c-9r29v\" (UID: \"be6adcf5-c27f-4d20-a164-170a1d12f060\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v"
Sep 11 17:42:20.980389 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:20.980267    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be6adcf5-c27f-4d20-a164-170a1d12f060-maas-api-tls\") pod \"maas-api-e2e-api-a-40f1fc-694886b94c-9r29v\" (UID: \"be6adcf5-c27f-4d20-a164-170a1d12f060\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v"
Sep 11 17:42:20.980389 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:20.980285    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzvkx\" (UniqueName: \"kubernetes.io/projected/be6adcf5-c27f-4d20-a164-170a1d12f060-kube-api-access-pzvkx\") pod \"maas-api-e2e-api-a-40f1fc-694886b94c-9r29v\" (UID: \"be6adcf5-c27f-4d20-a164-170a1d12f060\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v"
Sep 11 17:42:21.081089 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:21.081048    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be6adcf5-c27f-4d20-a164-170a1d12f060-openshift-service-ca\") pod \"maas-api-e2e-api-a-40f1fc-694886b94c-9r29v\" (UID: \"be6adcf5-c27f-4d20-a164-170a1d12f060\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v"
Sep 11 17:42:21.081282 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:21.081163    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be6adcf5-c27f-4d20-a164-170a1d12f060-metrics-certs\") pod \"maas-api-e2e-api-a-40f1fc-694886b94c-9r29v\" (UID: \"be6adcf5-c27f-4d20-a164-170a1d12f060\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v"
Sep 11 17:42:21.081282 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:21.081197    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be6adcf5-c27f-4d20-a164-170a1d12f060-maas-api-tls\") pod \"maas-api-e2e-api-a-40f1fc-694886b94c-9r29v\" (UID: \"be6adcf5-c27f-4d20-a164-170a1d12f060\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v"
Sep 11 17:42:21.081282 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:21.081220    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzvkx\" (UniqueName: \"kubernetes.io/projected/be6adcf5-c27f-4d20-a164-170a1d12f060-kube-api-access-pzvkx\") pod \"maas-api-e2e-api-a-40f1fc-694886b94c-9r29v\" (UID: \"be6adcf5-c27f-4d20-a164-170a1d12f060\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v"
Sep 11 17:42:21.081456 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:42:21.081324    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-serving-cert: secret "maas-api-e2e-api-a-40f1fc-serving-cert" not found
Sep 11 17:42:21.081456 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:42:21.081418    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be6adcf5-c27f-4d20-a164-170a1d12f060-maas-api-tls podName:be6adcf5-c27f-4d20-a164-170a1d12f060 nodeName:}" failed. No retries permitted until 2026-09-11 17:42:21.581395937 +0000 UTC m=+1309.385568335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/be6adcf5-c27f-4d20-a164-170a1d12f060-maas-api-tls") pod "maas-api-e2e-api-a-40f1fc-694886b94c-9r29v" (UID: "be6adcf5-c27f-4d20-a164-170a1d12f060") : secret "maas-api-e2e-api-a-40f1fc-serving-cert" not found
Sep 11 17:42:21.081877 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:21.081854    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be6adcf5-c27f-4d20-a164-170a1d12f060-openshift-service-ca\") pod \"maas-api-e2e-api-a-40f1fc-694886b94c-9r29v\" (UID: \"be6adcf5-c27f-4d20-a164-170a1d12f060\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v"
Sep 11 17:42:21.083734 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:21.083717    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be6adcf5-c27f-4d20-a164-170a1d12f060-metrics-certs\") pod \"maas-api-e2e-api-a-40f1fc-694886b94c-9r29v\" (UID: \"be6adcf5-c27f-4d20-a164-170a1d12f060\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v"
Sep 11 17:42:21.091945 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:21.091919    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzvkx\" (UniqueName: \"kubernetes.io/projected/be6adcf5-c27f-4d20-a164-170a1d12f060-kube-api-access-pzvkx\") pod \"maas-api-e2e-api-a-40f1fc-694886b94c-9r29v\" (UID: \"be6adcf5-c27f-4d20-a164-170a1d12f060\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v"
Sep 11 17:42:21.586021 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:21.585987    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be6adcf5-c27f-4d20-a164-170a1d12f060-maas-api-tls\") pod \"maas-api-e2e-api-a-40f1fc-694886b94c-9r29v\" (UID: \"be6adcf5-c27f-4d20-a164-170a1d12f060\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v"
Sep 11 17:42:21.588345 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:21.588323    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be6adcf5-c27f-4d20-a164-170a1d12f060-maas-api-tls\") pod \"maas-api-e2e-api-a-40f1fc-694886b94c-9r29v\" (UID: \"be6adcf5-c27f-4d20-a164-170a1d12f060\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v"
Sep 11 17:42:21.734176 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:21.734134    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v"
Sep 11 17:42:22.008218 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:22.008192    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v"]
Sep 11 17:42:22.010203 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:42:22.010176    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe6adcf5_c27f_4d20_a164_170a1d12f060.slice/crio-3b3ed76fae6f5bb132bc41d26f39da859c373388e1f94e78cef219f66d1229db WatchSource:0}: Error finding container 3b3ed76fae6f5bb132bc41d26f39da859c373388e1f94e78cef219f66d1229db: Status 404 returned error can't find the container with id 3b3ed76fae6f5bb132bc41d26f39da859c373388e1f94e78cef219f66d1229db
Sep 11 17:42:22.011734 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:22.011712    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:42:22.240066 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:22.240029    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v" event={"ID":"be6adcf5-c27f-4d20-a164-170a1d12f060","Type":"ContainerStarted","Data":"3b3ed76fae6f5bb132bc41d26f39da859c373388e1f94e78cef219f66d1229db"}
Sep 11 17:42:23.247074 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:23.247027    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v" event={"ID":"be6adcf5-c27f-4d20-a164-170a1d12f060","Type":"ContainerStarted","Data":"7fe229883c805da7b9647be5abf16c1536e10116f8d2d8188dd14360c46a42a2"}
Sep 11 17:42:23.247457 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:23.247123    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v"
Sep 11 17:42:23.267475 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:23.267428    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v" podStartSLOduration=2.825507559 podStartE2EDuration="3.267412272s" totalImagesPullingTime="441.904713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:42:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:42:23.265317298 +0000 UTC m=+1311.069489727" watchObservedRunningTime="2026-09-11 17:42:23.267412272 +0000 UTC m=+1311.071584722"
Sep 11 17:42:29.259261 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:29.259231    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v"
Sep 11 17:42:57.641573 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:57.641480    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c"]
Sep 11 17:42:57.645632 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:57.645608    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c"
Sep 11 17:42:57.650036 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:57.650016    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-86db1c\""
Sep 11 17:42:57.669342 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:57.669313    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c"]
Sep 11 17:42:57.719625 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:57.719585    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7a9943b-e86a-43f7-9a86-042795d21fa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c\" (UID: \"f7a9943b-e86a-43f7-9a86-042795d21fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c"
Sep 11 17:42:57.719625 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:57.719631    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj6vj\" (UniqueName: \"kubernetes.io/projected/f7a9943b-e86a-43f7-9a86-042795d21fa9-kube-api-access-zj6vj\") pod \"payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c\" (UID: \"f7a9943b-e86a-43f7-9a86-042795d21fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c"
Sep 11 17:42:57.820753 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:57.820718    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7a9943b-e86a-43f7-9a86-042795d21fa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c\" (UID: \"f7a9943b-e86a-43f7-9a86-042795d21fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c"
Sep 11 17:42:57.820965 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:57.820764    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj6vj\" (UniqueName: \"kubernetes.io/projected/f7a9943b-e86a-43f7-9a86-042795d21fa9-kube-api-access-zj6vj\") pod \"payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c\" (UID: \"f7a9943b-e86a-43f7-9a86-042795d21fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c"
Sep 11 17:42:57.821590 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:57.821568    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7a9943b-e86a-43f7-9a86-042795d21fa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c\" (UID: \"f7a9943b-e86a-43f7-9a86-042795d21fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c"
Sep 11 17:42:57.832166 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:57.832135    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj6vj\" (UniqueName: \"kubernetes.io/projected/f7a9943b-e86a-43f7-9a86-042795d21fa9-kube-api-access-zj6vj\") pod \"payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c\" (UID: \"f7a9943b-e86a-43f7-9a86-042795d21fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c"
Sep 11 17:42:57.956262 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:57.956162    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c"
Sep 11 17:42:58.236937 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:58.236909    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c"]
Sep 11 17:42:58.239111 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:42:58.239080    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a9943b_e86a_43f7_9a86_042795d21fa9.slice/crio-3a95574a08165902057a1cf4adf3e726b443ae0545d870e2683ec8228d395d2d WatchSource:0}: Error finding container 3a95574a08165902057a1cf4adf3e726b443ae0545d870e2683ec8228d395d2d: Status 404 returned error can't find the container with id 3a95574a08165902057a1cf4adf3e726b443ae0545d870e2683ec8228d395d2d
Sep 11 17:42:58.445206 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:58.445173    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c" event={"ID":"f7a9943b-e86a-43f7-9a86-042795d21fa9","Type":"ContainerStarted","Data":"4b1fe7930bc767c5c63a733d9def2161de5d5c298287b9c4fe6d57396fd98fc0"}
Sep 11 17:42:58.445407 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:58.445234    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c" event={"ID":"f7a9943b-e86a-43f7-9a86-042795d21fa9","Type":"ContainerStarted","Data":"3a95574a08165902057a1cf4adf3e726b443ae0545d870e2683ec8228d395d2d"}
Sep 11 17:42:58.445407 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:58.445293    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c"
Sep 11 17:42:58.468579 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:42:58.468525    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c" podStartSLOduration=1.468508538 podStartE2EDuration="1.468508538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:42:58.466567 +0000 UTC m=+1346.270739418" watchObservedRunningTime="2026-09-11 17:42:58.468508538 +0000 UTC m=+1346.272680957"
Sep 11 17:43:09.453542 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:43:09.453510    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c"
Sep 11 17:43:41.408234 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:43:41.408196    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v"]
Sep 11 17:43:41.408758 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:43:41.408425    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v" podUID="be6adcf5-c27f-4d20-a164-170a1d12f060" containerName="maas-api" containerID="cri-o://7fe229883c805da7b9647be5abf16c1536e10116f8d2d8188dd14360c46a42a2" gracePeriod=30
Sep 11 17:43:41.668759 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:43:41.668729    2560 generic.go:356] "Generic (PLEG): container finished" podID="be6adcf5-c27f-4d20-a164-170a1d12f060" containerID="7fe229883c805da7b9647be5abf16c1536e10116f8d2d8188dd14360c46a42a2" exitCode=0
Sep 11 17:43:41.668923 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:43:41.668786    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v" event={"ID":"be6adcf5-c27f-4d20-a164-170a1d12f060","Type":"ContainerDied","Data":"7fe229883c805da7b9647be5abf16c1536e10116f8d2d8188dd14360c46a42a2"}
Sep 11 17:43:41.668923 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:43:41.668836    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v" event={"ID":"be6adcf5-c27f-4d20-a164-170a1d12f060","Type":"ContainerDied","Data":"3b3ed76fae6f5bb132bc41d26f39da859c373388e1f94e78cef219f66d1229db"}
Sep 11 17:43:41.668923 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:43:41.668863    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b3ed76fae6f5bb132bc41d26f39da859c373388e1f94e78cef219f66d1229db"
Sep 11 17:43:41.682079 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:43:41.682055    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v"
Sep 11 17:43:41.845646 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:43:41.845598    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be6adcf5-c27f-4d20-a164-170a1d12f060-metrics-certs\") pod \"be6adcf5-c27f-4d20-a164-170a1d12f060\" (UID: \"be6adcf5-c27f-4d20-a164-170a1d12f060\") "
Sep 11 17:43:41.845845 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:43:41.845656    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be6adcf5-c27f-4d20-a164-170a1d12f060-openshift-service-ca\") pod \"be6adcf5-c27f-4d20-a164-170a1d12f060\" (UID: \"be6adcf5-c27f-4d20-a164-170a1d12f060\") "
Sep 11 17:43:41.845845 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:43:41.845683    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzvkx\" (UniqueName: \"kubernetes.io/projected/be6adcf5-c27f-4d20-a164-170a1d12f060-kube-api-access-pzvkx\") pod \"be6adcf5-c27f-4d20-a164-170a1d12f060\" (UID: \"be6adcf5-c27f-4d20-a164-170a1d12f060\") "
Sep 11 17:43:41.845845 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:43:41.845732    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be6adcf5-c27f-4d20-a164-170a1d12f060-maas-api-tls\") pod \"be6adcf5-c27f-4d20-a164-170a1d12f060\" (UID: \"be6adcf5-c27f-4d20-a164-170a1d12f060\") "
Sep 11 17:43:41.846171 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:43:41.846127    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be6adcf5-c27f-4d20-a164-170a1d12f060-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "be6adcf5-c27f-4d20-a164-170a1d12f060" (UID: "be6adcf5-c27f-4d20-a164-170a1d12f060"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:43:41.847925 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:43:41.847865    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be6adcf5-c27f-4d20-a164-170a1d12f060-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "be6adcf5-c27f-4d20-a164-170a1d12f060" (UID: "be6adcf5-c27f-4d20-a164-170a1d12f060"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:43:41.848383 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:43:41.848358    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be6adcf5-c27f-4d20-a164-170a1d12f060-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "be6adcf5-c27f-4d20-a164-170a1d12f060" (UID: "be6adcf5-c27f-4d20-a164-170a1d12f060"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:43:41.848383 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:43:41.848375    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6adcf5-c27f-4d20-a164-170a1d12f060-kube-api-access-pzvkx" (OuterVolumeSpecName: "kube-api-access-pzvkx") pod "be6adcf5-c27f-4d20-a164-170a1d12f060" (UID: "be6adcf5-c27f-4d20-a164-170a1d12f060"). InnerVolumeSpecName "kube-api-access-pzvkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:43:41.947151 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:43:41.947052    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be6adcf5-c27f-4d20-a164-170a1d12f060-metrics-certs\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:43:41.947151 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:43:41.947087    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be6adcf5-c27f-4d20-a164-170a1d12f060-openshift-service-ca\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:43:41.947151 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:43:41.947098    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzvkx\" (UniqueName: \"kubernetes.io/projected/be6adcf5-c27f-4d20-a164-170a1d12f060-kube-api-access-pzvkx\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:43:41.947151 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:43:41.947107    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be6adcf5-c27f-4d20-a164-170a1d12f060-maas-api-tls\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:43:42.673793 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:43:42.673760    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v"
Sep 11 17:43:42.704465 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:43:42.704430    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v"]
Sep 11 17:43:42.718521 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:43:42.718491    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-40f1fc-694886b94c-9r29v"]
Sep 11 17:43:44.715982 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:43:44.715942    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be6adcf5-c27f-4d20-a164-170a1d12f060" path="/var/lib/kubelet/pods/be6adcf5-c27f-4d20-a164-170a1d12f060/volumes"
Sep 11 17:44:06.427338 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:06.427304    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c"]
Sep 11 17:44:06.427801 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:06.427585    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c" podUID="f7a9943b-e86a-43f7-9a86-042795d21fa9" containerName="payload-pre-processing" containerID="cri-o://4b1fe7930bc767c5c63a733d9def2161de5d5c298287b9c4fe6d57396fd98fc0" gracePeriod=30
Sep 11 17:44:06.688950 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:06.688864    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c"
Sep 11 17:44:06.789683 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:06.789645    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zj6vj\" (UniqueName: \"kubernetes.io/projected/f7a9943b-e86a-43f7-9a86-042795d21fa9-kube-api-access-zj6vj\") pod \"f7a9943b-e86a-43f7-9a86-042795d21fa9\" (UID: \"f7a9943b-e86a-43f7-9a86-042795d21fa9\") "
Sep 11 17:44:06.789832 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:06.789775    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7a9943b-e86a-43f7-9a86-042795d21fa9-plugins-config-volume\") pod \"f7a9943b-e86a-43f7-9a86-042795d21fa9\" (UID: \"f7a9943b-e86a-43f7-9a86-042795d21fa9\") "
Sep 11 17:44:06.790273 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:06.790240    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7a9943b-e86a-43f7-9a86-042795d21fa9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f7a9943b-e86a-43f7-9a86-042795d21fa9" (UID: "f7a9943b-e86a-43f7-9a86-042795d21fa9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:44:06.791950 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:06.791919    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a9943b-e86a-43f7-9a86-042795d21fa9-kube-api-access-zj6vj" (OuterVolumeSpecName: "kube-api-access-zj6vj") pod "f7a9943b-e86a-43f7-9a86-042795d21fa9" (UID: "f7a9943b-e86a-43f7-9a86-042795d21fa9"). InnerVolumeSpecName "kube-api-access-zj6vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:44:06.803957 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:06.803930    2560 generic.go:356] "Generic (PLEG): container finished" podID="f7a9943b-e86a-43f7-9a86-042795d21fa9" containerID="4b1fe7930bc767c5c63a733d9def2161de5d5c298287b9c4fe6d57396fd98fc0" exitCode=0
Sep 11 17:44:06.804074 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:06.804042    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c"
Sep 11 17:44:06.804122 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:06.804067    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c" event={"ID":"f7a9943b-e86a-43f7-9a86-042795d21fa9","Type":"ContainerDied","Data":"4b1fe7930bc767c5c63a733d9def2161de5d5c298287b9c4fe6d57396fd98fc0"}
Sep 11 17:44:06.804164 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:06.804131    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c" event={"ID":"f7a9943b-e86a-43f7-9a86-042795d21fa9","Type":"ContainerDied","Data":"3a95574a08165902057a1cf4adf3e726b443ae0545d870e2683ec8228d395d2d"}
Sep 11 17:44:06.804207 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:06.804160    2560 scope.go:117] "RemoveContainer" containerID="4b1fe7930bc767c5c63a733d9def2161de5d5c298287b9c4fe6d57396fd98fc0"
Sep 11 17:44:06.816218 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:06.816195    2560 scope.go:117] "RemoveContainer" containerID="4b1fe7930bc767c5c63a733d9def2161de5d5c298287b9c4fe6d57396fd98fc0"
Sep 11 17:44:06.816533 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:44:06.816514    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b1fe7930bc767c5c63a733d9def2161de5d5c298287b9c4fe6d57396fd98fc0\": container with ID starting with 4b1fe7930bc767c5c63a733d9def2161de5d5c298287b9c4fe6d57396fd98fc0 not found: ID does not exist" containerID="4b1fe7930bc767c5c63a733d9def2161de5d5c298287b9c4fe6d57396fd98fc0"
Sep 11 17:44:06.816591 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:06.816543    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b1fe7930bc767c5c63a733d9def2161de5d5c298287b9c4fe6d57396fd98fc0"} err="failed to get container status \"4b1fe7930bc767c5c63a733d9def2161de5d5c298287b9c4fe6d57396fd98fc0\": rpc error: code = NotFound desc = could not find container \"4b1fe7930bc767c5c63a733d9def2161de5d5c298287b9c4fe6d57396fd98fc0\": container with ID starting with 4b1fe7930bc767c5c63a733d9def2161de5d5c298287b9c4fe6d57396fd98fc0 not found: ID does not exist"
Sep 11 17:44:06.833946 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:06.833909    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c"]
Sep 11 17:44:06.837424 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:06.837399    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-86db1c-75569d74c8-2vz2c"]
Sep 11 17:44:06.891405 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:06.891373    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7a9943b-e86a-43f7-9a86-042795d21fa9-plugins-config-volume\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:44:06.891405 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:06.891401    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj6vj\" (UniqueName: \"kubernetes.io/projected/f7a9943b-e86a-43f7-9a86-042795d21fa9-kube-api-access-zj6vj\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:44:08.715681 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:08.715646    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7a9943b-e86a-43f7-9a86-042795d21fa9" path="/var/lib/kubelet/pods/f7a9943b-e86a-43f7-9a86-042795d21fa9/volumes"
Sep 11 17:44:25.060866 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.060831    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm"]
Sep 11 17:44:25.061428 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.061096    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm" podUID="94e3e2a7-dcb6-4d61-b62c-37da2674c99a" containerName="maas-api" containerID="cri-o://5796f3d7547ebc5c58b45633b5641ccdfe791861f2111a7af519c1d47a8255e3" gracePeriod=30
Sep 11 17:44:25.184634 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.184603    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn"]
Sep 11 17:44:25.184843 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.184815    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn" podUID="a3454246-63c2-4f32-b368-f12af2cab60d" containerName="payload-processing" containerID="cri-o://a0f3bf54cec9281088828ef4bc4673bb29983e67abf1ee7f9923cdbbfaeaac14" gracePeriod=30
Sep 11 17:44:25.210313 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.210281    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw"]
Sep 11 17:44:25.210506 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.210483    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw" podUID="4c80eed9-b8a3-409a-b85f-d1c7fea05e94" containerName="payload-pre-processing" containerID="cri-o://10b4a8e4f92f81312ad7a5a7238062393be686463cabd00e6c1926c9249c50a9" gracePeriod=30
Sep 11 17:44:25.341290 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.341265    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm"
Sep 11 17:44:25.476423 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.476400    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn"
Sep 11 17:44:25.476861 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.476806    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94e3e2a7-dcb6-4d61-b62c-37da2674c99a-metrics-certs\") pod \"94e3e2a7-dcb6-4d61-b62c-37da2674c99a\" (UID: \"94e3e2a7-dcb6-4d61-b62c-37da2674c99a\") "
Sep 11 17:44:25.476986 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.476875    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94e3e2a7-dcb6-4d61-b62c-37da2674c99a-openshift-service-ca\") pod \"94e3e2a7-dcb6-4d61-b62c-37da2674c99a\" (UID: \"94e3e2a7-dcb6-4d61-b62c-37da2674c99a\") "
Sep 11 17:44:25.476986 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.476953    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94e3e2a7-dcb6-4d61-b62c-37da2674c99a-maas-api-tls\") pod \"94e3e2a7-dcb6-4d61-b62c-37da2674c99a\" (UID: \"94e3e2a7-dcb6-4d61-b62c-37da2674c99a\") "
Sep 11 17:44:25.477104 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.477046    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsmlm\" (UniqueName: \"kubernetes.io/projected/94e3e2a7-dcb6-4d61-b62c-37da2674c99a-kube-api-access-nsmlm\") pod \"94e3e2a7-dcb6-4d61-b62c-37da2674c99a\" (UID: \"94e3e2a7-dcb6-4d61-b62c-37da2674c99a\") "
Sep 11 17:44:25.477277 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.477246    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94e3e2a7-dcb6-4d61-b62c-37da2674c99a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "94e3e2a7-dcb6-4d61-b62c-37da2674c99a" (UID: "94e3e2a7-dcb6-4d61-b62c-37da2674c99a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:44:25.477403 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.477371    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94e3e2a7-dcb6-4d61-b62c-37da2674c99a-openshift-service-ca\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:44:25.478974 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.478949    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94e3e2a7-dcb6-4d61-b62c-37da2674c99a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "94e3e2a7-dcb6-4d61-b62c-37da2674c99a" (UID: "94e3e2a7-dcb6-4d61-b62c-37da2674c99a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:44:25.479065 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.479001    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94e3e2a7-dcb6-4d61-b62c-37da2674c99a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "94e3e2a7-dcb6-4d61-b62c-37da2674c99a" (UID: "94e3e2a7-dcb6-4d61-b62c-37da2674c99a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:44:25.479176 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.479160    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e3e2a7-dcb6-4d61-b62c-37da2674c99a-kube-api-access-nsmlm" (OuterVolumeSpecName: "kube-api-access-nsmlm") pod "94e3e2a7-dcb6-4d61-b62c-37da2674c99a" (UID: "94e3e2a7-dcb6-4d61-b62c-37da2674c99a"). InnerVolumeSpecName "kube-api-access-nsmlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:44:25.487024 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.487001    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw"
Sep 11 17:44:25.577864 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.577826    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-666xq\" (UniqueName: \"kubernetes.io/projected/a3454246-63c2-4f32-b368-f12af2cab60d-kube-api-access-666xq\") pod \"a3454246-63c2-4f32-b368-f12af2cab60d\" (UID: \"a3454246-63c2-4f32-b368-f12af2cab60d\") "
Sep 11 17:44:25.578098 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.577925    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3454246-63c2-4f32-b368-f12af2cab60d-plugins-config-volume\") pod \"a3454246-63c2-4f32-b368-f12af2cab60d\" (UID: \"a3454246-63c2-4f32-b368-f12af2cab60d\") "
Sep 11 17:44:25.580319 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.578522    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3454246-63c2-4f32-b368-f12af2cab60d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a3454246-63c2-4f32-b368-f12af2cab60d" (UID: "a3454246-63c2-4f32-b368-f12af2cab60d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:44:25.580319 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.578752    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3454246-63c2-4f32-b368-f12af2cab60d-plugins-config-volume\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:44:25.580319 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.578779    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94e3e2a7-dcb6-4d61-b62c-37da2674c99a-metrics-certs\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:44:25.580319 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.578792    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94e3e2a7-dcb6-4d61-b62c-37da2674c99a-maas-api-tls\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:44:25.580319 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.578806    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsmlm\" (UniqueName: \"kubernetes.io/projected/94e3e2a7-dcb6-4d61-b62c-37da2674c99a-kube-api-access-nsmlm\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:44:25.580779 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.580746    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3454246-63c2-4f32-b368-f12af2cab60d-kube-api-access-666xq" (OuterVolumeSpecName: "kube-api-access-666xq") pod "a3454246-63c2-4f32-b368-f12af2cab60d" (UID: "a3454246-63c2-4f32-b368-f12af2cab60d"). InnerVolumeSpecName "kube-api-access-666xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:44:25.680130 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.680089    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c80eed9-b8a3-409a-b85f-d1c7fea05e94-plugins-config-volume\") pod \"4c80eed9-b8a3-409a-b85f-d1c7fea05e94\" (UID: \"4c80eed9-b8a3-409a-b85f-d1c7fea05e94\") "
Sep 11 17:44:25.680130 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.680128    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wzwg\" (UniqueName: \"kubernetes.io/projected/4c80eed9-b8a3-409a-b85f-d1c7fea05e94-kube-api-access-4wzwg\") pod \"4c80eed9-b8a3-409a-b85f-d1c7fea05e94\" (UID: \"4c80eed9-b8a3-409a-b85f-d1c7fea05e94\") "
Sep 11 17:44:25.680383 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.680304    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-666xq\" (UniqueName: \"kubernetes.io/projected/a3454246-63c2-4f32-b368-f12af2cab60d-kube-api-access-666xq\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:44:25.680489 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.680464    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c80eed9-b8a3-409a-b85f-d1c7fea05e94-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4c80eed9-b8a3-409a-b85f-d1c7fea05e94" (UID: "4c80eed9-b8a3-409a-b85f-d1c7fea05e94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:44:25.682296 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.682263    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c80eed9-b8a3-409a-b85f-d1c7fea05e94-kube-api-access-4wzwg" (OuterVolumeSpecName: "kube-api-access-4wzwg") pod "4c80eed9-b8a3-409a-b85f-d1c7fea05e94" (UID: "4c80eed9-b8a3-409a-b85f-d1c7fea05e94"). InnerVolumeSpecName "kube-api-access-4wzwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:44:25.781050 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.781015    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c80eed9-b8a3-409a-b85f-d1c7fea05e94-plugins-config-volume\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:44:25.781050 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.781044    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wzwg\" (UniqueName: \"kubernetes.io/projected/4c80eed9-b8a3-409a-b85f-d1c7fea05e94-kube-api-access-4wzwg\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:44:25.908835 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.908750    2560 generic.go:356] "Generic (PLEG): container finished" podID="94e3e2a7-dcb6-4d61-b62c-37da2674c99a" containerID="5796f3d7547ebc5c58b45633b5641ccdfe791861f2111a7af519c1d47a8255e3" exitCode=0
Sep 11 17:44:25.908835 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.908796    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm"
Sep 11 17:44:25.909053 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.908840    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm" event={"ID":"94e3e2a7-dcb6-4d61-b62c-37da2674c99a","Type":"ContainerDied","Data":"5796f3d7547ebc5c58b45633b5641ccdfe791861f2111a7af519c1d47a8255e3"}
Sep 11 17:44:25.909053 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.908888    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm" event={"ID":"94e3e2a7-dcb6-4d61-b62c-37da2674c99a","Type":"ContainerDied","Data":"a277aff9fa9c7ffa38ad5afa50cebdfa8838c5f874f180e9c93a00f0cbaa0b68"}
Sep 11 17:44:25.909053 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.908923    2560 scope.go:117] "RemoveContainer" containerID="5796f3d7547ebc5c58b45633b5641ccdfe791861f2111a7af519c1d47a8255e3"
Sep 11 17:44:25.910160 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.910142    2560 generic.go:356] "Generic (PLEG): container finished" podID="a3454246-63c2-4f32-b368-f12af2cab60d" containerID="a0f3bf54cec9281088828ef4bc4673bb29983e67abf1ee7f9923cdbbfaeaac14" exitCode=0
Sep 11 17:44:25.910294 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.910189    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn"
Sep 11 17:44:25.910294 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.910232    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn" event={"ID":"a3454246-63c2-4f32-b368-f12af2cab60d","Type":"ContainerDied","Data":"a0f3bf54cec9281088828ef4bc4673bb29983e67abf1ee7f9923cdbbfaeaac14"}
Sep 11 17:44:25.910294 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.910276    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn" event={"ID":"a3454246-63c2-4f32-b368-f12af2cab60d","Type":"ContainerDied","Data":"8ae37d72ee5b1d193d173a8513fe57abfa957d686e69188d6953d5f1a25b160a"}
Sep 11 17:44:25.911706 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.911677    2560 generic.go:356] "Generic (PLEG): container finished" podID="4c80eed9-b8a3-409a-b85f-d1c7fea05e94" containerID="10b4a8e4f92f81312ad7a5a7238062393be686463cabd00e6c1926c9249c50a9" exitCode=0
Sep 11 17:44:25.911808 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.911717    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw"
Sep 11 17:44:25.911856 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.911713    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw" event={"ID":"4c80eed9-b8a3-409a-b85f-d1c7fea05e94","Type":"ContainerDied","Data":"10b4a8e4f92f81312ad7a5a7238062393be686463cabd00e6c1926c9249c50a9"}
Sep 11 17:44:25.911921 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.911848    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw" event={"ID":"4c80eed9-b8a3-409a-b85f-d1c7fea05e94","Type":"ContainerDied","Data":"ddbb5d92a72c7e0148dff21f1d1a3fe924aebd8839f13d303e3ccfb92b170709"}
Sep 11 17:44:25.926859 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.926711    2560 scope.go:117] "RemoveContainer" containerID="5796f3d7547ebc5c58b45633b5641ccdfe791861f2111a7af519c1d47a8255e3"
Sep 11 17:44:25.927470 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:44:25.927383    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5796f3d7547ebc5c58b45633b5641ccdfe791861f2111a7af519c1d47a8255e3\": container with ID starting with 5796f3d7547ebc5c58b45633b5641ccdfe791861f2111a7af519c1d47a8255e3 not found: ID does not exist" containerID="5796f3d7547ebc5c58b45633b5641ccdfe791861f2111a7af519c1d47a8255e3"
Sep 11 17:44:25.927586 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.927463    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5796f3d7547ebc5c58b45633b5641ccdfe791861f2111a7af519c1d47a8255e3"} err="failed to get container status \"5796f3d7547ebc5c58b45633b5641ccdfe791861f2111a7af519c1d47a8255e3\": rpc error: code = NotFound desc = could not find container \"5796f3d7547ebc5c58b45633b5641ccdfe791861f2111a7af519c1d47a8255e3\": container with ID starting with 5796f3d7547ebc5c58b45633b5641ccdfe791861f2111a7af519c1d47a8255e3 not found: ID does not exist"
Sep 11 17:44:25.927586 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.927488    2560 scope.go:117] "RemoveContainer" containerID="a0f3bf54cec9281088828ef4bc4673bb29983e67abf1ee7f9923cdbbfaeaac14"
Sep 11 17:44:25.937628 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.937608    2560 scope.go:117] "RemoveContainer" containerID="a0f3bf54cec9281088828ef4bc4673bb29983e67abf1ee7f9923cdbbfaeaac14"
Sep 11 17:44:25.937953 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:44:25.937933    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0f3bf54cec9281088828ef4bc4673bb29983e67abf1ee7f9923cdbbfaeaac14\": container with ID starting with a0f3bf54cec9281088828ef4bc4673bb29983e67abf1ee7f9923cdbbfaeaac14 not found: ID does not exist" containerID="a0f3bf54cec9281088828ef4bc4673bb29983e67abf1ee7f9923cdbbfaeaac14"
Sep 11 17:44:25.938017 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.937964    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0f3bf54cec9281088828ef4bc4673bb29983e67abf1ee7f9923cdbbfaeaac14"} err="failed to get container status \"a0f3bf54cec9281088828ef4bc4673bb29983e67abf1ee7f9923cdbbfaeaac14\": rpc error: code = NotFound desc = could not find container \"a0f3bf54cec9281088828ef4bc4673bb29983e67abf1ee7f9923cdbbfaeaac14\": container with ID starting with a0f3bf54cec9281088828ef4bc4673bb29983e67abf1ee7f9923cdbbfaeaac14 not found: ID does not exist"
Sep 11 17:44:25.938017 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.937983    2560 scope.go:117] "RemoveContainer" containerID="10b4a8e4f92f81312ad7a5a7238062393be686463cabd00e6c1926c9249c50a9"
Sep 11 17:44:25.947791 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.947759    2560 scope.go:117] "RemoveContainer" containerID="10b4a8e4f92f81312ad7a5a7238062393be686463cabd00e6c1926c9249c50a9"
Sep 11 17:44:25.948170 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:44:25.948142    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10b4a8e4f92f81312ad7a5a7238062393be686463cabd00e6c1926c9249c50a9\": container with ID starting with 10b4a8e4f92f81312ad7a5a7238062393be686463cabd00e6c1926c9249c50a9 not found: ID does not exist" containerID="10b4a8e4f92f81312ad7a5a7238062393be686463cabd00e6c1926c9249c50a9"
Sep 11 17:44:25.948240 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.948185    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10b4a8e4f92f81312ad7a5a7238062393be686463cabd00e6c1926c9249c50a9"} err="failed to get container status \"10b4a8e4f92f81312ad7a5a7238062393be686463cabd00e6c1926c9249c50a9\": rpc error: code = NotFound desc = could not find container \"10b4a8e4f92f81312ad7a5a7238062393be686463cabd00e6c1926c9249c50a9\": container with ID starting with 10b4a8e4f92f81312ad7a5a7238062393be686463cabd00e6c1926c9249c50a9 not found: ID does not exist"
Sep 11 17:44:25.964095 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.964060    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn"]
Sep 11 17:44:25.979274 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:25.979240    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-470521-7bdd6575c5-stfwn"]
Sep 11 17:44:26.016321 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:26.016287    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw"]
Sep 11 17:44:26.029175 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:26.029148    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-470521-674944978b-8nwkw"]
Sep 11 17:44:26.061647 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:26.061606    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm"]
Sep 11 17:44:26.081407 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:26.081367    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-470521-54bf6f4f5c-k2fjm"]
Sep 11 17:44:26.715992 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:26.715963    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c80eed9-b8a3-409a-b85f-d1c7fea05e94" path="/var/lib/kubelet/pods/4c80eed9-b8a3-409a-b85f-d1c7fea05e94/volumes"
Sep 11 17:44:26.716370 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:26.716357    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94e3e2a7-dcb6-4d61-b62c-37da2674c99a" path="/var/lib/kubelet/pods/94e3e2a7-dcb6-4d61-b62c-37da2674c99a/volumes"
Sep 11 17:44:26.716675 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:26.716664    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3454246-63c2-4f32-b368-f12af2cab60d" path="/var/lib/kubelet/pods/a3454246-63c2-4f32-b368-f12af2cab60d/volumes"
Sep 11 17:44:45.415988 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:45.415952    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv"]
Sep 11 17:44:45.416425 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:45.416163    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv" podUID="9aeb1857-85d7-4fd0-b267-68d0e60738dc" containerName="payload-pre-processing" containerID="cri-o://2f3f32526b5ec68851be1b2d85510d18088c47ac5b6f996e9308da12ad75f42d" gracePeriod=30
Sep 11 17:44:45.663832 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:45.663808    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv"
Sep 11 17:44:45.767623 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:45.767532    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aeb1857-85d7-4fd0-b267-68d0e60738dc-plugins-config-volume\") pod \"9aeb1857-85d7-4fd0-b267-68d0e60738dc\" (UID: \"9aeb1857-85d7-4fd0-b267-68d0e60738dc\") "
Sep 11 17:44:45.767800 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:45.767675    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w7gj\" (UniqueName: \"kubernetes.io/projected/9aeb1857-85d7-4fd0-b267-68d0e60738dc-kube-api-access-4w7gj\") pod \"9aeb1857-85d7-4fd0-b267-68d0e60738dc\" (UID: \"9aeb1857-85d7-4fd0-b267-68d0e60738dc\") "
Sep 11 17:44:45.768017 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:45.767992    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aeb1857-85d7-4fd0-b267-68d0e60738dc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9aeb1857-85d7-4fd0-b267-68d0e60738dc" (UID: "9aeb1857-85d7-4fd0-b267-68d0e60738dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:44:45.769833 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:45.769810    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aeb1857-85d7-4fd0-b267-68d0e60738dc-kube-api-access-4w7gj" (OuterVolumeSpecName: "kube-api-access-4w7gj") pod "9aeb1857-85d7-4fd0-b267-68d0e60738dc" (UID: "9aeb1857-85d7-4fd0-b267-68d0e60738dc"). InnerVolumeSpecName "kube-api-access-4w7gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:44:45.869200 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:45.869164    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w7gj\" (UniqueName: \"kubernetes.io/projected/9aeb1857-85d7-4fd0-b267-68d0e60738dc-kube-api-access-4w7gj\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:44:45.869200 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:45.869193    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aeb1857-85d7-4fd0-b267-68d0e60738dc-plugins-config-volume\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:44:46.026115 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:46.026027    2560 generic.go:356] "Generic (PLEG): container finished" podID="9aeb1857-85d7-4fd0-b267-68d0e60738dc" containerID="2f3f32526b5ec68851be1b2d85510d18088c47ac5b6f996e9308da12ad75f42d" exitCode=0
Sep 11 17:44:46.026115 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:46.026073    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv"
Sep 11 17:44:46.026115 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:46.026098    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv" event={"ID":"9aeb1857-85d7-4fd0-b267-68d0e60738dc","Type":"ContainerDied","Data":"2f3f32526b5ec68851be1b2d85510d18088c47ac5b6f996e9308da12ad75f42d"}
Sep 11 17:44:46.026364 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:46.026140    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv" event={"ID":"9aeb1857-85d7-4fd0-b267-68d0e60738dc","Type":"ContainerDied","Data":"c28e8555720cdd63259bc585af753fe4f5ac6cd9236fed4ca6be81c66d5ee2ef"}
Sep 11 17:44:46.026364 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:46.026161    2560 scope.go:117] "RemoveContainer" containerID="2f3f32526b5ec68851be1b2d85510d18088c47ac5b6f996e9308da12ad75f42d"
Sep 11 17:44:46.035702 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:46.035684    2560 scope.go:117] "RemoveContainer" containerID="2f3f32526b5ec68851be1b2d85510d18088c47ac5b6f996e9308da12ad75f42d"
Sep 11 17:44:46.035969 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:44:46.035946    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f3f32526b5ec68851be1b2d85510d18088c47ac5b6f996e9308da12ad75f42d\": container with ID starting with 2f3f32526b5ec68851be1b2d85510d18088c47ac5b6f996e9308da12ad75f42d not found: ID does not exist" containerID="2f3f32526b5ec68851be1b2d85510d18088c47ac5b6f996e9308da12ad75f42d"
Sep 11 17:44:46.036036 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:46.035978    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f3f32526b5ec68851be1b2d85510d18088c47ac5b6f996e9308da12ad75f42d"} err="failed to get container status \"2f3f32526b5ec68851be1b2d85510d18088c47ac5b6f996e9308da12ad75f42d\": rpc error: code = NotFound desc = could not find container \"2f3f32526b5ec68851be1b2d85510d18088c47ac5b6f996e9308da12ad75f42d\": container with ID starting with 2f3f32526b5ec68851be1b2d85510d18088c47ac5b6f996e9308da12ad75f42d not found: ID does not exist"
Sep 11 17:44:46.084561 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:46.084526    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv"]
Sep 11 17:44:46.086392 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:46.086367    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b6b979-5bbf8b64f4-sc4jv"]
Sep 11 17:44:46.714976 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:44:46.714942    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9aeb1857-85d7-4fd0-b267-68d0e60738dc" path="/var/lib/kubelet/pods/9aeb1857-85d7-4fd0-b267-68d0e60738dc/volumes"
Sep 11 17:45:32.704625 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:32.704589    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6fcjs_477cac01-ab9e-4441-b07d-b7e53401c616/console-operator/1.log"
Sep 11 17:45:32.710130 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:32.710104    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6fcjs_477cac01-ab9e-4441-b07d-b7e53401c616/console-operator/1.log"
Sep 11 17:45:50.017391 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.017349    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts"]
Sep 11 17:45:50.017875 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.017786    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c80eed9-b8a3-409a-b85f-d1c7fea05e94" containerName="payload-pre-processing"
Sep 11 17:45:50.017875 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.017800    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c80eed9-b8a3-409a-b85f-d1c7fea05e94" containerName="payload-pre-processing"
Sep 11 17:45:50.017875 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.017815    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3454246-63c2-4f32-b368-f12af2cab60d" containerName="payload-processing"
Sep 11 17:45:50.017875 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.017820    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3454246-63c2-4f32-b368-f12af2cab60d" containerName="payload-processing"
Sep 11 17:45:50.017875 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.017828    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be6adcf5-c27f-4d20-a164-170a1d12f060" containerName="maas-api"
Sep 11 17:45:50.017875 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.017834    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="be6adcf5-c27f-4d20-a164-170a1d12f060" containerName="maas-api"
Sep 11 17:45:50.017875 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.017845    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7a9943b-e86a-43f7-9a86-042795d21fa9" containerName="payload-pre-processing"
Sep 11 17:45:50.017875 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.017851    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7a9943b-e86a-43f7-9a86-042795d21fa9" containerName="payload-pre-processing"
Sep 11 17:45:50.017875 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.017860    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aeb1857-85d7-4fd0-b267-68d0e60738dc" containerName="payload-pre-processing"
Sep 11 17:45:50.017875 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.017864    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aeb1857-85d7-4fd0-b267-68d0e60738dc" containerName="payload-pre-processing"
Sep 11 17:45:50.017875 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.017872    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94e3e2a7-dcb6-4d61-b62c-37da2674c99a" containerName="maas-api"
Sep 11 17:45:50.017875 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.017877    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="94e3e2a7-dcb6-4d61-b62c-37da2674c99a" containerName="maas-api"
Sep 11 17:45:50.018317 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.017965    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c80eed9-b8a3-409a-b85f-d1c7fea05e94" containerName="payload-pre-processing"
Sep 11 17:45:50.018317 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.017974    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3454246-63c2-4f32-b368-f12af2cab60d" containerName="payload-processing"
Sep 11 17:45:50.018317 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.017979    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="94e3e2a7-dcb6-4d61-b62c-37da2674c99a" containerName="maas-api"
Sep 11 17:45:50.018317 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.017987    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7a9943b-e86a-43f7-9a86-042795d21fa9" containerName="payload-pre-processing"
Sep 11 17:45:50.018317 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.017994    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="be6adcf5-c27f-4d20-a164-170a1d12f060" containerName="maas-api"
Sep 11 17:45:50.018317 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.018000    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aeb1857-85d7-4fd0-b267-68d0e60738dc" containerName="payload-pre-processing"
Sep 11 17:45:50.021280 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.021256    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts"
Sep 11 17:45:50.024947 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.024843    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-d8a0d0-serving-cert\""
Sep 11 17:45:50.034949 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.034871    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts"]
Sep 11 17:45:50.073717 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.073678    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec058a31-1d64-4538-9ba5-4859c9c99223-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts\" (UID: \"ec058a31-1d64-4538-9ba5-4859c9c99223\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts"
Sep 11 17:45:50.073888 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.073751    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzp4r\" (UniqueName: \"kubernetes.io/projected/ec058a31-1d64-4538-9ba5-4859c9c99223-kube-api-access-tzp4r\") pod \"maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts\" (UID: \"ec058a31-1d64-4538-9ba5-4859c9c99223\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts"
Sep 11 17:45:50.073888 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.073791    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec058a31-1d64-4538-9ba5-4859c9c99223-metrics-certs\") pod \"maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts\" (UID: \"ec058a31-1d64-4538-9ba5-4859c9c99223\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts"
Sep 11 17:45:50.073888 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.073865    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec058a31-1d64-4538-9ba5-4859c9c99223-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts\" (UID: \"ec058a31-1d64-4538-9ba5-4859c9c99223\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts"
Sep 11 17:45:50.174424 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.174379    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec058a31-1d64-4538-9ba5-4859c9c99223-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts\" (UID: \"ec058a31-1d64-4538-9ba5-4859c9c99223\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts"
Sep 11 17:45:50.174424 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.174432    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzp4r\" (UniqueName: \"kubernetes.io/projected/ec058a31-1d64-4538-9ba5-4859c9c99223-kube-api-access-tzp4r\") pod \"maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts\" (UID: \"ec058a31-1d64-4538-9ba5-4859c9c99223\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts"
Sep 11 17:45:50.174675 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.174547    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec058a31-1d64-4538-9ba5-4859c9c99223-metrics-certs\") pod \"maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts\" (UID: \"ec058a31-1d64-4538-9ba5-4859c9c99223\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts"
Sep 11 17:45:50.174675 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.174655    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec058a31-1d64-4538-9ba5-4859c9c99223-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts\" (UID: \"ec058a31-1d64-4538-9ba5-4859c9c99223\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts"
Sep 11 17:45:50.174825 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:45:50.174779    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-serving-cert: secret "maas-api-e2e-ipp-a-d8a0d0-serving-cert" not found
Sep 11 17:45:50.175005 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:45:50.174870    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec058a31-1d64-4538-9ba5-4859c9c99223-maas-api-tls podName:ec058a31-1d64-4538-9ba5-4859c9c99223 nodeName:}" failed. No retries permitted until 2026-09-11 17:45:50.674854511 +0000 UTC m=+1518.479026908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ec058a31-1d64-4538-9ba5-4859c9c99223-maas-api-tls") pod "maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts" (UID: "ec058a31-1d64-4538-9ba5-4859c9c99223") : secret "maas-api-e2e-ipp-a-d8a0d0-serving-cert" not found
Sep 11 17:45:50.175242 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.175220    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec058a31-1d64-4538-9ba5-4859c9c99223-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts\" (UID: \"ec058a31-1d64-4538-9ba5-4859c9c99223\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts"
Sep 11 17:45:50.177057 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.177036    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec058a31-1d64-4538-9ba5-4859c9c99223-metrics-certs\") pod \"maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts\" (UID: \"ec058a31-1d64-4538-9ba5-4859c9c99223\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts"
Sep 11 17:45:50.184243 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.184214    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzp4r\" (UniqueName: \"kubernetes.io/projected/ec058a31-1d64-4538-9ba5-4859c9c99223-kube-api-access-tzp4r\") pod \"maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts\" (UID: \"ec058a31-1d64-4538-9ba5-4859c9c99223\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts"
Sep 11 17:45:50.680384 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.680345    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec058a31-1d64-4538-9ba5-4859c9c99223-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts\" (UID: \"ec058a31-1d64-4538-9ba5-4859c9c99223\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts"
Sep 11 17:45:50.682989 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.682956    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec058a31-1d64-4538-9ba5-4859c9c99223-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts\" (UID: \"ec058a31-1d64-4538-9ba5-4859c9c99223\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts"
Sep 11 17:45:50.934529 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:50.934447    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts"
Sep 11 17:45:51.208353 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:51.208320    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts"]
Sep 11 17:45:51.210471 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:45:51.210444    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec058a31_1d64_4538_9ba5_4859c9c99223.slice/crio-7897eb99541406b7b13012b7609ec7696731def61e7d90d488bbc66b32996711 WatchSource:0}: Error finding container 7897eb99541406b7b13012b7609ec7696731def61e7d90d488bbc66b32996711: Status 404 returned error can't find the container with id 7897eb99541406b7b13012b7609ec7696731def61e7d90d488bbc66b32996711
Sep 11 17:45:51.353384 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:51.353348    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts" event={"ID":"ec058a31-1d64-4538-9ba5-4859c9c99223","Type":"ContainerStarted","Data":"7897eb99541406b7b13012b7609ec7696731def61e7d90d488bbc66b32996711"}
Sep 11 17:45:53.365343 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:53.365307    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts" event={"ID":"ec058a31-1d64-4538-9ba5-4859c9c99223","Type":"ContainerStarted","Data":"7118c18efbb074b5f01adc90999450a76be50b7a93c736bcb2d3238c66b48eee"}
Sep 11 17:45:53.365725 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:53.365440    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts"
Sep 11 17:45:53.385665 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:53.385609    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts" podStartSLOduration=3.29173207 podStartE2EDuration="4.385590739s" totalImagesPullingTime="1.093858669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:45:53.384323889 +0000 UTC m=+1521.188496309" watchObservedRunningTime="2026-09-11 17:45:53.385590739 +0000 UTC m=+1521.189763157"
Sep 11 17:45:59.376002 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:45:59.375919    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts"
Sep 11 17:46:20.516599 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:46:20.516561    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z"]
Sep 11 17:46:20.521511 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:46:20.521487    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z"
Sep 11 17:46:20.524004 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:46:20.523978    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-1d492f-serving-cert\""
Sep 11 17:46:20.533612 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:46:20.533585    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z"]
Sep 11 17:46:20.673946 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:46:20.673883    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ceaf894-7532-4183-b2e2-1a98e7698708-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z\" (UID: \"8ceaf894-7532-4183-b2e2-1a98e7698708\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z"
Sep 11 17:46:20.674116 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:46:20.673968    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ceaf894-7532-4183-b2e2-1a98e7698708-metrics-certs\") pod \"maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z\" (UID: \"8ceaf894-7532-4183-b2e2-1a98e7698708\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z"
Sep 11 17:46:20.674116 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:46:20.674009    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ceaf894-7532-4183-b2e2-1a98e7698708-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z\" (UID: \"8ceaf894-7532-4183-b2e2-1a98e7698708\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z"
Sep 11 17:46:20.674116 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:46:20.674073    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r22qg\" (UniqueName: \"kubernetes.io/projected/8ceaf894-7532-4183-b2e2-1a98e7698708-kube-api-access-r22qg\") pod \"maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z\" (UID: \"8ceaf894-7532-4183-b2e2-1a98e7698708\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z"
Sep 11 17:46:20.775468 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:46:20.775378    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ceaf894-7532-4183-b2e2-1a98e7698708-metrics-certs\") pod \"maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z\" (UID: \"8ceaf894-7532-4183-b2e2-1a98e7698708\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z"
Sep 11 17:46:20.775468 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:46:20.775431    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ceaf894-7532-4183-b2e2-1a98e7698708-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z\" (UID: \"8ceaf894-7532-4183-b2e2-1a98e7698708\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z"
Sep 11 17:46:20.775671 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:46:20.775472    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r22qg\" (UniqueName: \"kubernetes.io/projected/8ceaf894-7532-4183-b2e2-1a98e7698708-kube-api-access-r22qg\") pod \"maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z\" (UID: \"8ceaf894-7532-4183-b2e2-1a98e7698708\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z"
Sep 11 17:46:20.775671 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:46:20.775525    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ceaf894-7532-4183-b2e2-1a98e7698708-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z\" (UID: \"8ceaf894-7532-4183-b2e2-1a98e7698708\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z"
Sep 11 17:46:20.775671 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:46:20.775619    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-serving-cert: secret "maas-api-e2e-ipp-b-1d492f-serving-cert" not found
Sep 11 17:46:20.775790 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:46:20.775680    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ceaf894-7532-4183-b2e2-1a98e7698708-maas-api-tls podName:8ceaf894-7532-4183-b2e2-1a98e7698708 nodeName:}" failed. No retries permitted until 2026-09-11 17:46:21.27566651 +0000 UTC m=+1549.079838908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8ceaf894-7532-4183-b2e2-1a98e7698708-maas-api-tls") pod "maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z" (UID: "8ceaf894-7532-4183-b2e2-1a98e7698708") : secret "maas-api-e2e-ipp-b-1d492f-serving-cert" not found
Sep 11 17:46:20.776204 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:46:20.776177    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ceaf894-7532-4183-b2e2-1a98e7698708-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z\" (UID: \"8ceaf894-7532-4183-b2e2-1a98e7698708\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z"
Sep 11 17:46:20.777830 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:46:20.777811    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ceaf894-7532-4183-b2e2-1a98e7698708-metrics-certs\") pod \"maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z\" (UID: \"8ceaf894-7532-4183-b2e2-1a98e7698708\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z"
Sep 11 17:46:20.784619 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:46:20.784589    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r22qg\" (UniqueName: \"kubernetes.io/projected/8ceaf894-7532-4183-b2e2-1a98e7698708-kube-api-access-r22qg\") pod \"maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z\" (UID: \"8ceaf894-7532-4183-b2e2-1a98e7698708\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z"
Sep 11 17:46:21.280251 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:46:21.280211    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ceaf894-7532-4183-b2e2-1a98e7698708-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z\" (UID: \"8ceaf894-7532-4183-b2e2-1a98e7698708\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z"
Sep 11 17:46:21.282796 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:46:21.282767    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ceaf894-7532-4183-b2e2-1a98e7698708-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z\" (UID: \"8ceaf894-7532-4183-b2e2-1a98e7698708\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z"
Sep 11 17:46:21.433621 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:46:21.433582    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z"
Sep 11 17:46:21.704173 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:46:21.704149    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z"]
Sep 11 17:46:21.706259 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:46:21.706227    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ceaf894_7532_4183_b2e2_1a98e7698708.slice/crio-bb188c2ce04f96e86fa7d86596f77afeab8176968d5968ce6a047ce296d240e6 WatchSource:0}: Error finding container bb188c2ce04f96e86fa7d86596f77afeab8176968d5968ce6a047ce296d240e6: Status 404 returned error can't find the container with id bb188c2ce04f96e86fa7d86596f77afeab8176968d5968ce6a047ce296d240e6
Sep 11 17:46:22.518653 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:46:22.518600    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z" event={"ID":"8ceaf894-7532-4183-b2e2-1a98e7698708","Type":"ContainerStarted","Data":"661c738f19ff432e49463a0f3364168a7c881d12b3d6468ec41434f864c0773d"}
Sep 11 17:46:22.518852 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:46:22.518669    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z" event={"ID":"8ceaf894-7532-4183-b2e2-1a98e7698708","Type":"ContainerStarted","Data":"bb188c2ce04f96e86fa7d86596f77afeab8176968d5968ce6a047ce296d240e6"}
Sep 11 17:46:22.518852 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:46:22.518699    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z"
Sep 11 17:46:22.541063 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:46:22.540953    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z" podStartSLOduration=1.967515722 podStartE2EDuration="2.540937367s" totalImagesPullingTime="573.421645ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:46:22.539102171 +0000 UTC m=+1550.343274599" watchObservedRunningTime="2026-09-11 17:46:22.540937367 +0000 UTC m=+1550.345109819"
Sep 11 17:46:28.530022 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:46:28.529991    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z"
Sep 11 17:46:47.614623 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:46:47.614585    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zn9cq_521643fa-f596-493a-89e3-5b0582f6ed0b/payload-processing/0.log"
Sep 11 17:47:28.658150 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:28.658080    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zn9cq_521643fa-f596-493a-89e3-5b0582f6ed0b/payload-processing/0.log"
Sep 11 17:47:36.546728 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:36.546691    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb"]
Sep 11 17:47:36.550932 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:36.550879    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb"
Sep 11 17:47:36.554096 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:36.554068    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-877b01-serving-cert\""
Sep 11 17:47:36.567761 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:36.567716    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb"]
Sep 11 17:47:36.693584 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:36.693550    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ff5d0a-1c37-4667-9277-c5f6901ba8ef-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb\" (UID: \"26ff5d0a-1c37-4667-9277-c5f6901ba8ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb"
Sep 11 17:47:36.693756 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:36.693605    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26ff5d0a-1c37-4667-9277-c5f6901ba8ef-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb\" (UID: \"26ff5d0a-1c37-4667-9277-c5f6901ba8ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb"
Sep 11 17:47:36.693756 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:36.693663    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ntfv\" (UniqueName: \"kubernetes.io/projected/26ff5d0a-1c37-4667-9277-c5f6901ba8ef-kube-api-access-8ntfv\") pod \"maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb\" (UID: \"26ff5d0a-1c37-4667-9277-c5f6901ba8ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb"
Sep 11 17:47:36.693756 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:36.693703    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ff5d0a-1c37-4667-9277-c5f6901ba8ef-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb\" (UID: \"26ff5d0a-1c37-4667-9277-c5f6901ba8ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb"
Sep 11 17:47:36.794586 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:36.794552    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ff5d0a-1c37-4667-9277-c5f6901ba8ef-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb\" (UID: \"26ff5d0a-1c37-4667-9277-c5f6901ba8ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb"
Sep 11 17:47:36.794786 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:36.794609    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26ff5d0a-1c37-4667-9277-c5f6901ba8ef-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb\" (UID: \"26ff5d0a-1c37-4667-9277-c5f6901ba8ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb"
Sep 11 17:47:36.794786 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:36.794666    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ntfv\" (UniqueName: \"kubernetes.io/projected/26ff5d0a-1c37-4667-9277-c5f6901ba8ef-kube-api-access-8ntfv\") pod \"maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb\" (UID: \"26ff5d0a-1c37-4667-9277-c5f6901ba8ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb"
Sep 11 17:47:36.794786 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:36.794722    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ff5d0a-1c37-4667-9277-c5f6901ba8ef-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb\" (UID: \"26ff5d0a-1c37-4667-9277-c5f6901ba8ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb"
Sep 11 17:47:36.794997 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:47:36.794838    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-serving-cert: secret "maas-api-e2e-ipp-cleanup-877b01-serving-cert" not found
Sep 11 17:47:36.794997 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:47:36.794954    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ff5d0a-1c37-4667-9277-c5f6901ba8ef-maas-api-tls podName:26ff5d0a-1c37-4667-9277-c5f6901ba8ef nodeName:}" failed. No retries permitted until 2026-09-11 17:47:37.294930805 +0000 UTC m=+1625.099103208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/26ff5d0a-1c37-4667-9277-c5f6901ba8ef-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb" (UID: "26ff5d0a-1c37-4667-9277-c5f6901ba8ef") : secret "maas-api-e2e-ipp-cleanup-877b01-serving-cert" not found
Sep 11 17:47:36.795364 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:36.795340    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ff5d0a-1c37-4667-9277-c5f6901ba8ef-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb\" (UID: \"26ff5d0a-1c37-4667-9277-c5f6901ba8ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb"
Sep 11 17:47:36.797219 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:36.797162    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ff5d0a-1c37-4667-9277-c5f6901ba8ef-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb\" (UID: \"26ff5d0a-1c37-4667-9277-c5f6901ba8ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb"
Sep 11 17:47:36.805201 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:36.805171    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ntfv\" (UniqueName: \"kubernetes.io/projected/26ff5d0a-1c37-4667-9277-c5f6901ba8ef-kube-api-access-8ntfv\") pod \"maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb\" (UID: \"26ff5d0a-1c37-4667-9277-c5f6901ba8ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb"
Sep 11 17:47:37.267371 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:37.267336    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6"]
Sep 11 17:47:37.271106 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:37.271081    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6"
Sep 11 17:47:37.273661 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:37.273633    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-877b01\""
Sep 11 17:47:37.273809 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:37.273688    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-877b01-dockercfg-vgtww\""
Sep 11 17:47:37.281948 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:37.281889    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6"]
Sep 11 17:47:37.300889 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:37.300841    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26ff5d0a-1c37-4667-9277-c5f6901ba8ef-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb\" (UID: \"26ff5d0a-1c37-4667-9277-c5f6901ba8ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb"
Sep 11 17:47:37.311587 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:37.311554    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26ff5d0a-1c37-4667-9277-c5f6901ba8ef-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb\" (UID: \"26ff5d0a-1c37-4667-9277-c5f6901ba8ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb"
Sep 11 17:47:37.402356 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:37.402318    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f888c936-827c-4871-96bc-fbcec11ee251-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6\" (UID: \"f888c936-827c-4871-96bc-fbcec11ee251\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6"
Sep 11 17:47:37.402539 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:37.402400    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj9kj\" (UniqueName: \"kubernetes.io/projected/f888c936-827c-4871-96bc-fbcec11ee251-kube-api-access-pj9kj\") pod \"payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6\" (UID: \"f888c936-827c-4871-96bc-fbcec11ee251\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6"
Sep 11 17:47:37.463198 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:37.463163    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb"
Sep 11 17:47:37.503858 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:37.503639    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj9kj\" (UniqueName: \"kubernetes.io/projected/f888c936-827c-4871-96bc-fbcec11ee251-kube-api-access-pj9kj\") pod \"payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6\" (UID: \"f888c936-827c-4871-96bc-fbcec11ee251\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6"
Sep 11 17:47:37.503858 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:37.503752    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f888c936-827c-4871-96bc-fbcec11ee251-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6\" (UID: \"f888c936-827c-4871-96bc-fbcec11ee251\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6"
Sep 11 17:47:37.504672 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:37.504645    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f888c936-827c-4871-96bc-fbcec11ee251-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6\" (UID: \"f888c936-827c-4871-96bc-fbcec11ee251\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6"
Sep 11 17:47:37.512885 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:37.512659    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj9kj\" (UniqueName: \"kubernetes.io/projected/f888c936-827c-4871-96bc-fbcec11ee251-kube-api-access-pj9kj\") pod \"payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6\" (UID: \"f888c936-827c-4871-96bc-fbcec11ee251\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6"
Sep 11 17:47:37.583885 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:37.583846    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6"
Sep 11 17:47:37.739976 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:37.739950    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb"]
Sep 11 17:47:37.741640 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:47:37.741617    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ff5d0a_1c37_4667_9277_c5f6901ba8ef.slice/crio-e93f8eebb22c41cefb91f4837f26e9592ad8e099e899dd94ea48207be71bfb38 WatchSource:0}: Error finding container e93f8eebb22c41cefb91f4837f26e9592ad8e099e899dd94ea48207be71bfb38: Status 404 returned error can't find the container with id e93f8eebb22c41cefb91f4837f26e9592ad8e099e899dd94ea48207be71bfb38
Sep 11 17:47:37.743054 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:37.743033    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:47:37.863312 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:37.863286    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6"]
Sep 11 17:47:37.865277 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:47:37.865244    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf888c936_827c_4871_96bc_fbcec11ee251.slice/crio-9e43600c503f3fd8738bf9859c95f93b49766deebc1cc1c6a2b55a0f217f2291 WatchSource:0}: Error finding container 9e43600c503f3fd8738bf9859c95f93b49766deebc1cc1c6a2b55a0f217f2291: Status 404 returned error can't find the container with id 9e43600c503f3fd8738bf9859c95f93b49766deebc1cc1c6a2b55a0f217f2291
Sep 11 17:47:37.902952 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:37.902918    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb" event={"ID":"26ff5d0a-1c37-4667-9277-c5f6901ba8ef","Type":"ContainerStarted","Data":"e93f8eebb22c41cefb91f4837f26e9592ad8e099e899dd94ea48207be71bfb38"}
Sep 11 17:47:37.903975 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:37.903955    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6" event={"ID":"f888c936-827c-4871-96bc-fbcec11ee251","Type":"ContainerStarted","Data":"9e43600c503f3fd8738bf9859c95f93b49766deebc1cc1c6a2b55a0f217f2291"}
Sep 11 17:47:38.910714 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:38.910674    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6" event={"ID":"f888c936-827c-4871-96bc-fbcec11ee251","Type":"ContainerStarted","Data":"db9517017280bdd7ec3cf94a95e9b85301d7b7617530d9498a3702a62d3da8fa"}
Sep 11 17:47:38.911192 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:38.910731    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6"
Sep 11 17:47:38.912162 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:38.912140    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb" event={"ID":"26ff5d0a-1c37-4667-9277-c5f6901ba8ef","Type":"ContainerStarted","Data":"d3adbb3bd9e62686eb6d1cb3c2db868fbc46870836aec4c77ed3f10bf5d0f7dc"}
Sep 11 17:47:38.912284 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:38.912268    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb"
Sep 11 17:47:38.933088 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:38.933038    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6" podStartSLOduration=1.933021627 podStartE2EDuration="1.933021627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:47:38.931321995 +0000 UTC m=+1626.735494415" watchObservedRunningTime="2026-09-11 17:47:38.933021627 +0000 UTC m=+1626.737194045"
Sep 11 17:47:38.959764 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:38.959713    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb" podStartSLOduration=2.558036535 podStartE2EDuration="2.959698131s" totalImagesPullingTime="401.661596ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:47:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:47:38.959031382 +0000 UTC m=+1626.763203801" watchObservedRunningTime="2026-09-11 17:47:38.959698131 +0000 UTC m=+1626.763870564"
Sep 11 17:47:44.923289 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:44.923261    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb"
Sep 11 17:47:49.919704 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:47:49.919671    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6"
Sep 11 17:48:14.636750 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:14.636714    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb"]
Sep 11 17:48:14.637251 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:14.636956    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb" podUID="26ff5d0a-1c37-4667-9277-c5f6901ba8ef" containerName="maas-api" containerID="cri-o://d3adbb3bd9e62686eb6d1cb3c2db868fbc46870836aec4c77ed3f10bf5d0f7dc" gracePeriod=30
Sep 11 17:48:14.720123 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:14.720087    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6"]
Sep 11 17:48:14.720381 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:14.720332    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6" podUID="f888c936-827c-4871-96bc-fbcec11ee251" containerName="payload-pre-processing" containerID="cri-o://db9517017280bdd7ec3cf94a95e9b85301d7b7617530d9498a3702a62d3da8fa" gracePeriod=30
Sep 11 17:48:14.984417 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:14.984372    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb"
Sep 11 17:48:15.017397 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.017371    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6"
Sep 11 17:48:15.064013 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.063982    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ff5d0a-1c37-4667-9277-c5f6901ba8ef-openshift-service-ca\") pod \"26ff5d0a-1c37-4667-9277-c5f6901ba8ef\" (UID: \"26ff5d0a-1c37-4667-9277-c5f6901ba8ef\") "
Sep 11 17:48:15.064179 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.064030    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ff5d0a-1c37-4667-9277-c5f6901ba8ef-metrics-certs\") pod \"26ff5d0a-1c37-4667-9277-c5f6901ba8ef\" (UID: \"26ff5d0a-1c37-4667-9277-c5f6901ba8ef\") "
Sep 11 17:48:15.064179 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.064080    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26ff5d0a-1c37-4667-9277-c5f6901ba8ef-maas-api-tls\") pod \"26ff5d0a-1c37-4667-9277-c5f6901ba8ef\" (UID: \"26ff5d0a-1c37-4667-9277-c5f6901ba8ef\") "
Sep 11 17:48:15.064179 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.064103    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ntfv\" (UniqueName: \"kubernetes.io/projected/26ff5d0a-1c37-4667-9277-c5f6901ba8ef-kube-api-access-8ntfv\") pod \"26ff5d0a-1c37-4667-9277-c5f6901ba8ef\" (UID: \"26ff5d0a-1c37-4667-9277-c5f6901ba8ef\") "
Sep 11 17:48:15.064400 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.064373    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ff5d0a-1c37-4667-9277-c5f6901ba8ef-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "26ff5d0a-1c37-4667-9277-c5f6901ba8ef" (UID: "26ff5d0a-1c37-4667-9277-c5f6901ba8ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:48:15.064482 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.064457    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ff5d0a-1c37-4667-9277-c5f6901ba8ef-openshift-service-ca\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:48:15.066150 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.066120    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ff5d0a-1c37-4667-9277-c5f6901ba8ef-kube-api-access-8ntfv" (OuterVolumeSpecName: "kube-api-access-8ntfv") pod "26ff5d0a-1c37-4667-9277-c5f6901ba8ef" (UID: "26ff5d0a-1c37-4667-9277-c5f6901ba8ef"). InnerVolumeSpecName "kube-api-access-8ntfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:48:15.066267 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.066158    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26ff5d0a-1c37-4667-9277-c5f6901ba8ef-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "26ff5d0a-1c37-4667-9277-c5f6901ba8ef" (UID: "26ff5d0a-1c37-4667-9277-c5f6901ba8ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:48:15.066267 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.066212    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26ff5d0a-1c37-4667-9277-c5f6901ba8ef-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "26ff5d0a-1c37-4667-9277-c5f6901ba8ef" (UID: "26ff5d0a-1c37-4667-9277-c5f6901ba8ef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:48:15.099336 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.099301    2560 generic.go:356] "Generic (PLEG): container finished" podID="f888c936-827c-4871-96bc-fbcec11ee251" containerID="db9517017280bdd7ec3cf94a95e9b85301d7b7617530d9498a3702a62d3da8fa" exitCode=0
Sep 11 17:48:15.099519 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.099346    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6"
Sep 11 17:48:15.099519 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.099421    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6" event={"ID":"f888c936-827c-4871-96bc-fbcec11ee251","Type":"ContainerDied","Data":"db9517017280bdd7ec3cf94a95e9b85301d7b7617530d9498a3702a62d3da8fa"}
Sep 11 17:48:15.099519 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.099478    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6" event={"ID":"f888c936-827c-4871-96bc-fbcec11ee251","Type":"ContainerDied","Data":"9e43600c503f3fd8738bf9859c95f93b49766deebc1cc1c6a2b55a0f217f2291"}
Sep 11 17:48:15.099519 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.099515    2560 scope.go:117] "RemoveContainer" containerID="db9517017280bdd7ec3cf94a95e9b85301d7b7617530d9498a3702a62d3da8fa"
Sep 11 17:48:15.100950 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.100929    2560 generic.go:356] "Generic (PLEG): container finished" podID="26ff5d0a-1c37-4667-9277-c5f6901ba8ef" containerID="d3adbb3bd9e62686eb6d1cb3c2db868fbc46870836aec4c77ed3f10bf5d0f7dc" exitCode=0
Sep 11 17:48:15.101058 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.100961    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb" event={"ID":"26ff5d0a-1c37-4667-9277-c5f6901ba8ef","Type":"ContainerDied","Data":"d3adbb3bd9e62686eb6d1cb3c2db868fbc46870836aec4c77ed3f10bf5d0f7dc"}
Sep 11 17:48:15.101058 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.101045    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb" event={"ID":"26ff5d0a-1c37-4667-9277-c5f6901ba8ef","Type":"ContainerDied","Data":"e93f8eebb22c41cefb91f4837f26e9592ad8e099e899dd94ea48207be71bfb38"}
Sep 11 17:48:15.101144 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.100974    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb"
Sep 11 17:48:15.109947 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.109891    2560 scope.go:117] "RemoveContainer" containerID="db9517017280bdd7ec3cf94a95e9b85301d7b7617530d9498a3702a62d3da8fa"
Sep 11 17:48:15.110315 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:48:15.110213    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db9517017280bdd7ec3cf94a95e9b85301d7b7617530d9498a3702a62d3da8fa\": container with ID starting with db9517017280bdd7ec3cf94a95e9b85301d7b7617530d9498a3702a62d3da8fa not found: ID does not exist" containerID="db9517017280bdd7ec3cf94a95e9b85301d7b7617530d9498a3702a62d3da8fa"
Sep 11 17:48:15.110315 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.110280    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db9517017280bdd7ec3cf94a95e9b85301d7b7617530d9498a3702a62d3da8fa"} err="failed to get container status \"db9517017280bdd7ec3cf94a95e9b85301d7b7617530d9498a3702a62d3da8fa\": rpc error: code = NotFound desc = could not find container \"db9517017280bdd7ec3cf94a95e9b85301d7b7617530d9498a3702a62d3da8fa\": container with ID starting with db9517017280bdd7ec3cf94a95e9b85301d7b7617530d9498a3702a62d3da8fa not found: ID does not exist"
Sep 11 17:48:15.110461 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.110303    2560 scope.go:117] "RemoveContainer" containerID="d3adbb3bd9e62686eb6d1cb3c2db868fbc46870836aec4c77ed3f10bf5d0f7dc"
Sep 11 17:48:15.126639 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.126582    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb"]
Sep 11 17:48:15.127835 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.127815    2560 scope.go:117] "RemoveContainer" containerID="d3adbb3bd9e62686eb6d1cb3c2db868fbc46870836aec4c77ed3f10bf5d0f7dc"
Sep 11 17:48:15.130558 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:48:15.130529    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3adbb3bd9e62686eb6d1cb3c2db868fbc46870836aec4c77ed3f10bf5d0f7dc\": container with ID starting with d3adbb3bd9e62686eb6d1cb3c2db868fbc46870836aec4c77ed3f10bf5d0f7dc not found: ID does not exist" containerID="d3adbb3bd9e62686eb6d1cb3c2db868fbc46870836aec4c77ed3f10bf5d0f7dc"
Sep 11 17:48:15.130645 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.130567    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3adbb3bd9e62686eb6d1cb3c2db868fbc46870836aec4c77ed3f10bf5d0f7dc"} err="failed to get container status \"d3adbb3bd9e62686eb6d1cb3c2db868fbc46870836aec4c77ed3f10bf5d0f7dc\": rpc error: code = NotFound desc = could not find container \"d3adbb3bd9e62686eb6d1cb3c2db868fbc46870836aec4c77ed3f10bf5d0f7dc\": container with ID starting with d3adbb3bd9e62686eb6d1cb3c2db868fbc46870836aec4c77ed3f10bf5d0f7dc not found: ID does not exist"
Sep 11 17:48:15.132310 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.132288    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb"]
Sep 11 17:48:15.165630 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.165594    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pj9kj\" (UniqueName: \"kubernetes.io/projected/f888c936-827c-4871-96bc-fbcec11ee251-kube-api-access-pj9kj\") pod \"f888c936-827c-4871-96bc-fbcec11ee251\" (UID: \"f888c936-827c-4871-96bc-fbcec11ee251\") "
Sep 11 17:48:15.165832 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.165682    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f888c936-827c-4871-96bc-fbcec11ee251-plugins-config-volume\") pod \"f888c936-827c-4871-96bc-fbcec11ee251\" (UID: \"f888c936-827c-4871-96bc-fbcec11ee251\") "
Sep 11 17:48:15.165920 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.165876    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26ff5d0a-1c37-4667-9277-c5f6901ba8ef-maas-api-tls\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:48:15.165920 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.165887    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ntfv\" (UniqueName: \"kubernetes.io/projected/26ff5d0a-1c37-4667-9277-c5f6901ba8ef-kube-api-access-8ntfv\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:48:15.166043 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.165919    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ff5d0a-1c37-4667-9277-c5f6901ba8ef-metrics-certs\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:48:15.166101 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.166039    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f888c936-827c-4871-96bc-fbcec11ee251-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f888c936-827c-4871-96bc-fbcec11ee251" (UID: "f888c936-827c-4871-96bc-fbcec11ee251"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:48:15.167826 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.167800    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f888c936-827c-4871-96bc-fbcec11ee251-kube-api-access-pj9kj" (OuterVolumeSpecName: "kube-api-access-pj9kj") pod "f888c936-827c-4871-96bc-fbcec11ee251" (UID: "f888c936-827c-4871-96bc-fbcec11ee251"). InnerVolumeSpecName "kube-api-access-pj9kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:48:15.267126 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.267036    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f888c936-827c-4871-96bc-fbcec11ee251-plugins-config-volume\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:48:15.267126 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.267063    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj9kj\" (UniqueName: \"kubernetes.io/projected/f888c936-827c-4871-96bc-fbcec11ee251-kube-api-access-pj9kj\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:48:15.426626 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.426588    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6"]
Sep 11 17:48:15.437549 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:15.437515    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-877b01-79d7fff77d-7c7f6"]
Sep 11 17:48:16.722182 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:16.722140    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26ff5d0a-1c37-4667-9277-c5f6901ba8ef" path="/var/lib/kubelet/pods/26ff5d0a-1c37-4667-9277-c5f6901ba8ef/volumes"
Sep 11 17:48:16.722782 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:16.722651    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f888c936-827c-4871-96bc-fbcec11ee251" path="/var/lib/kubelet/pods/f888c936-827c-4871-96bc-fbcec11ee251/volumes"
Sep 11 17:48:17.918748 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:17.918695    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-877b01-655ccd556-qkqxb" podUID="26ff5d0a-1c37-4667-9277-c5f6901ba8ef" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.68:8443/health\": context deadline exceeded"
Sep 11 17:48:32.872973 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:32.872938    2560 scope.go:117] "RemoveContainer" containerID="7fe229883c805da7b9647be5abf16c1536e10116f8d2d8188dd14360c46a42a2"
Sep 11 17:48:41.213811 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:41.213774    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts"]
Sep 11 17:48:41.214285 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:41.214026    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts" podUID="ec058a31-1d64-4538-9ba5-4859c9c99223" containerName="maas-api" containerID="cri-o://7118c18efbb074b5f01adc90999450a76be50b7a93c736bcb2d3238c66b48eee" gracePeriod=30
Sep 11 17:48:41.487633 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:41.487602    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts"
Sep 11 17:48:41.523130 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:41.523084    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec058a31-1d64-4538-9ba5-4859c9c99223-metrics-certs\") pod \"ec058a31-1d64-4538-9ba5-4859c9c99223\" (UID: \"ec058a31-1d64-4538-9ba5-4859c9c99223\") "
Sep 11 17:48:41.523310 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:41.523140    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec058a31-1d64-4538-9ba5-4859c9c99223-maas-api-tls\") pod \"ec058a31-1d64-4538-9ba5-4859c9c99223\" (UID: \"ec058a31-1d64-4538-9ba5-4859c9c99223\") "
Sep 11 17:48:41.523310 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:41.523190    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzp4r\" (UniqueName: \"kubernetes.io/projected/ec058a31-1d64-4538-9ba5-4859c9c99223-kube-api-access-tzp4r\") pod \"ec058a31-1d64-4538-9ba5-4859c9c99223\" (UID: \"ec058a31-1d64-4538-9ba5-4859c9c99223\") "
Sep 11 17:48:41.523310 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:41.523265    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec058a31-1d64-4538-9ba5-4859c9c99223-openshift-service-ca\") pod \"ec058a31-1d64-4538-9ba5-4859c9c99223\" (UID: \"ec058a31-1d64-4538-9ba5-4859c9c99223\") "
Sep 11 17:48:41.523939 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:41.523888    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec058a31-1d64-4538-9ba5-4859c9c99223-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ec058a31-1d64-4538-9ba5-4859c9c99223" (UID: "ec058a31-1d64-4538-9ba5-4859c9c99223"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:48:41.525648 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:41.525616    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec058a31-1d64-4538-9ba5-4859c9c99223-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ec058a31-1d64-4538-9ba5-4859c9c99223" (UID: "ec058a31-1d64-4538-9ba5-4859c9c99223"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:48:41.525912 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:41.525873    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec058a31-1d64-4538-9ba5-4859c9c99223-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ec058a31-1d64-4538-9ba5-4859c9c99223" (UID: "ec058a31-1d64-4538-9ba5-4859c9c99223"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:48:41.525999 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:41.525889    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec058a31-1d64-4538-9ba5-4859c9c99223-kube-api-access-tzp4r" (OuterVolumeSpecName: "kube-api-access-tzp4r") pod "ec058a31-1d64-4538-9ba5-4859c9c99223" (UID: "ec058a31-1d64-4538-9ba5-4859c9c99223"). InnerVolumeSpecName "kube-api-access-tzp4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:48:41.624998 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:41.624963    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec058a31-1d64-4538-9ba5-4859c9c99223-metrics-certs\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:48:41.624998 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:41.624992    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec058a31-1d64-4538-9ba5-4859c9c99223-maas-api-tls\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:48:41.624998 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:41.625002    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzp4r\" (UniqueName: \"kubernetes.io/projected/ec058a31-1d64-4538-9ba5-4859c9c99223-kube-api-access-tzp4r\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:48:41.625224 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:41.625012    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec058a31-1d64-4538-9ba5-4859c9c99223-openshift-service-ca\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:48:42.244841 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:42.244806    2560 generic.go:356] "Generic (PLEG): container finished" podID="ec058a31-1d64-4538-9ba5-4859c9c99223" containerID="7118c18efbb074b5f01adc90999450a76be50b7a93c736bcb2d3238c66b48eee" exitCode=0
Sep 11 17:48:42.245358 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:42.244846    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts"
Sep 11 17:48:42.245358 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:42.244883    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts" event={"ID":"ec058a31-1d64-4538-9ba5-4859c9c99223","Type":"ContainerDied","Data":"7118c18efbb074b5f01adc90999450a76be50b7a93c736bcb2d3238c66b48eee"}
Sep 11 17:48:42.245358 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:42.244951    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts" event={"ID":"ec058a31-1d64-4538-9ba5-4859c9c99223","Type":"ContainerDied","Data":"7897eb99541406b7b13012b7609ec7696731def61e7d90d488bbc66b32996711"}
Sep 11 17:48:42.245358 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:42.244976    2560 scope.go:117] "RemoveContainer" containerID="7118c18efbb074b5f01adc90999450a76be50b7a93c736bcb2d3238c66b48eee"
Sep 11 17:48:42.256459 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:42.256438    2560 scope.go:117] "RemoveContainer" containerID="7118c18efbb074b5f01adc90999450a76be50b7a93c736bcb2d3238c66b48eee"
Sep 11 17:48:42.256776 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:48:42.256753    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7118c18efbb074b5f01adc90999450a76be50b7a93c736bcb2d3238c66b48eee\": container with ID starting with 7118c18efbb074b5f01adc90999450a76be50b7a93c736bcb2d3238c66b48eee not found: ID does not exist" containerID="7118c18efbb074b5f01adc90999450a76be50b7a93c736bcb2d3238c66b48eee"
Sep 11 17:48:42.256853 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:42.256788    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7118c18efbb074b5f01adc90999450a76be50b7a93c736bcb2d3238c66b48eee"} err="failed to get container status \"7118c18efbb074b5f01adc90999450a76be50b7a93c736bcb2d3238c66b48eee\": rpc error: code = NotFound desc = could not find container \"7118c18efbb074b5f01adc90999450a76be50b7a93c736bcb2d3238c66b48eee\": container with ID starting with 7118c18efbb074b5f01adc90999450a76be50b7a93c736bcb2d3238c66b48eee not found: ID does not exist"
Sep 11 17:48:42.294323 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:42.294280    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts"]
Sep 11 17:48:42.302532 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:42.302499    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d8a0d0-db66565bf-q9hts"]
Sep 11 17:48:42.715285 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:48:42.715252    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec058a31-1d64-4538-9ba5-4859c9c99223" path="/var/lib/kubelet/pods/ec058a31-1d64-4538-9ba5-4859c9c99223/volumes"
Sep 11 17:49:41.724204 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:41.724167    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z"]
Sep 11 17:49:41.724752 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:41.724443    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z" podUID="8ceaf894-7532-4183-b2e2-1a98e7698708" containerName="maas-api" containerID="cri-o://661c738f19ff432e49463a0f3364168a7c881d12b3d6468ec41434f864c0773d" gracePeriod=30
Sep 11 17:49:41.981676 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:41.981607    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z"
Sep 11 17:49:42.098908 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:42.098869    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ceaf894-7532-4183-b2e2-1a98e7698708-metrics-certs\") pod \"8ceaf894-7532-4183-b2e2-1a98e7698708\" (UID: \"8ceaf894-7532-4183-b2e2-1a98e7698708\") "
Sep 11 17:49:42.099108 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:42.098932    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r22qg\" (UniqueName: \"kubernetes.io/projected/8ceaf894-7532-4183-b2e2-1a98e7698708-kube-api-access-r22qg\") pod \"8ceaf894-7532-4183-b2e2-1a98e7698708\" (UID: \"8ceaf894-7532-4183-b2e2-1a98e7698708\") "
Sep 11 17:49:42.099108 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:42.098975    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ceaf894-7532-4183-b2e2-1a98e7698708-maas-api-tls\") pod \"8ceaf894-7532-4183-b2e2-1a98e7698708\" (UID: \"8ceaf894-7532-4183-b2e2-1a98e7698708\") "
Sep 11 17:49:42.099108 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:42.099001    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ceaf894-7532-4183-b2e2-1a98e7698708-openshift-service-ca\") pod \"8ceaf894-7532-4183-b2e2-1a98e7698708\" (UID: \"8ceaf894-7532-4183-b2e2-1a98e7698708\") "
Sep 11 17:49:42.099433 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:42.099400    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ceaf894-7532-4183-b2e2-1a98e7698708-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8ceaf894-7532-4183-b2e2-1a98e7698708" (UID: "8ceaf894-7532-4183-b2e2-1a98e7698708"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:49:42.101098 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:42.101067    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ceaf894-7532-4183-b2e2-1a98e7698708-kube-api-access-r22qg" (OuterVolumeSpecName: "kube-api-access-r22qg") pod "8ceaf894-7532-4183-b2e2-1a98e7698708" (UID: "8ceaf894-7532-4183-b2e2-1a98e7698708"). InnerVolumeSpecName "kube-api-access-r22qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:49:42.101220 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:42.101120    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ceaf894-7532-4183-b2e2-1a98e7698708-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8ceaf894-7532-4183-b2e2-1a98e7698708" (UID: "8ceaf894-7532-4183-b2e2-1a98e7698708"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:49:42.101220 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:42.101164    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ceaf894-7532-4183-b2e2-1a98e7698708-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8ceaf894-7532-4183-b2e2-1a98e7698708" (UID: "8ceaf894-7532-4183-b2e2-1a98e7698708"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:49:42.200396 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:42.200358    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ceaf894-7532-4183-b2e2-1a98e7698708-maas-api-tls\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:49:42.200396 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:42.200393    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ceaf894-7532-4183-b2e2-1a98e7698708-openshift-service-ca\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:49:42.200396 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:42.200402    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ceaf894-7532-4183-b2e2-1a98e7698708-metrics-certs\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:49:42.200641 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:42.200412    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r22qg\" (UniqueName: \"kubernetes.io/projected/8ceaf894-7532-4183-b2e2-1a98e7698708-kube-api-access-r22qg\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:49:42.544973 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:42.544942    2560 generic.go:356] "Generic (PLEG): container finished" podID="8ceaf894-7532-4183-b2e2-1a98e7698708" containerID="661c738f19ff432e49463a0f3364168a7c881d12b3d6468ec41434f864c0773d" exitCode=0
Sep 11 17:49:42.545152 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:42.544983    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z"
Sep 11 17:49:42.545152 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:42.544989    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z" event={"ID":"8ceaf894-7532-4183-b2e2-1a98e7698708","Type":"ContainerDied","Data":"661c738f19ff432e49463a0f3364168a7c881d12b3d6468ec41434f864c0773d"}
Sep 11 17:49:42.545152 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:42.545092    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z" event={"ID":"8ceaf894-7532-4183-b2e2-1a98e7698708","Type":"ContainerDied","Data":"bb188c2ce04f96e86fa7d86596f77afeab8176968d5968ce6a047ce296d240e6"}
Sep 11 17:49:42.545152 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:42.545122    2560 scope.go:117] "RemoveContainer" containerID="661c738f19ff432e49463a0f3364168a7c881d12b3d6468ec41434f864c0773d"
Sep 11 17:49:42.554571 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:42.554549    2560 scope.go:117] "RemoveContainer" containerID="661c738f19ff432e49463a0f3364168a7c881d12b3d6468ec41434f864c0773d"
Sep 11 17:49:42.554811 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:49:42.554792    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"661c738f19ff432e49463a0f3364168a7c881d12b3d6468ec41434f864c0773d\": container with ID starting with 661c738f19ff432e49463a0f3364168a7c881d12b3d6468ec41434f864c0773d not found: ID does not exist" containerID="661c738f19ff432e49463a0f3364168a7c881d12b3d6468ec41434f864c0773d"
Sep 11 17:49:42.554865 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:42.554820    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"661c738f19ff432e49463a0f3364168a7c881d12b3d6468ec41434f864c0773d"} err="failed to get container status \"661c738f19ff432e49463a0f3364168a7c881d12b3d6468ec41434f864c0773d\": rpc error: code = NotFound desc = could not find container \"661c738f19ff432e49463a0f3364168a7c881d12b3d6468ec41434f864c0773d\": container with ID starting with 661c738f19ff432e49463a0f3364168a7c881d12b3d6468ec41434f864c0773d not found: ID does not exist"
Sep 11 17:49:42.577749 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:42.577724    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z"]
Sep 11 17:49:42.584736 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:42.584712    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1d492f-77b989fb77-dgv5z"]
Sep 11 17:49:42.715811 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:42.715778    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ceaf894-7532-4183-b2e2-1a98e7698708" path="/var/lib/kubelet/pods/8ceaf894-7532-4183-b2e2-1a98e7698708/volumes"
Sep 11 17:49:59.917693 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:59.917657    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct"]
Sep 11 17:49:59.918236 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:59.917915    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct" podUID="8d27723b-8c92-45d2-ab77-80ecb372d5ef" containerName="payload-processing" containerID="cri-o://96c8a74b412bc828e0856a745e1d4e80a43a041e365f1fe2f298f6ef2bd35eb7" gracePeriod=30
Sep 11 17:49:59.926652 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:59.926609    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd"]
Sep 11 17:49:59.927671 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:49:59.927636    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd" podUID="16b9fd20-840b-47a2-a977-2b67f535c831" containerName="payload-pre-processing" containerID="cri-o://35e4e37aab1e606acbae1ce93f61a84f0f5c72563ec91e05f45e193fa47a275b" gracePeriod=30
Sep 11 17:50:00.224224 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.224195    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct"
Sep 11 17:50:00.227240 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.227214    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd"
Sep 11 17:50:00.271078 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.271046    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d27723b-8c92-45d2-ab77-80ecb372d5ef-plugins-config-volume\") pod \"8d27723b-8c92-45d2-ab77-80ecb372d5ef\" (UID: \"8d27723b-8c92-45d2-ab77-80ecb372d5ef\") "
Sep 11 17:50:00.271262 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.271108    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b9fd20-840b-47a2-a977-2b67f535c831-plugins-config-volume\") pod \"16b9fd20-840b-47a2-a977-2b67f535c831\" (UID: \"16b9fd20-840b-47a2-a977-2b67f535c831\") "
Sep 11 17:50:00.271262 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.271150    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dp4z7\" (UniqueName: \"kubernetes.io/projected/8d27723b-8c92-45d2-ab77-80ecb372d5ef-kube-api-access-dp4z7\") pod \"8d27723b-8c92-45d2-ab77-80ecb372d5ef\" (UID: \"8d27723b-8c92-45d2-ab77-80ecb372d5ef\") "
Sep 11 17:50:00.271262 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.271179    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhmj4\" (UniqueName: \"kubernetes.io/projected/16b9fd20-840b-47a2-a977-2b67f535c831-kube-api-access-rhmj4\") pod \"16b9fd20-840b-47a2-a977-2b67f535c831\" (UID: \"16b9fd20-840b-47a2-a977-2b67f535c831\") "
Sep 11 17:50:00.271572 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.271540    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d27723b-8c92-45d2-ab77-80ecb372d5ef-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8d27723b-8c92-45d2-ab77-80ecb372d5ef" (UID: "8d27723b-8c92-45d2-ab77-80ecb372d5ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:50:00.271737 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.271605    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b9fd20-840b-47a2-a977-2b67f535c831-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "16b9fd20-840b-47a2-a977-2b67f535c831" (UID: "16b9fd20-840b-47a2-a977-2b67f535c831"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:50:00.274098 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.274070    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b9fd20-840b-47a2-a977-2b67f535c831-kube-api-access-rhmj4" (OuterVolumeSpecName: "kube-api-access-rhmj4") pod "16b9fd20-840b-47a2-a977-2b67f535c831" (UID: "16b9fd20-840b-47a2-a977-2b67f535c831"). InnerVolumeSpecName "kube-api-access-rhmj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:50:00.274219 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.274164    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d27723b-8c92-45d2-ab77-80ecb372d5ef-kube-api-access-dp4z7" (OuterVolumeSpecName: "kube-api-access-dp4z7") pod "8d27723b-8c92-45d2-ab77-80ecb372d5ef" (UID: "8d27723b-8c92-45d2-ab77-80ecb372d5ef"). InnerVolumeSpecName "kube-api-access-dp4z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:50:00.372298 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.372261    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp4z7\" (UniqueName: \"kubernetes.io/projected/8d27723b-8c92-45d2-ab77-80ecb372d5ef-kube-api-access-dp4z7\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:50:00.372298 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.372292    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhmj4\" (UniqueName: \"kubernetes.io/projected/16b9fd20-840b-47a2-a977-2b67f535c831-kube-api-access-rhmj4\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:50:00.372298 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.372300    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d27723b-8c92-45d2-ab77-80ecb372d5ef-plugins-config-volume\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:50:00.372298 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.372310    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b9fd20-840b-47a2-a977-2b67f535c831-plugins-config-volume\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:50:00.639829 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.639790    2560 generic.go:356] "Generic (PLEG): container finished" podID="8d27723b-8c92-45d2-ab77-80ecb372d5ef" containerID="96c8a74b412bc828e0856a745e1d4e80a43a041e365f1fe2f298f6ef2bd35eb7" exitCode=0
Sep 11 17:50:00.640020 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.639837    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct"
Sep 11 17:50:00.640020 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.639877    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct" event={"ID":"8d27723b-8c92-45d2-ab77-80ecb372d5ef","Type":"ContainerDied","Data":"96c8a74b412bc828e0856a745e1d4e80a43a041e365f1fe2f298f6ef2bd35eb7"}
Sep 11 17:50:00.640020 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.639977    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct" event={"ID":"8d27723b-8c92-45d2-ab77-80ecb372d5ef","Type":"ContainerDied","Data":"973a6ac62fad0649ee58d7b26a0ab3a8fdfc2e37b5ec04438c48b1b418a8760b"}
Sep 11 17:50:00.640020 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.640007    2560 scope.go:117] "RemoveContainer" containerID="96c8a74b412bc828e0856a745e1d4e80a43a041e365f1fe2f298f6ef2bd35eb7"
Sep 11 17:50:00.641411 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.641386    2560 generic.go:356] "Generic (PLEG): container finished" podID="16b9fd20-840b-47a2-a977-2b67f535c831" containerID="35e4e37aab1e606acbae1ce93f61a84f0f5c72563ec91e05f45e193fa47a275b" exitCode=0
Sep 11 17:50:00.641540 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.641418    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd"
Sep 11 17:50:00.641540 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.641428    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd" event={"ID":"16b9fd20-840b-47a2-a977-2b67f535c831","Type":"ContainerDied","Data":"35e4e37aab1e606acbae1ce93f61a84f0f5c72563ec91e05f45e193fa47a275b"}
Sep 11 17:50:00.641540 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.641467    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd" event={"ID":"16b9fd20-840b-47a2-a977-2b67f535c831","Type":"ContainerDied","Data":"caa034113ca4a64c4c3b5eef59b608a1f4326bb94d8c5e4149263530042edc77"}
Sep 11 17:50:00.649067 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.648728    2560 scope.go:117] "RemoveContainer" containerID="96c8a74b412bc828e0856a745e1d4e80a43a041e365f1fe2f298f6ef2bd35eb7"
Sep 11 17:50:00.649155 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:50:00.649115    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96c8a74b412bc828e0856a745e1d4e80a43a041e365f1fe2f298f6ef2bd35eb7\": container with ID starting with 96c8a74b412bc828e0856a745e1d4e80a43a041e365f1fe2f298f6ef2bd35eb7 not found: ID does not exist" containerID="96c8a74b412bc828e0856a745e1d4e80a43a041e365f1fe2f298f6ef2bd35eb7"
Sep 11 17:50:00.649155 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.649140    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96c8a74b412bc828e0856a745e1d4e80a43a041e365f1fe2f298f6ef2bd35eb7"} err="failed to get container status \"96c8a74b412bc828e0856a745e1d4e80a43a041e365f1fe2f298f6ef2bd35eb7\": rpc error: code = NotFound desc = could not find container \"96c8a74b412bc828e0856a745e1d4e80a43a041e365f1fe2f298f6ef2bd35eb7\": container with ID starting with 96c8a74b412bc828e0856a745e1d4e80a43a041e365f1fe2f298f6ef2bd35eb7 not found: ID does not exist"
Sep 11 17:50:00.649268 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.649157    2560 scope.go:117] "RemoveContainer" containerID="35e4e37aab1e606acbae1ce93f61a84f0f5c72563ec91e05f45e193fa47a275b"
Sep 11 17:50:00.657078 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.657061    2560 scope.go:117] "RemoveContainer" containerID="35e4e37aab1e606acbae1ce93f61a84f0f5c72563ec91e05f45e193fa47a275b"
Sep 11 17:50:00.657321 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:50:00.657306    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35e4e37aab1e606acbae1ce93f61a84f0f5c72563ec91e05f45e193fa47a275b\": container with ID starting with 35e4e37aab1e606acbae1ce93f61a84f0f5c72563ec91e05f45e193fa47a275b not found: ID does not exist" containerID="35e4e37aab1e606acbae1ce93f61a84f0f5c72563ec91e05f45e193fa47a275b"
Sep 11 17:50:00.657360 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.657329    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e4e37aab1e606acbae1ce93f61a84f0f5c72563ec91e05f45e193fa47a275b"} err="failed to get container status \"35e4e37aab1e606acbae1ce93f61a84f0f5c72563ec91e05f45e193fa47a275b\": rpc error: code = NotFound desc = could not find container \"35e4e37aab1e606acbae1ce93f61a84f0f5c72563ec91e05f45e193fa47a275b\": container with ID starting with 35e4e37aab1e606acbae1ce93f61a84f0f5c72563ec91e05f45e193fa47a275b not found: ID does not exist"
Sep 11 17:50:00.666159 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.666137    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd"]
Sep 11 17:50:00.670366 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.670344    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4c36f4-bf87fcc4-997jd"]
Sep 11 17:50:00.680920 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.680872    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct"]
Sep 11 17:50:00.686048 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.686026    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4c36f4-6f7979b48c-56hct"]
Sep 11 17:50:00.715341 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.715314    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16b9fd20-840b-47a2-a977-2b67f535c831" path="/var/lib/kubelet/pods/16b9fd20-840b-47a2-a977-2b67f535c831/volumes"
Sep 11 17:50:00.715712 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:00.715698    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d27723b-8c92-45d2-ab77-80ecb372d5ef" path="/var/lib/kubelet/pods/8d27723b-8c92-45d2-ab77-80ecb372d5ef/volumes"
Sep 11 17:50:17.933958 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:17.933922    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2"]
Sep 11 17:50:17.934533 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:17.934117    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2" podUID="8e9776f8-d9e6-4fda-a3ab-df684c35a71e" containerName="maas-api" containerID="cri-o://db603b63f6b32c0f2003cd1b7dff037ef6726be302dbcbb725cac7d0d7cd3a09" gracePeriod=30
Sep 11 17:50:18.053658 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.053632    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq"]
Sep 11 17:50:18.053879 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.053859    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq" podUID="1cc5f681-b4a2-438e-9460-71f081ba5c73" containerName="payload-processing" containerID="cri-o://d0379ae05b21843938729073b178b46b97d0990f70901376637baa498eb28a63" gracePeriod=30
Sep 11 17:50:18.195603 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.195534    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2"
Sep 11 17:50:18.307159 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.307132    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq"
Sep 11 17:50:18.342787 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.342759    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e9776f8-d9e6-4fda-a3ab-df684c35a71e-maas-api-tls\") pod \"8e9776f8-d9e6-4fda-a3ab-df684c35a71e\" (UID: \"8e9776f8-d9e6-4fda-a3ab-df684c35a71e\") "
Sep 11 17:50:18.342998 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.342853    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e9776f8-d9e6-4fda-a3ab-df684c35a71e-openshift-service-ca\") pod \"8e9776f8-d9e6-4fda-a3ab-df684c35a71e\" (UID: \"8e9776f8-d9e6-4fda-a3ab-df684c35a71e\") "
Sep 11 17:50:18.342998 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.342961    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e9776f8-d9e6-4fda-a3ab-df684c35a71e-metrics-certs\") pod \"8e9776f8-d9e6-4fda-a3ab-df684c35a71e\" (UID: \"8e9776f8-d9e6-4fda-a3ab-df684c35a71e\") "
Sep 11 17:50:18.343121 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.343052    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rg25\" (UniqueName: \"kubernetes.io/projected/8e9776f8-d9e6-4fda-a3ab-df684c35a71e-kube-api-access-2rg25\") pod \"8e9776f8-d9e6-4fda-a3ab-df684c35a71e\" (UID: \"8e9776f8-d9e6-4fda-a3ab-df684c35a71e\") "
Sep 11 17:50:18.343212 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.343192    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e9776f8-d9e6-4fda-a3ab-df684c35a71e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8e9776f8-d9e6-4fda-a3ab-df684c35a71e" (UID: "8e9776f8-d9e6-4fda-a3ab-df684c35a71e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:50:18.343411 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.343382    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e9776f8-d9e6-4fda-a3ab-df684c35a71e-openshift-service-ca\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:50:18.345022 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.345001    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e9776f8-d9e6-4fda-a3ab-df684c35a71e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8e9776f8-d9e6-4fda-a3ab-df684c35a71e" (UID: "8e9776f8-d9e6-4fda-a3ab-df684c35a71e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:50:18.345104 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.345021    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e9776f8-d9e6-4fda-a3ab-df684c35a71e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8e9776f8-d9e6-4fda-a3ab-df684c35a71e" (UID: "8e9776f8-d9e6-4fda-a3ab-df684c35a71e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:50:18.345457 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.345441    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e9776f8-d9e6-4fda-a3ab-df684c35a71e-kube-api-access-2rg25" (OuterVolumeSpecName: "kube-api-access-2rg25") pod "8e9776f8-d9e6-4fda-a3ab-df684c35a71e" (UID: "8e9776f8-d9e6-4fda-a3ab-df684c35a71e"). InnerVolumeSpecName "kube-api-access-2rg25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:50:18.443875 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.443844    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc5f681-b4a2-438e-9460-71f081ba5c73-plugins-config-volume\") pod \"1cc5f681-b4a2-438e-9460-71f081ba5c73\" (UID: \"1cc5f681-b4a2-438e-9460-71f081ba5c73\") "
Sep 11 17:50:18.444187 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.443941    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c72tf\" (UniqueName: \"kubernetes.io/projected/1cc5f681-b4a2-438e-9460-71f081ba5c73-kube-api-access-c72tf\") pod \"1cc5f681-b4a2-438e-9460-71f081ba5c73\" (UID: \"1cc5f681-b4a2-438e-9460-71f081ba5c73\") "
Sep 11 17:50:18.444284 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.444260    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cc5f681-b4a2-438e-9460-71f081ba5c73-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1cc5f681-b4a2-438e-9460-71f081ba5c73" (UID: "1cc5f681-b4a2-438e-9460-71f081ba5c73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:50:18.444352 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.444284    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e9776f8-d9e6-4fda-a3ab-df684c35a71e-metrics-certs\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:50:18.444352 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.444304    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rg25\" (UniqueName: \"kubernetes.io/projected/8e9776f8-d9e6-4fda-a3ab-df684c35a71e-kube-api-access-2rg25\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:50:18.444352 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.444318    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e9776f8-d9e6-4fda-a3ab-df684c35a71e-maas-api-tls\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:50:18.446396 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.446335    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc5f681-b4a2-438e-9460-71f081ba5c73-kube-api-access-c72tf" (OuterVolumeSpecName: "kube-api-access-c72tf") pod "1cc5f681-b4a2-438e-9460-71f081ba5c73" (UID: "1cc5f681-b4a2-438e-9460-71f081ba5c73"). InnerVolumeSpecName "kube-api-access-c72tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:50:18.545406 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.545364    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c72tf\" (UniqueName: \"kubernetes.io/projected/1cc5f681-b4a2-438e-9460-71f081ba5c73-kube-api-access-c72tf\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:50:18.545406 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.545397    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc5f681-b4a2-438e-9460-71f081ba5c73-plugins-config-volume\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:50:18.738445 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.738416    2560 generic.go:356] "Generic (PLEG): container finished" podID="8e9776f8-d9e6-4fda-a3ab-df684c35a71e" containerID="db603b63f6b32c0f2003cd1b7dff037ef6726be302dbcbb725cac7d0d7cd3a09" exitCode=0
Sep 11 17:50:18.738542 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.738472    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2"
Sep 11 17:50:18.738542 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.738500    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2" event={"ID":"8e9776f8-d9e6-4fda-a3ab-df684c35a71e","Type":"ContainerDied","Data":"db603b63f6b32c0f2003cd1b7dff037ef6726be302dbcbb725cac7d0d7cd3a09"}
Sep 11 17:50:18.738633 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.738544    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2" event={"ID":"8e9776f8-d9e6-4fda-a3ab-df684c35a71e","Type":"ContainerDied","Data":"d78114ddea4f9b73971e3b3687831c28cf97c08da5a04d01a02000983f674d67"}
Sep 11 17:50:18.738633 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.738570    2560 scope.go:117] "RemoveContainer" containerID="db603b63f6b32c0f2003cd1b7dff037ef6726be302dbcbb725cac7d0d7cd3a09"
Sep 11 17:50:18.740051 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.740027    2560 generic.go:356] "Generic (PLEG): container finished" podID="1cc5f681-b4a2-438e-9460-71f081ba5c73" containerID="d0379ae05b21843938729073b178b46b97d0990f70901376637baa498eb28a63" exitCode=0
Sep 11 17:50:18.740203 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.740055    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq" event={"ID":"1cc5f681-b4a2-438e-9460-71f081ba5c73","Type":"ContainerDied","Data":"d0379ae05b21843938729073b178b46b97d0990f70901376637baa498eb28a63"}
Sep 11 17:50:18.740203 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.740081    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq"
Sep 11 17:50:18.740203 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.740103    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq" event={"ID":"1cc5f681-b4a2-438e-9460-71f081ba5c73","Type":"ContainerDied","Data":"71405a34792dc98e38f870dfc6b570ad6e4592ac3f870dcb6ce3da593a7ca6da"}
Sep 11 17:50:18.750555 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.750526    2560 scope.go:117] "RemoveContainer" containerID="db603b63f6b32c0f2003cd1b7dff037ef6726be302dbcbb725cac7d0d7cd3a09"
Sep 11 17:50:18.750833 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:50:18.750814    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db603b63f6b32c0f2003cd1b7dff037ef6726be302dbcbb725cac7d0d7cd3a09\": container with ID starting with db603b63f6b32c0f2003cd1b7dff037ef6726be302dbcbb725cac7d0d7cd3a09 not found: ID does not exist" containerID="db603b63f6b32c0f2003cd1b7dff037ef6726be302dbcbb725cac7d0d7cd3a09"
Sep 11 17:50:18.750890 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.750841    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db603b63f6b32c0f2003cd1b7dff037ef6726be302dbcbb725cac7d0d7cd3a09"} err="failed to get container status \"db603b63f6b32c0f2003cd1b7dff037ef6726be302dbcbb725cac7d0d7cd3a09\": rpc error: code = NotFound desc = could not find container \"db603b63f6b32c0f2003cd1b7dff037ef6726be302dbcbb725cac7d0d7cd3a09\": container with ID starting with db603b63f6b32c0f2003cd1b7dff037ef6726be302dbcbb725cac7d0d7cd3a09 not found: ID does not exist"
Sep 11 17:50:18.750890 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.750858    2560 scope.go:117] "RemoveContainer" containerID="d0379ae05b21843938729073b178b46b97d0990f70901376637baa498eb28a63"
Sep 11 17:50:18.758594 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.758579    2560 scope.go:117] "RemoveContainer" containerID="d0379ae05b21843938729073b178b46b97d0990f70901376637baa498eb28a63"
Sep 11 17:50:18.758855 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:50:18.758836    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0379ae05b21843938729073b178b46b97d0990f70901376637baa498eb28a63\": container with ID starting with d0379ae05b21843938729073b178b46b97d0990f70901376637baa498eb28a63 not found: ID does not exist" containerID="d0379ae05b21843938729073b178b46b97d0990f70901376637baa498eb28a63"
Sep 11 17:50:18.758926 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.758861    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0379ae05b21843938729073b178b46b97d0990f70901376637baa498eb28a63"} err="failed to get container status \"d0379ae05b21843938729073b178b46b97d0990f70901376637baa498eb28a63\": rpc error: code = NotFound desc = could not find container \"d0379ae05b21843938729073b178b46b97d0990f70901376637baa498eb28a63\": container with ID starting with d0379ae05b21843938729073b178b46b97d0990f70901376637baa498eb28a63 not found: ID does not exist"
Sep 11 17:50:18.777566 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.777535    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq"]
Sep 11 17:50:18.790769 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.790740    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-db95fd-96dccf65c-hvkjq"]
Sep 11 17:50:18.808786 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.808757    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2"]
Sep 11 17:50:18.813887 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:18.813863    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db95fd-689b756f5d-552h2"]
Sep 11 17:50:20.715466 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:20.715435    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cc5f681-b4a2-438e-9460-71f081ba5c73" path="/var/lib/kubelet/pods/1cc5f681-b4a2-438e-9460-71f081ba5c73/volumes"
Sep 11 17:50:20.715857 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:20.715844    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e9776f8-d9e6-4fda-a3ab-df684c35a71e" path="/var/lib/kubelet/pods/8e9776f8-d9e6-4fda-a3ab-df684c35a71e/volumes"
Sep 11 17:50:28.228920 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:28.228821    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6ddbd79556-qd9nn"]
Sep 11 17:50:28.229290 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:28.229094    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6ddbd79556-qd9nn" podUID="70ff87e0-8bf9-4f4a-b32b-242baf27ceaa" containerName="manager" containerID="cri-o://c4ae291c4090858ff53a06d4190170cec6c8fee52819addf85f516e88e3c0df7" gracePeriod=30
Sep 11 17:50:28.472539 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:28.472516    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ddbd79556-qd9nn"
Sep 11 17:50:28.641370 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:28.641334    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70ff87e0-8bf9-4f4a-b32b-242baf27ceaa-metrics-certs\") pod \"70ff87e0-8bf9-4f4a-b32b-242baf27ceaa\" (UID: \"70ff87e0-8bf9-4f4a-b32b-242baf27ceaa\") "
Sep 11 17:50:28.641370 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:28.641377    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70ff87e0-8bf9-4f4a-b32b-242baf27ceaa-webhook-cert\") pod \"70ff87e0-8bf9-4f4a-b32b-242baf27ceaa\" (UID: \"70ff87e0-8bf9-4f4a-b32b-242baf27ceaa\") "
Sep 11 17:50:28.641644 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:28.641469    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5p5r\" (UniqueName: \"kubernetes.io/projected/70ff87e0-8bf9-4f4a-b32b-242baf27ceaa-kube-api-access-b5p5r\") pod \"70ff87e0-8bf9-4f4a-b32b-242baf27ceaa\" (UID: \"70ff87e0-8bf9-4f4a-b32b-242baf27ceaa\") "
Sep 11 17:50:28.643496 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:28.643468    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70ff87e0-8bf9-4f4a-b32b-242baf27ceaa-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "70ff87e0-8bf9-4f4a-b32b-242baf27ceaa" (UID: "70ff87e0-8bf9-4f4a-b32b-242baf27ceaa"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:50:28.643612 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:28.643531    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ff87e0-8bf9-4f4a-b32b-242baf27ceaa-kube-api-access-b5p5r" (OuterVolumeSpecName: "kube-api-access-b5p5r") pod "70ff87e0-8bf9-4f4a-b32b-242baf27ceaa" (UID: "70ff87e0-8bf9-4f4a-b32b-242baf27ceaa"). InnerVolumeSpecName "kube-api-access-b5p5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:50:28.643612 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:28.643564    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70ff87e0-8bf9-4f4a-b32b-242baf27ceaa-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "70ff87e0-8bf9-4f4a-b32b-242baf27ceaa" (UID: "70ff87e0-8bf9-4f4a-b32b-242baf27ceaa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:50:28.742736 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:28.742700    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5p5r\" (UniqueName: \"kubernetes.io/projected/70ff87e0-8bf9-4f4a-b32b-242baf27ceaa-kube-api-access-b5p5r\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:50:28.742736 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:28.742730    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70ff87e0-8bf9-4f4a-b32b-242baf27ceaa-metrics-certs\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:50:28.742736 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:28.742739    2560 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70ff87e0-8bf9-4f4a-b32b-242baf27ceaa-webhook-cert\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:50:28.791016 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:28.790979    2560 generic.go:356] "Generic (PLEG): container finished" podID="70ff87e0-8bf9-4f4a-b32b-242baf27ceaa" containerID="c4ae291c4090858ff53a06d4190170cec6c8fee52819addf85f516e88e3c0df7" exitCode=0
Sep 11 17:50:28.791152 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:28.791025    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ddbd79556-qd9nn"
Sep 11 17:50:28.791152 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:28.791036    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ddbd79556-qd9nn" event={"ID":"70ff87e0-8bf9-4f4a-b32b-242baf27ceaa","Type":"ContainerDied","Data":"c4ae291c4090858ff53a06d4190170cec6c8fee52819addf85f516e88e3c0df7"}
Sep 11 17:50:28.791152 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:28.791078    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ddbd79556-qd9nn" event={"ID":"70ff87e0-8bf9-4f4a-b32b-242baf27ceaa","Type":"ContainerDied","Data":"c21cd95163122036f4c5dff3cf7978f1f269b39f7fc4ca87494f0f446f6dc586"}
Sep 11 17:50:28.791152 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:28.791099    2560 scope.go:117] "RemoveContainer" containerID="c4ae291c4090858ff53a06d4190170cec6c8fee52819addf85f516e88e3c0df7"
Sep 11 17:50:28.800091 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:28.800068    2560 scope.go:117] "RemoveContainer" containerID="c4ae291c4090858ff53a06d4190170cec6c8fee52819addf85f516e88e3c0df7"
Sep 11 17:50:28.800357 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:50:28.800338    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4ae291c4090858ff53a06d4190170cec6c8fee52819addf85f516e88e3c0df7\": container with ID starting with c4ae291c4090858ff53a06d4190170cec6c8fee52819addf85f516e88e3c0df7 not found: ID does not exist" containerID="c4ae291c4090858ff53a06d4190170cec6c8fee52819addf85f516e88e3c0df7"
Sep 11 17:50:28.800404 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:28.800365    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4ae291c4090858ff53a06d4190170cec6c8fee52819addf85f516e88e3c0df7"} err="failed to get container status \"c4ae291c4090858ff53a06d4190170cec6c8fee52819addf85f516e88e3c0df7\": rpc error: code = NotFound desc = could not find container \"c4ae291c4090858ff53a06d4190170cec6c8fee52819addf85f516e88e3c0df7\": container with ID starting with c4ae291c4090858ff53a06d4190170cec6c8fee52819addf85f516e88e3c0df7 not found: ID does not exist"
Sep 11 17:50:28.824051 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:28.824020    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6ddbd79556-qd9nn"]
Sep 11 17:50:28.832892 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:28.832869    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6ddbd79556-qd9nn"]
Sep 11 17:50:30.715053 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:30.715017    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70ff87e0-8bf9-4f4a-b32b-242baf27ceaa" path="/var/lib/kubelet/pods/70ff87e0-8bf9-4f4a-b32b-242baf27ceaa/volumes"
Sep 11 17:50:32.736404 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:32.736374    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6fcjs_477cac01-ab9e-4441-b07d-b7e53401c616/console-operator/1.log"
Sep 11 17:50:32.743612 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:50:32.743590    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6fcjs_477cac01-ab9e-4441-b07d-b7e53401c616/console-operator/1.log"
Sep 11 17:51:29.672762 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.672728    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6ddbd79556-l75pc"]
Sep 11 17:51:29.673214 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.673129    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec058a31-1d64-4538-9ba5-4859c9c99223" containerName="maas-api"
Sep 11 17:51:29.673214 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.673143    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec058a31-1d64-4538-9ba5-4859c9c99223" containerName="maas-api"
Sep 11 17:51:29.673214 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.673159    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16b9fd20-840b-47a2-a977-2b67f535c831" containerName="payload-pre-processing"
Sep 11 17:51:29.673214 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.673165    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="16b9fd20-840b-47a2-a977-2b67f535c831" containerName="payload-pre-processing"
Sep 11 17:51:29.673214 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.673174    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e9776f8-d9e6-4fda-a3ab-df684c35a71e" containerName="maas-api"
Sep 11 17:51:29.673214 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.673180    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e9776f8-d9e6-4fda-a3ab-df684c35a71e" containerName="maas-api"
Sep 11 17:51:29.673214 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.673195    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d27723b-8c92-45d2-ab77-80ecb372d5ef" containerName="payload-processing"
Sep 11 17:51:29.673214 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.673201    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d27723b-8c92-45d2-ab77-80ecb372d5ef" containerName="payload-processing"
Sep 11 17:51:29.673214 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.673212    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cc5f681-b4a2-438e-9460-71f081ba5c73" containerName="payload-processing"
Sep 11 17:51:29.673214 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.673217    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cc5f681-b4a2-438e-9460-71f081ba5c73" containerName="payload-processing"
Sep 11 17:51:29.673539 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.673225    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f888c936-827c-4871-96bc-fbcec11ee251" containerName="payload-pre-processing"
Sep 11 17:51:29.673539 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.673231    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f888c936-827c-4871-96bc-fbcec11ee251" containerName="payload-pre-processing"
Sep 11 17:51:29.673539 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.673238    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26ff5d0a-1c37-4667-9277-c5f6901ba8ef" containerName="maas-api"
Sep 11 17:51:29.673539 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.673243    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="26ff5d0a-1c37-4667-9277-c5f6901ba8ef" containerName="maas-api"
Sep 11 17:51:29.673539 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.673250    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ceaf894-7532-4183-b2e2-1a98e7698708" containerName="maas-api"
Sep 11 17:51:29.673539 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.673255    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ceaf894-7532-4183-b2e2-1a98e7698708" containerName="maas-api"
Sep 11 17:51:29.673539 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.673264    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70ff87e0-8bf9-4f4a-b32b-242baf27ceaa" containerName="manager"
Sep 11 17:51:29.673539 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.673272    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="70ff87e0-8bf9-4f4a-b32b-242baf27ceaa" containerName="manager"
Sep 11 17:51:29.673539 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.673328    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e9776f8-d9e6-4fda-a3ab-df684c35a71e" containerName="maas-api"
Sep 11 17:51:29.673539 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.673339    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f888c936-827c-4871-96bc-fbcec11ee251" containerName="payload-pre-processing"
Sep 11 17:51:29.673539 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.673348    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="26ff5d0a-1c37-4667-9277-c5f6901ba8ef" containerName="maas-api"
Sep 11 17:51:29.673539 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.673356    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="70ff87e0-8bf9-4f4a-b32b-242baf27ceaa" containerName="manager"
Sep 11 17:51:29.673539 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.673362    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="16b9fd20-840b-47a2-a977-2b67f535c831" containerName="payload-pre-processing"
Sep 11 17:51:29.673539 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.673366    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ceaf894-7532-4183-b2e2-1a98e7698708" containerName="maas-api"
Sep 11 17:51:29.673539 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.673372    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cc5f681-b4a2-438e-9460-71f081ba5c73" containerName="payload-processing"
Sep 11 17:51:29.673539 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.673377    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec058a31-1d64-4538-9ba5-4859c9c99223" containerName="maas-api"
Sep 11 17:51:29.673539 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.673383    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d27723b-8c92-45d2-ab77-80ecb372d5ef" containerName="payload-processing"
Sep 11 17:51:29.676440 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.676413    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ddbd79556-l75pc"
Sep 11 17:51:29.679144 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.679123    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 17:51:29.679238 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.679126    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 17:51:29.680216 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.680197    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 17:51:29.680287 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.680228    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-r7knj\""
Sep 11 17:51:29.684889 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.684872    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4dd86d2f-e24e-41d4-8497-c93fe6c8936d-webhook-cert\") pod \"maas-controller-6ddbd79556-l75pc\" (UID: \"4dd86d2f-e24e-41d4-8497-c93fe6c8936d\") " pod="opendatahub/maas-controller-6ddbd79556-l75pc"
Sep 11 17:51:29.685007 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.684992    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvz9j\" (UniqueName: \"kubernetes.io/projected/4dd86d2f-e24e-41d4-8497-c93fe6c8936d-kube-api-access-mvz9j\") pod \"maas-controller-6ddbd79556-l75pc\" (UID: \"4dd86d2f-e24e-41d4-8497-c93fe6c8936d\") " pod="opendatahub/maas-controller-6ddbd79556-l75pc"
Sep 11 17:51:29.685049 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.685016    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd86d2f-e24e-41d4-8497-c93fe6c8936d-metrics-certs\") pod \"maas-controller-6ddbd79556-l75pc\" (UID: \"4dd86d2f-e24e-41d4-8497-c93fe6c8936d\") " pod="opendatahub/maas-controller-6ddbd79556-l75pc"
Sep 11 17:51:29.690693 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.690663    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6ddbd79556-l75pc"]
Sep 11 17:51:29.785713 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.785672    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4dd86d2f-e24e-41d4-8497-c93fe6c8936d-webhook-cert\") pod \"maas-controller-6ddbd79556-l75pc\" (UID: \"4dd86d2f-e24e-41d4-8497-c93fe6c8936d\") " pod="opendatahub/maas-controller-6ddbd79556-l75pc"
Sep 11 17:51:29.785974 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.785795    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvz9j\" (UniqueName: \"kubernetes.io/projected/4dd86d2f-e24e-41d4-8497-c93fe6c8936d-kube-api-access-mvz9j\") pod \"maas-controller-6ddbd79556-l75pc\" (UID: \"4dd86d2f-e24e-41d4-8497-c93fe6c8936d\") " pod="opendatahub/maas-controller-6ddbd79556-l75pc"
Sep 11 17:51:29.785974 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.785827    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd86d2f-e24e-41d4-8497-c93fe6c8936d-metrics-certs\") pod \"maas-controller-6ddbd79556-l75pc\" (UID: \"4dd86d2f-e24e-41d4-8497-c93fe6c8936d\") " pod="opendatahub/maas-controller-6ddbd79556-l75pc"
Sep 11 17:51:29.788100 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.788074    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4dd86d2f-e24e-41d4-8497-c93fe6c8936d-webhook-cert\") pod \"maas-controller-6ddbd79556-l75pc\" (UID: \"4dd86d2f-e24e-41d4-8497-c93fe6c8936d\") " pod="opendatahub/maas-controller-6ddbd79556-l75pc"
Sep 11 17:51:29.788249 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.788230    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd86d2f-e24e-41d4-8497-c93fe6c8936d-metrics-certs\") pod \"maas-controller-6ddbd79556-l75pc\" (UID: \"4dd86d2f-e24e-41d4-8497-c93fe6c8936d\") " pod="opendatahub/maas-controller-6ddbd79556-l75pc"
Sep 11 17:51:29.794689 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.794668    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvz9j\" (UniqueName: \"kubernetes.io/projected/4dd86d2f-e24e-41d4-8497-c93fe6c8936d-kube-api-access-mvz9j\") pod \"maas-controller-6ddbd79556-l75pc\" (UID: \"4dd86d2f-e24e-41d4-8497-c93fe6c8936d\") " pod="opendatahub/maas-controller-6ddbd79556-l75pc"
Sep 11 17:51:29.987171 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:29.987079    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ddbd79556-l75pc"
Sep 11 17:51:30.262111 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:30.262086    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6ddbd79556-l75pc"]
Sep 11 17:51:30.264002 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:51:30.263971    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd86d2f_e24e_41d4_8497_c93fe6c8936d.slice/crio-ec5c25123214a8f561a9085f49ad1c522432fc6549c212c7e116509341d1afd7 WatchSource:0}: Error finding container ec5c25123214a8f561a9085f49ad1c522432fc6549c212c7e116509341d1afd7: Status 404 returned error can't find the container with id ec5c25123214a8f561a9085f49ad1c522432fc6549c212c7e116509341d1afd7
Sep 11 17:51:31.098210 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:31.098175    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ddbd79556-l75pc" event={"ID":"4dd86d2f-e24e-41d4-8497-c93fe6c8936d","Type":"ContainerStarted","Data":"d468cc65edf292ad6c4b7bbdb24ce269295bb8655a0fa36d17e590500abb92be"}
Sep 11 17:51:31.098700 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:31.098225    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ddbd79556-l75pc" event={"ID":"4dd86d2f-e24e-41d4-8497-c93fe6c8936d","Type":"ContainerStarted","Data":"ec5c25123214a8f561a9085f49ad1c522432fc6549c212c7e116509341d1afd7"}
Sep 11 17:51:31.098700 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:31.098305    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6ddbd79556-l75pc"
Sep 11 17:51:31.118484 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:31.118427    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6ddbd79556-l75pc" podStartSLOduration=1.699949879 podStartE2EDuration="2.118411907s" totalImagesPullingTime="418.462028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:51:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:51:31.117159563 +0000 UTC m=+1858.921331984" watchObservedRunningTime="2026-09-11 17:51:31.118411907 +0000 UTC m=+1858.922584334"
Sep 11 17:51:42.104855 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:42.104815    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6ddbd79556-l75pc"
Sep 11 17:51:53.599658 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:53.599617    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6ddbd79556-l75pc"]
Sep 11 17:51:53.600239 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:53.599861    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6ddbd79556-l75pc" podUID="4dd86d2f-e24e-41d4-8497-c93fe6c8936d" containerName="manager" containerID="cri-o://d468cc65edf292ad6c4b7bbdb24ce269295bb8655a0fa36d17e590500abb92be" gracePeriod=30
Sep 11 17:51:53.847549 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:53.847521    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ddbd79556-l75pc"
Sep 11 17:51:54.010997 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:54.010877    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvz9j\" (UniqueName: \"kubernetes.io/projected/4dd86d2f-e24e-41d4-8497-c93fe6c8936d-kube-api-access-mvz9j\") pod \"4dd86d2f-e24e-41d4-8497-c93fe6c8936d\" (UID: \"4dd86d2f-e24e-41d4-8497-c93fe6c8936d\") "
Sep 11 17:51:54.010997 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:54.010992    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd86d2f-e24e-41d4-8497-c93fe6c8936d-metrics-certs\") pod \"4dd86d2f-e24e-41d4-8497-c93fe6c8936d\" (UID: \"4dd86d2f-e24e-41d4-8497-c93fe6c8936d\") "
Sep 11 17:51:54.011240 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:54.011059    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4dd86d2f-e24e-41d4-8497-c93fe6c8936d-webhook-cert\") pod \"4dd86d2f-e24e-41d4-8497-c93fe6c8936d\" (UID: \"4dd86d2f-e24e-41d4-8497-c93fe6c8936d\") "
Sep 11 17:51:54.013102 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:54.013073    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dd86d2f-e24e-41d4-8497-c93fe6c8936d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "4dd86d2f-e24e-41d4-8497-c93fe6c8936d" (UID: "4dd86d2f-e24e-41d4-8497-c93fe6c8936d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:51:54.013202 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:54.013110    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd86d2f-e24e-41d4-8497-c93fe6c8936d-kube-api-access-mvz9j" (OuterVolumeSpecName: "kube-api-access-mvz9j") pod "4dd86d2f-e24e-41d4-8497-c93fe6c8936d" (UID: "4dd86d2f-e24e-41d4-8497-c93fe6c8936d"). InnerVolumeSpecName "kube-api-access-mvz9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:51:54.013202 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:54.013120    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dd86d2f-e24e-41d4-8497-c93fe6c8936d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4dd86d2f-e24e-41d4-8497-c93fe6c8936d" (UID: "4dd86d2f-e24e-41d4-8497-c93fe6c8936d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:51:54.111825 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:54.111792    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd86d2f-e24e-41d4-8497-c93fe6c8936d-metrics-certs\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:51:54.111825 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:54.111822    2560 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4dd86d2f-e24e-41d4-8497-c93fe6c8936d-webhook-cert\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:51:54.112076 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:54.111835    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvz9j\" (UniqueName: \"kubernetes.io/projected/4dd86d2f-e24e-41d4-8497-c93fe6c8936d-kube-api-access-mvz9j\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:51:54.208240 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:54.208205    2560 generic.go:356] "Generic (PLEG): container finished" podID="4dd86d2f-e24e-41d4-8497-c93fe6c8936d" containerID="d468cc65edf292ad6c4b7bbdb24ce269295bb8655a0fa36d17e590500abb92be" exitCode=0
Sep 11 17:51:54.208420 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:54.208250    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ddbd79556-l75pc"
Sep 11 17:51:54.208420 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:54.208294    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ddbd79556-l75pc" event={"ID":"4dd86d2f-e24e-41d4-8497-c93fe6c8936d","Type":"ContainerDied","Data":"d468cc65edf292ad6c4b7bbdb24ce269295bb8655a0fa36d17e590500abb92be"}
Sep 11 17:51:54.208420 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:54.208345    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ddbd79556-l75pc" event={"ID":"4dd86d2f-e24e-41d4-8497-c93fe6c8936d","Type":"ContainerDied","Data":"ec5c25123214a8f561a9085f49ad1c522432fc6549c212c7e116509341d1afd7"}
Sep 11 17:51:54.208420 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:54.208369    2560 scope.go:117] "RemoveContainer" containerID="d468cc65edf292ad6c4b7bbdb24ce269295bb8655a0fa36d17e590500abb92be"
Sep 11 17:51:54.217775 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:54.217758    2560 scope.go:117] "RemoveContainer" containerID="d468cc65edf292ad6c4b7bbdb24ce269295bb8655a0fa36d17e590500abb92be"
Sep 11 17:51:54.218083 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:51:54.218062    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d468cc65edf292ad6c4b7bbdb24ce269295bb8655a0fa36d17e590500abb92be\": container with ID starting with d468cc65edf292ad6c4b7bbdb24ce269295bb8655a0fa36d17e590500abb92be not found: ID does not exist" containerID="d468cc65edf292ad6c4b7bbdb24ce269295bb8655a0fa36d17e590500abb92be"
Sep 11 17:51:54.218158 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:54.218090    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d468cc65edf292ad6c4b7bbdb24ce269295bb8655a0fa36d17e590500abb92be"} err="failed to get container status \"d468cc65edf292ad6c4b7bbdb24ce269295bb8655a0fa36d17e590500abb92be\": rpc error: code = NotFound desc = could not find container \"d468cc65edf292ad6c4b7bbdb24ce269295bb8655a0fa36d17e590500abb92be\": container with ID starting with d468cc65edf292ad6c4b7bbdb24ce269295bb8655a0fa36d17e590500abb92be not found: ID does not exist"
Sep 11 17:51:54.230735 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:54.230715    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6ddbd79556-l75pc"]
Sep 11 17:51:54.235087 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:54.235066    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6ddbd79556-l75pc"]
Sep 11 17:51:54.720977 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:54.720936    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dd86d2f-e24e-41d4-8497-c93fe6c8936d" path="/var/lib/kubelet/pods/4dd86d2f-e24e-41d4-8497-c93fe6c8936d/volumes"
Sep 11 17:51:55.021553 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:55.021469    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6ddbd79556-cgf4t"]
Sep 11 17:51:55.021843 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:55.021831    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dd86d2f-e24e-41d4-8497-c93fe6c8936d" containerName="manager"
Sep 11 17:51:55.021885 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:55.021845    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dd86d2f-e24e-41d4-8497-c93fe6c8936d" containerName="manager"
Sep 11 17:51:55.021950 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:55.021921    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dd86d2f-e24e-41d4-8497-c93fe6c8936d" containerName="manager"
Sep 11 17:51:55.026131 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:55.026113    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ddbd79556-cgf4t"
Sep 11 17:51:55.030553 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:55.030526    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 17:51:55.030686 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:55.030557    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 17:51:55.030686 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:55.030526    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-r7knj\""
Sep 11 17:51:55.030686 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:55.030534    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 17:51:55.040759 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:55.040734    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6ddbd79556-cgf4t"]
Sep 11 17:51:55.121802 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:55.121701    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/23afaf50-6113-4ebb-beac-e55bd9791a18-webhook-cert\") pod \"maas-controller-6ddbd79556-cgf4t\" (UID: \"23afaf50-6113-4ebb-beac-e55bd9791a18\") " pod="opendatahub/maas-controller-6ddbd79556-cgf4t"
Sep 11 17:51:55.121802 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:55.121753    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wbwd\" (UniqueName: \"kubernetes.io/projected/23afaf50-6113-4ebb-beac-e55bd9791a18-kube-api-access-7wbwd\") pod \"maas-controller-6ddbd79556-cgf4t\" (UID: \"23afaf50-6113-4ebb-beac-e55bd9791a18\") " pod="opendatahub/maas-controller-6ddbd79556-cgf4t"
Sep 11 17:51:55.122049 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:55.121855    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23afaf50-6113-4ebb-beac-e55bd9791a18-metrics-certs\") pod \"maas-controller-6ddbd79556-cgf4t\" (UID: \"23afaf50-6113-4ebb-beac-e55bd9791a18\") " pod="opendatahub/maas-controller-6ddbd79556-cgf4t"
Sep 11 17:51:55.222620 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:55.222586    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/23afaf50-6113-4ebb-beac-e55bd9791a18-webhook-cert\") pod \"maas-controller-6ddbd79556-cgf4t\" (UID: \"23afaf50-6113-4ebb-beac-e55bd9791a18\") " pod="opendatahub/maas-controller-6ddbd79556-cgf4t"
Sep 11 17:51:55.222820 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:55.222631    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wbwd\" (UniqueName: \"kubernetes.io/projected/23afaf50-6113-4ebb-beac-e55bd9791a18-kube-api-access-7wbwd\") pod \"maas-controller-6ddbd79556-cgf4t\" (UID: \"23afaf50-6113-4ebb-beac-e55bd9791a18\") " pod="opendatahub/maas-controller-6ddbd79556-cgf4t"
Sep 11 17:51:55.222820 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:55.222703    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23afaf50-6113-4ebb-beac-e55bd9791a18-metrics-certs\") pod \"maas-controller-6ddbd79556-cgf4t\" (UID: \"23afaf50-6113-4ebb-beac-e55bd9791a18\") " pod="opendatahub/maas-controller-6ddbd79556-cgf4t"
Sep 11 17:51:55.225101 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:55.225071    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/23afaf50-6113-4ebb-beac-e55bd9791a18-webhook-cert\") pod \"maas-controller-6ddbd79556-cgf4t\" (UID: \"23afaf50-6113-4ebb-beac-e55bd9791a18\") " pod="opendatahub/maas-controller-6ddbd79556-cgf4t"
Sep 11 17:51:55.225210 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:55.225126    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23afaf50-6113-4ebb-beac-e55bd9791a18-metrics-certs\") pod \"maas-controller-6ddbd79556-cgf4t\" (UID: \"23afaf50-6113-4ebb-beac-e55bd9791a18\") " pod="opendatahub/maas-controller-6ddbd79556-cgf4t"
Sep 11 17:51:55.232061 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:55.232036    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wbwd\" (UniqueName: \"kubernetes.io/projected/23afaf50-6113-4ebb-beac-e55bd9791a18-kube-api-access-7wbwd\") pod \"maas-controller-6ddbd79556-cgf4t\" (UID: \"23afaf50-6113-4ebb-beac-e55bd9791a18\") " pod="opendatahub/maas-controller-6ddbd79556-cgf4t"
Sep 11 17:51:55.336598 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:55.336564    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ddbd79556-cgf4t"
Sep 11 17:51:55.607631 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:55.607605    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6ddbd79556-cgf4t"]
Sep 11 17:51:55.609854 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:51:55.609826    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23afaf50_6113_4ebb_beac_e55bd9791a18.slice/crio-561ddf87278e261751e830c88e82f827d59226a536cf5f7e6717094bd51dbe7f WatchSource:0}: Error finding container 561ddf87278e261751e830c88e82f827d59226a536cf5f7e6717094bd51dbe7f: Status 404 returned error can't find the container with id 561ddf87278e261751e830c88e82f827d59226a536cf5f7e6717094bd51dbe7f
Sep 11 17:51:56.221289 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:56.221253    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ddbd79556-cgf4t" event={"ID":"23afaf50-6113-4ebb-beac-e55bd9791a18","Type":"ContainerStarted","Data":"ad3ce30732eaf98bcd7c61ee001439ac75cb8791ffd98c08d5d8d09e4f7e2331"}
Sep 11 17:51:56.221672 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:56.221303    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ddbd79556-cgf4t" event={"ID":"23afaf50-6113-4ebb-beac-e55bd9791a18","Type":"ContainerStarted","Data":"561ddf87278e261751e830c88e82f827d59226a536cf5f7e6717094bd51dbe7f"}
Sep 11 17:51:56.221672 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:56.221348    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6ddbd79556-cgf4t"
Sep 11 17:51:56.241443 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:51:56.241335    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6ddbd79556-cgf4t" podStartSLOduration=0.877570187 podStartE2EDuration="1.241317922s" totalImagesPullingTime="363.747735ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:51:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:51:56.239258642 +0000 UTC m=+1884.043431066" watchObservedRunningTime="2026-09-11 17:51:56.241317922 +0000 UTC m=+1884.045490340"
Sep 11 17:52:07.231864 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:52:07.231821    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6ddbd79556-cgf4t"
Sep 11 17:55:32.766655 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:55:32.766628    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6fcjs_477cac01-ab9e-4441-b07d-b7e53401c616/console-operator/1.log"
Sep 11 17:55:32.773581 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:55:32.773557    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6fcjs_477cac01-ab9e-4441-b07d-b7e53401c616/console-operator/1.log"
Sep 11 17:55:56.049965 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:55:56.049929    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-858bf7b9cb-pdjs2"]
Sep 11 17:55:56.053630 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:55:56.053608    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-858bf7b9cb-pdjs2"
Sep 11 17:55:56.065279 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:55:56.065246    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-858bf7b9cb-pdjs2"]
Sep 11 17:55:56.112192 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:55:56.112154    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd8c7ec3-e6ed-4e68-9c73-678475dd2df0-maas-api-tls\") pod \"maas-api-858bf7b9cb-pdjs2\" (UID: \"cd8c7ec3-e6ed-4e68-9c73-678475dd2df0\") " pod="odh-ai-gateway-infra/maas-api-858bf7b9cb-pdjs2"
Sep 11 17:55:56.112359 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:55:56.112216    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24kmh\" (UniqueName: \"kubernetes.io/projected/cd8c7ec3-e6ed-4e68-9c73-678475dd2df0-kube-api-access-24kmh\") pod \"maas-api-858bf7b9cb-pdjs2\" (UID: \"cd8c7ec3-e6ed-4e68-9c73-678475dd2df0\") " pod="odh-ai-gateway-infra/maas-api-858bf7b9cb-pdjs2"
Sep 11 17:55:56.112359 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:55:56.112317    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd8c7ec3-e6ed-4e68-9c73-678475dd2df0-metrics-certs\") pod \"maas-api-858bf7b9cb-pdjs2\" (UID: \"cd8c7ec3-e6ed-4e68-9c73-678475dd2df0\") " pod="odh-ai-gateway-infra/maas-api-858bf7b9cb-pdjs2"
Sep 11 17:55:56.112434 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:55:56.112366    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8c7ec3-e6ed-4e68-9c73-678475dd2df0-openshift-service-ca\") pod \"maas-api-858bf7b9cb-pdjs2\" (UID: \"cd8c7ec3-e6ed-4e68-9c73-678475dd2df0\") " pod="odh-ai-gateway-infra/maas-api-858bf7b9cb-pdjs2"
Sep 11 17:55:56.213836 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:55:56.213798    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24kmh\" (UniqueName: \"kubernetes.io/projected/cd8c7ec3-e6ed-4e68-9c73-678475dd2df0-kube-api-access-24kmh\") pod \"maas-api-858bf7b9cb-pdjs2\" (UID: \"cd8c7ec3-e6ed-4e68-9c73-678475dd2df0\") " pod="odh-ai-gateway-infra/maas-api-858bf7b9cb-pdjs2"
Sep 11 17:55:56.214040 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:55:56.213848    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd8c7ec3-e6ed-4e68-9c73-678475dd2df0-metrics-certs\") pod \"maas-api-858bf7b9cb-pdjs2\" (UID: \"cd8c7ec3-e6ed-4e68-9c73-678475dd2df0\") " pod="odh-ai-gateway-infra/maas-api-858bf7b9cb-pdjs2"
Sep 11 17:55:56.214040 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:55:56.213868    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8c7ec3-e6ed-4e68-9c73-678475dd2df0-openshift-service-ca\") pod \"maas-api-858bf7b9cb-pdjs2\" (UID: \"cd8c7ec3-e6ed-4e68-9c73-678475dd2df0\") " pod="odh-ai-gateway-infra/maas-api-858bf7b9cb-pdjs2"
Sep 11 17:55:56.214040 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:55:56.213948    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd8c7ec3-e6ed-4e68-9c73-678475dd2df0-maas-api-tls\") pod \"maas-api-858bf7b9cb-pdjs2\" (UID: \"cd8c7ec3-e6ed-4e68-9c73-678475dd2df0\") " pod="odh-ai-gateway-infra/maas-api-858bf7b9cb-pdjs2"
Sep 11 17:55:56.214639 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:55:56.214615    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8c7ec3-e6ed-4e68-9c73-678475dd2df0-openshift-service-ca\") pod \"maas-api-858bf7b9cb-pdjs2\" (UID: \"cd8c7ec3-e6ed-4e68-9c73-678475dd2df0\") " pod="odh-ai-gateway-infra/maas-api-858bf7b9cb-pdjs2"
Sep 11 17:55:56.216458 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:55:56.216430    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd8c7ec3-e6ed-4e68-9c73-678475dd2df0-maas-api-tls\") pod \"maas-api-858bf7b9cb-pdjs2\" (UID: \"cd8c7ec3-e6ed-4e68-9c73-678475dd2df0\") " pod="odh-ai-gateway-infra/maas-api-858bf7b9cb-pdjs2"
Sep 11 17:55:56.216588 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:55:56.216565    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd8c7ec3-e6ed-4e68-9c73-678475dd2df0-metrics-certs\") pod \"maas-api-858bf7b9cb-pdjs2\" (UID: \"cd8c7ec3-e6ed-4e68-9c73-678475dd2df0\") " pod="odh-ai-gateway-infra/maas-api-858bf7b9cb-pdjs2"
Sep 11 17:55:56.223483 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:55:56.223460    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24kmh\" (UniqueName: \"kubernetes.io/projected/cd8c7ec3-e6ed-4e68-9c73-678475dd2df0-kube-api-access-24kmh\") pod \"maas-api-858bf7b9cb-pdjs2\" (UID: \"cd8c7ec3-e6ed-4e68-9c73-678475dd2df0\") " pod="odh-ai-gateway-infra/maas-api-858bf7b9cb-pdjs2"
Sep 11 17:55:56.365136 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:55:56.365102    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-858bf7b9cb-pdjs2"
Sep 11 17:55:56.645931 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:55:56.645885    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-858bf7b9cb-pdjs2"]
Sep 11 17:55:56.647986 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:55:56.647946    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd8c7ec3_e6ed_4e68_9c73_678475dd2df0.slice/crio-577a412aa720e6d7d0a93da81e8d834cf1459293cae4cb40a04eb01aa183fd77 WatchSource:0}: Error finding container 577a412aa720e6d7d0a93da81e8d834cf1459293cae4cb40a04eb01aa183fd77: Status 404 returned error can't find the container with id 577a412aa720e6d7d0a93da81e8d834cf1459293cae4cb40a04eb01aa183fd77
Sep 11 17:55:56.649359 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:55:56.649338    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:55:57.371261 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:55:57.371222    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-858bf7b9cb-pdjs2" event={"ID":"cd8c7ec3-e6ed-4e68-9c73-678475dd2df0","Type":"ContainerStarted","Data":"0158455acdfec961919628b1d8c4b891290ee449a4eb7087f513a559d722b24d"}
Sep 11 17:55:57.371743 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:55:57.371281    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-858bf7b9cb-pdjs2" event={"ID":"cd8c7ec3-e6ed-4e68-9c73-678475dd2df0","Type":"ContainerStarted","Data":"577a412aa720e6d7d0a93da81e8d834cf1459293cae4cb40a04eb01aa183fd77"}
Sep 11 17:55:57.371743 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:55:57.371419    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-858bf7b9cb-pdjs2"
Sep 11 17:56:03.381749 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:03.381719    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-858bf7b9cb-pdjs2"
Sep 11 17:56:03.403179 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:03.403126    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-858bf7b9cb-pdjs2" podStartSLOduration=6.915002502 podStartE2EDuration="7.403111477s" totalImagesPullingTime="488.108975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:57.405653563 +0000 UTC m=+2125.209825983" watchObservedRunningTime="2026-09-11 17:56:03.403111477 +0000 UTC m=+2131.207283895"
Sep 11 17:56:11.774824 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:11.774786    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl"]
Sep 11 17:56:11.777235 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:11.775020    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl" podUID="397ddbb9-61bf-4952-8bc8-2763882ba7e2" containerName="maas-api" containerID="cri-o://0af2473dc5a086b54c5f12584cbcac8e9b6d5b2ecf66a1f3ff112f4b8d01cfd0" gracePeriod=30
Sep 11 17:56:13.014154 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:13.014129    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl"
Sep 11 17:56:13.073065 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:13.073035    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/397ddbb9-61bf-4952-8bc8-2763882ba7e2-maas-api-tls\") pod \"397ddbb9-61bf-4952-8bc8-2763882ba7e2\" (UID: \"397ddbb9-61bf-4952-8bc8-2763882ba7e2\") "
Sep 11 17:56:13.073234 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:13.073075    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/397ddbb9-61bf-4952-8bc8-2763882ba7e2-openshift-service-ca\") pod \"397ddbb9-61bf-4952-8bc8-2763882ba7e2\" (UID: \"397ddbb9-61bf-4952-8bc8-2763882ba7e2\") "
Sep 11 17:56:13.073234 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:13.073116    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/397ddbb9-61bf-4952-8bc8-2763882ba7e2-metrics-certs\") pod \"397ddbb9-61bf-4952-8bc8-2763882ba7e2\" (UID: \"397ddbb9-61bf-4952-8bc8-2763882ba7e2\") "
Sep 11 17:56:13.073234 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:13.073153    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4bzd\" (UniqueName: \"kubernetes.io/projected/397ddbb9-61bf-4952-8bc8-2763882ba7e2-kube-api-access-b4bzd\") pod \"397ddbb9-61bf-4952-8bc8-2763882ba7e2\" (UID: \"397ddbb9-61bf-4952-8bc8-2763882ba7e2\") "
Sep 11 17:56:13.073539 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:13.073506    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/397ddbb9-61bf-4952-8bc8-2763882ba7e2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "397ddbb9-61bf-4952-8bc8-2763882ba7e2" (UID: "397ddbb9-61bf-4952-8bc8-2763882ba7e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:56:13.075413 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:13.075343    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/397ddbb9-61bf-4952-8bc8-2763882ba7e2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "397ddbb9-61bf-4952-8bc8-2763882ba7e2" (UID: "397ddbb9-61bf-4952-8bc8-2763882ba7e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:56:13.075413 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:13.075363    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/397ddbb9-61bf-4952-8bc8-2763882ba7e2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "397ddbb9-61bf-4952-8bc8-2763882ba7e2" (UID: "397ddbb9-61bf-4952-8bc8-2763882ba7e2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:56:13.075413 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:13.075363    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/397ddbb9-61bf-4952-8bc8-2763882ba7e2-kube-api-access-b4bzd" (OuterVolumeSpecName: "kube-api-access-b4bzd") pod "397ddbb9-61bf-4952-8bc8-2763882ba7e2" (UID: "397ddbb9-61bf-4952-8bc8-2763882ba7e2"). InnerVolumeSpecName "kube-api-access-b4bzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:56:13.174349 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:13.174316    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4bzd\" (UniqueName: \"kubernetes.io/projected/397ddbb9-61bf-4952-8bc8-2763882ba7e2-kube-api-access-b4bzd\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:56:13.174349 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:13.174346    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/397ddbb9-61bf-4952-8bc8-2763882ba7e2-maas-api-tls\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:56:13.174349 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:13.174357    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/397ddbb9-61bf-4952-8bc8-2763882ba7e2-openshift-service-ca\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:56:13.174628 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:13.174370    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/397ddbb9-61bf-4952-8bc8-2763882ba7e2-metrics-certs\") on node \"ip-10-0-130-89.ec2.internal\" DevicePath \"\""
Sep 11 17:56:13.445407 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:13.445322    2560 generic.go:356] "Generic (PLEG): container finished" podID="397ddbb9-61bf-4952-8bc8-2763882ba7e2" containerID="0af2473dc5a086b54c5f12584cbcac8e9b6d5b2ecf66a1f3ff112f4b8d01cfd0" exitCode=0
Sep 11 17:56:13.445407 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:13.445367    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl"
Sep 11 17:56:13.445407 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:13.445382    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl" event={"ID":"397ddbb9-61bf-4952-8bc8-2763882ba7e2","Type":"ContainerDied","Data":"0af2473dc5a086b54c5f12584cbcac8e9b6d5b2ecf66a1f3ff112f4b8d01cfd0"}
Sep 11 17:56:13.445644 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:13.445423    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl" event={"ID":"397ddbb9-61bf-4952-8bc8-2763882ba7e2","Type":"ContainerDied","Data":"2b4c735516613120cb82b242f5e5afb11d9e8755aa00d42e3a1823e31658e258"}
Sep 11 17:56:13.445644 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:13.445453    2560 scope.go:117] "RemoveContainer" containerID="0af2473dc5a086b54c5f12584cbcac8e9b6d5b2ecf66a1f3ff112f4b8d01cfd0"
Sep 11 17:56:13.454599 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:13.454579    2560 scope.go:117] "RemoveContainer" containerID="0af2473dc5a086b54c5f12584cbcac8e9b6d5b2ecf66a1f3ff112f4b8d01cfd0"
Sep 11 17:56:13.454852 ip-10-0-130-89 kubenswrapper[2560]: E0911 17:56:13.454834    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0af2473dc5a086b54c5f12584cbcac8e9b6d5b2ecf66a1f3ff112f4b8d01cfd0\": container with ID starting with 0af2473dc5a086b54c5f12584cbcac8e9b6d5b2ecf66a1f3ff112f4b8d01cfd0 not found: ID does not exist" containerID="0af2473dc5a086b54c5f12584cbcac8e9b6d5b2ecf66a1f3ff112f4b8d01cfd0"
Sep 11 17:56:13.454916 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:13.454862    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af2473dc5a086b54c5f12584cbcac8e9b6d5b2ecf66a1f3ff112f4b8d01cfd0"} err="failed to get container status \"0af2473dc5a086b54c5f12584cbcac8e9b6d5b2ecf66a1f3ff112f4b8d01cfd0\": rpc error: code = NotFound desc = could not find container \"0af2473dc5a086b54c5f12584cbcac8e9b6d5b2ecf66a1f3ff112f4b8d01cfd0\": container with ID starting with 0af2473dc5a086b54c5f12584cbcac8e9b6d5b2ecf66a1f3ff112f4b8d01cfd0 not found: ID does not exist"
Sep 11 17:56:13.476857 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:13.476827    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl"]
Sep 11 17:56:13.481503 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:13.481477    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7bf4f9b548-rv2sl"]
Sep 11 17:56:14.715101 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:14.715069    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="397ddbb9-61bf-4952-8bc8-2763882ba7e2" path="/var/lib/kubelet/pods/397ddbb9-61bf-4952-8bc8-2763882ba7e2/volumes"
Sep 11 17:56:32.163912 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:32.163873    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6ddbd79556-cgf4t_23afaf50-6113-4ebb-beac-e55bd9791a18/manager/0.log"
Sep 11 17:56:32.544635 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:32.544559    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f9c58fc78-2xvmq_0d178cc4-c757-4046-ad96-82495b5f9530/copy-manifests/0.log"
Sep 11 17:56:32.552113 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:32.552084    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f9c58fc78-2xvmq_0d178cc4-c757-4046-ad96-82495b5f9530/manager/0.log"
Sep 11 17:56:33.181848 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:33.181810    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-858bf7b9cb-pdjs2_cd8c7ec3-e6ed-4e68-9c73-678475dd2df0/maas-api/0.log"
Sep 11 17:56:33.646141 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:33.646107    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-wx4sf_62cdf1c0-2378-4cdf-a084-11992f088ab8/postgres/0.log"
Sep 11 17:56:35.689567 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:35.689538    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-xfdns_81e4c195-c79e-4719-b867-cb1cdc7f5d40/discovery/0.log"
Sep 11 17:56:36.167042 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:36.167008    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-bq6mr_f6516a4b-de1c-4d62-80d1-7dec6d90eae3/payload-pre-processing/0.log"
Sep 11 17:56:36.289067 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:36.289039    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zn9cq_521643fa-f596-493a-89e3-5b0582f6ed0b/payload-processing/0.log"
Sep 11 17:56:36.873039 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:36.873010    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-fdsb8_21de8bc8-5de8-48a3-9b9e-47cbdff8f669/main/0.log"
Sep 11 17:56:37.107714 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:37.107683    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-jbhdw_35fa4045-cbed-45ab-9fee-a4f5a014f7eb/main/0.log"
Sep 11 17:56:46.464175 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:46.464144    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-82kzs_97afce2d-e357-4c76-b513-2cb05322bbde/global-pull-secret-syncer/0.log"
Sep 11 17:56:46.697011 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:46.696978    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pm66d_efb29080-6e59-4db3-8a34-bf363f39eb9c/konnectivity-agent/0.log"
Sep 11 17:56:46.726252 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:46.726175    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-89.ec2.internal_537070a60bc30f06ce6f629121ed76a8/haproxy/0.log"
Sep 11 17:56:54.065922 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:54.065826    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7hvgz_6dadf403-2628-4ab2-83d7-3bf3a3dac462/node-exporter/0.log"
Sep 11 17:56:54.088553 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:54.088521    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7hvgz_6dadf403-2628-4ab2-83d7-3bf3a3dac462/kube-rbac-proxy/0.log"
Sep 11 17:56:54.114949 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:54.114925    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7hvgz_6dadf403-2628-4ab2-83d7-3bf3a3dac462/init-textfile/0.log"
Sep 11 17:56:54.250167 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:54.249630    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-sczrs_b98862eb-9553-4b1d-b808-810adda98bc8/kube-rbac-proxy-main/0.log"
Sep 11 17:56:54.276411 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:54.276377    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-sczrs_b98862eb-9553-4b1d-b808-810adda98bc8/kube-rbac-proxy-self/0.log"
Sep 11 17:56:54.302723 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:54.302697    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-sczrs_b98862eb-9553-4b1d-b808-810adda98bc8/openshift-state-metrics/0.log"
Sep 11 17:56:54.343492 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:54.343467    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5c4004f2-61ce-4890-9c2a-766558008eba/prometheus/0.log"
Sep 11 17:56:54.369221 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:54.369199    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5c4004f2-61ce-4890-9c2a-766558008eba/config-reloader/0.log"
Sep 11 17:56:54.396825 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:54.396804    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5c4004f2-61ce-4890-9c2a-766558008eba/thanos-sidecar/0.log"
Sep 11 17:56:54.420586 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:54.420562    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5c4004f2-61ce-4890-9c2a-766558008eba/kube-rbac-proxy-web/0.log"
Sep 11 17:56:54.445386 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:54.445358    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5c4004f2-61ce-4890-9c2a-766558008eba/kube-rbac-proxy/0.log"
Sep 11 17:56:54.472482 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:54.472447    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5c4004f2-61ce-4890-9c2a-766558008eba/kube-rbac-proxy-thanos/0.log"
Sep 11 17:56:54.498472 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:54.498439    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5c4004f2-61ce-4890-9c2a-766558008eba/init-config-reloader/0.log"
Sep 11 17:56:54.530677 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:54.530646    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-cc4pf_dc2bf6b3-f8a2-4b4b-972c-919fb11c9000/prometheus-operator/0.log"
Sep 11 17:56:54.552729 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:54.552699    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-cc4pf_dc2bf6b3-f8a2-4b4b-972c-919fb11c9000/kube-rbac-proxy/0.log"
Sep 11 17:56:54.585503 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:54.585476    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-kh7vx_58c88952-e2eb-4729-a21b-251d7d1e6278/prometheus-operator-admission-webhook/0.log"
Sep 11 17:56:54.697962 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:54.697882    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f8d4ccfc-7n5f4_5b2f4bf0-4752-4905-ba63-3879494596bc/thanos-query/0.log"
Sep 11 17:56:54.723184 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:54.723158    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f8d4ccfc-7n5f4_5b2f4bf0-4752-4905-ba63-3879494596bc/kube-rbac-proxy-web/0.log"
Sep 11 17:56:54.746391 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:54.746368    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f8d4ccfc-7n5f4_5b2f4bf0-4752-4905-ba63-3879494596bc/kube-rbac-proxy/0.log"
Sep 11 17:56:54.769422 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:54.769395    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f8d4ccfc-7n5f4_5b2f4bf0-4752-4905-ba63-3879494596bc/prom-label-proxy/0.log"
Sep 11 17:56:54.795511 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:54.795487    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f8d4ccfc-7n5f4_5b2f4bf0-4752-4905-ba63-3879494596bc/kube-rbac-proxy-rules/0.log"
Sep 11 17:56:54.820775 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:54.820752    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f8d4ccfc-7n5f4_5b2f4bf0-4752-4905-ba63-3879494596bc/kube-rbac-proxy-metrics/0.log"
Sep 11 17:56:56.258234 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:56.258199    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rbzkp/perf-node-gather-daemonset-x5f68"]
Sep 11 17:56:56.258634 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:56.258574    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="397ddbb9-61bf-4952-8bc8-2763882ba7e2" containerName="maas-api"
Sep 11 17:56:56.258634 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:56.258585    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="397ddbb9-61bf-4952-8bc8-2763882ba7e2" containerName="maas-api"
Sep 11 17:56:56.258707 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:56.258651    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="397ddbb9-61bf-4952-8bc8-2763882ba7e2" containerName="maas-api"
Sep 11 17:56:56.262250 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:56.262227    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-x5f68"
Sep 11 17:56:56.264644 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:56.264616    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rbzkp\"/\"openshift-service-ca.crt\""
Sep 11 17:56:56.264644 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:56.264637    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rbzkp\"/\"kube-root-ca.crt\""
Sep 11 17:56:56.264842 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:56.264690    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rbzkp\"/\"default-dockercfg-h9zpw\""
Sep 11 17:56:56.272194 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:56.272168    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rbzkp/perf-node-gather-daemonset-x5f68"]
Sep 11 17:56:56.356054 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:56.356020    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e-proc\") pod \"perf-node-gather-daemonset-x5f68\" (UID: \"e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-x5f68"
Sep 11 17:56:56.356240 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:56.356062    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khdrn\" (UniqueName: \"kubernetes.io/projected/e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e-kube-api-access-khdrn\") pod \"perf-node-gather-daemonset-x5f68\" (UID: \"e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-x5f68"
Sep 11 17:56:56.356240 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:56.356097    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e-sys\") pod \"perf-node-gather-daemonset-x5f68\" (UID: \"e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-x5f68"
Sep 11 17:56:56.356240 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:56.356206    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e-lib-modules\") pod \"perf-node-gather-daemonset-x5f68\" (UID: \"e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-x5f68"
Sep 11 17:56:56.356346 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:56.356252    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e-podres\") pod \"perf-node-gather-daemonset-x5f68\" (UID: \"e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-x5f68"
Sep 11 17:56:56.457608 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:56.457569    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e-podres\") pod \"perf-node-gather-daemonset-x5f68\" (UID: \"e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-x5f68"
Sep 11 17:56:56.457787 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:56.457627    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e-proc\") pod \"perf-node-gather-daemonset-x5f68\" (UID: \"e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-x5f68"
Sep 11 17:56:56.457787 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:56.457660    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khdrn\" (UniqueName: \"kubernetes.io/projected/e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e-kube-api-access-khdrn\") pod \"perf-node-gather-daemonset-x5f68\" (UID: \"e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-x5f68"
Sep 11 17:56:56.457787 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:56.457689    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e-sys\") pod \"perf-node-gather-daemonset-x5f68\" (UID: \"e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-x5f68"
Sep 11 17:56:56.457787 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:56.457693    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e-proc\") pod \"perf-node-gather-daemonset-x5f68\" (UID: \"e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-x5f68"
Sep 11 17:56:56.457787 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:56.457731    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e-podres\") pod \"perf-node-gather-daemonset-x5f68\" (UID: \"e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-x5f68"
Sep 11 17:56:56.457787 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:56.457764    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e-lib-modules\") pod \"perf-node-gather-daemonset-x5f68\" (UID: \"e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-x5f68"
Sep 11 17:56:56.457787 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:56.457771    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e-sys\") pod \"perf-node-gather-daemonset-x5f68\" (UID: \"e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-x5f68"
Sep 11 17:56:56.458086 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:56.457815    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e-lib-modules\") pod \"perf-node-gather-daemonset-x5f68\" (UID: \"e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-x5f68"
Sep 11 17:56:56.466885 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:56.466861    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khdrn\" (UniqueName: \"kubernetes.io/projected/e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e-kube-api-access-khdrn\") pod \"perf-node-gather-daemonset-x5f68\" (UID: \"e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-x5f68"
Sep 11 17:56:56.572221 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:56.572130    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-x5f68"
Sep 11 17:56:56.670952 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:56.670923    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6fcjs_477cac01-ab9e-4441-b07d-b7e53401c616/console-operator/1.log"
Sep 11 17:56:56.674286 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:56.674256    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-6fcjs_477cac01-ab9e-4441-b07d-b7e53401c616/console-operator/2.log"
Sep 11 17:56:56.841847 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:56.841820    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rbzkp/perf-node-gather-daemonset-x5f68"]
Sep 11 17:56:56.843480 ip-10-0-130-89 kubenswrapper[2560]: W0911 17:56:56.843458    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode54330e3_5efa_4c5a_b2a6_920d1b3d7c3e.slice/crio-d4ef1c786628220227c381c2260b2a21c8cffa1b81dfcb36106283f3de3cab2a WatchSource:0}: Error finding container d4ef1c786628220227c381c2260b2a21c8cffa1b81dfcb36106283f3de3cab2a: Status 404 returned error can't find the container with id d4ef1c786628220227c381c2260b2a21c8cffa1b81dfcb36106283f3de3cab2a
Sep 11 17:56:57.651456 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:57.651417    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-x5f68" event={"ID":"e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e","Type":"ContainerStarted","Data":"2c26ff143504c55199715a46f58c20dca5c2c09f51465fc9b920cc351541e1b4"}
Sep 11 17:56:57.651830 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:57.651471    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-x5f68" event={"ID":"e54330e3-5efa-4c5a-b2a6-920d1b3d7c3e","Type":"ContainerStarted","Data":"d4ef1c786628220227c381c2260b2a21c8cffa1b81dfcb36106283f3de3cab2a"}
Sep 11 17:56:57.651830 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:57.651520    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-x5f68"
Sep 11 17:56:57.668985 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:57.668926    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-x5f68" podStartSLOduration=1.668889813 podStartE2EDuration="1.668889813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:56:57.668004309 +0000 UTC m=+2185.472176728" watchObservedRunningTime="2026-09-11 17:56:57.668889813 +0000 UTC m=+2185.473062232"
Sep 11 17:56:59.112498 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:59.112470    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mp4vh_1dbb8c42-1654-4f42-8237-cf79927d51db/dns/0.log"
Sep 11 17:56:59.139730 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:59.139704    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mp4vh_1dbb8c42-1654-4f42-8237-cf79927d51db/kube-rbac-proxy/0.log"
Sep 11 17:56:59.299407 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:59.299376    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4mxdh_c49e0b27-b621-4879-b168-3962a9a32c8f/dns-node-resolver/0.log"
Sep 11 17:56:59.976876 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:56:59.976848    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q9dr2_c11a0ba2-8654-4e79-a883-80d867a24d80/node-ca/0.log"
Sep 11 17:57:01.277536 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:57:01.277497    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-xfdns_81e4c195-c79e-4719-b867-cb1cdc7f5d40/discovery/0.log"
Sep 11 17:57:01.434597 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:57:01.434562    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-bq6mr_f6516a4b-de1c-4d62-80d1-7dec6d90eae3/payload-pre-processing/0.log"
Sep 11 17:57:01.471554 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:57:01.471526    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zn9cq_521643fa-f596-493a-89e3-5b0582f6ed0b/payload-processing/0.log"
Sep 11 17:57:02.414051 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:57:02.414020    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xz968_fcf52a64-2196-4fb8-9b91-a8f0c4956cdd/serve-healthcheck-canary/0.log"
Sep 11 17:57:02.942454 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:57:02.942425    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8hpnj_cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb/kube-rbac-proxy/0.log"
Sep 11 17:57:02.966358 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:57:02.966330    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8hpnj_cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb/exporter/0.log"
Sep 11 17:57:02.991770 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:57:02.991723    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8hpnj_cc8c333a-4c92-4b4d-8cc4-8acf75d3aafb/extractor/0.log"
Sep 11 17:57:03.665665 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:57:03.665639    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-x5f68"
Sep 11 17:57:05.740616 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:57:05.740587    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6ddbd79556-cgf4t_23afaf50-6113-4ebb-beac-e55bd9791a18/manager/0.log"
Sep 11 17:57:05.923853 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:57:05.923780    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f9c58fc78-2xvmq_0d178cc4-c757-4046-ad96-82495b5f9530/manager/0.log"
Sep 11 17:57:05.947263 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:57:05.947230    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f9c58fc78-2xvmq_0d178cc4-c757-4046-ad96-82495b5f9530/copy-manifests/0.log"
Sep 11 17:57:07.390538 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:57:07.390500    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5786cf49b6-qzhhh_28e028d7-1f3a-4335-91ad-835ee47f0635/manager/0.log"
Sep 11 17:57:07.455638 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:57:07.455607    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-lbdgf_39e8377d-780d-400a-bf5b-b73ec7e3a97d/openshift-lws-operator/0.log"
Sep 11 17:57:07.577285 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:57:07.577252    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-xfdns_81e4c195-c79e-4719-b867-cb1cdc7f5d40/discovery/0.log"
Sep 11 17:57:07.677194 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:57:07.677167    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-bq6mr_f6516a4b-de1c-4d62-80d1-7dec6d90eae3/payload-pre-processing/0.log"
Sep 11 17:57:07.704512 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:57:07.704482    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zn9cq_521643fa-f596-493a-89e3-5b0582f6ed0b/payload-processing/0.log"
Sep 11 17:57:09.421391 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:57:09.421354    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-xfdns_81e4c195-c79e-4719-b867-cb1cdc7f5d40/discovery/0.log"
Sep 11 17:57:09.528180 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:57:09.528148    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-bq6mr_f6516a4b-de1c-4d62-80d1-7dec6d90eae3/payload-pre-processing/0.log"
Sep 11 17:57:09.556953 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:57:09.556922    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zn9cq_521643fa-f596-493a-89e3-5b0582f6ed0b/payload-processing/0.log"
Sep 11 17:57:13.011666 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:57:13.011635    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-jd6vw_b996965e-b19e-4959-b58c-a3cb4039d465/kube-storage-version-migrator-operator/1.log"
Sep 11 17:57:13.012614 ip-10-0-130-89 kubenswrapper[2560]: I0911 17:57:13.012594    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-jd6vw_b996965e-b19e-4959-b58c-a3cb4039d465/kube-storage-version-migrator-operator/0.log"