Jul 22 01:37:07.969305 ip-10-0-131-56 systemd[1]: Starting Kubernetes Kubelet... Jul 22 01:37:08.377047 ip-10-0-131-56 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 01:37:08.377047 ip-10-0-131-56 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 22 01:37:08.377047 ip-10-0-131-56 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 01:37:08.377047 ip-10-0-131-56 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 22 01:37:08.377047 ip-10-0-131-56 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 01:37:08.377737 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.377126 2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 22 01:37:08.382142 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382127 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 01:37:08.382142 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382142 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 01:37:08.382205 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382145 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 01:37:08.382205 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382149 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 01:37:08.382205 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382151 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 01:37:08.382205 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382154 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 01:37:08.382205 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382157 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 01:37:08.382205 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382163 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 01:37:08.382205 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382167 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 01:37:08.382205 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382170 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 01:37:08.382205 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382173 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 01:37:08.382205 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382176 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 01:37:08.382205 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382178 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 01:37:08.382205 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382181 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 01:37:08.382205 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382183 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 01:37:08.382205 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382186 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 01:37:08.382205 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382190 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 01:37:08.382205 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382193 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 01:37:08.382205 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382196 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 01:37:08.382205 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382198 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 01:37:08.382205 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382201 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 01:37:08.382658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382204 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 01:37:08.382658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382210 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 01:37:08.382658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382213 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 01:37:08.382658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382216 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 01:37:08.382658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382219 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 01:37:08.382658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382221 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 01:37:08.382658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382224 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 01:37:08.382658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382228 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 01:37:08.382658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382232 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 22 01:37:08.382658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382235 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 01:37:08.382658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382237 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 01:37:08.382658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382240 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 01:37:08.382658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382243 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 01:37:08.382658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382245 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 01:37:08.382658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382248 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 01:37:08.382658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382251 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 01:37:08.382658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382253 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 01:37:08.382658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382257 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 01:37:08.382658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382259 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 01:37:08.383138 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382262 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 01:37:08.383138 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382264 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 01:37:08.383138 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382267 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 01:37:08.383138 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382269 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 01:37:08.383138 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382271 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 01:37:08.383138 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382274 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 01:37:08.383138 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382276 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 01:37:08.383138 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382279 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 01:37:08.383138 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382281 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 01:37:08.383138 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382284 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 01:37:08.383138 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382286 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 01:37:08.383138 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382289 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 01:37:08.383138 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382292 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 01:37:08.383138 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382294 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 01:37:08.383138 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382298 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 01:37:08.383138 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382300 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 01:37:08.383138 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382303 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 01:37:08.383138 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382306 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 01:37:08.383138 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382308 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 01:37:08.383138 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382311 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 01:37:08.383646 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382313 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 01:37:08.383646 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382316 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 01:37:08.383646 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382319 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 01:37:08.383646 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382321 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 01:37:08.383646 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382324 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 01:37:08.383646 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382327 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 01:37:08.383646 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382329 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 01:37:08.383646 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382331 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 01:37:08.383646 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382334 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 01:37:08.383646 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382336 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 01:37:08.383646 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382339 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 01:37:08.383646 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382342 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 01:37:08.383646 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382344 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 01:37:08.383646 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382346 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 01:37:08.383646 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382349 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 01:37:08.383646 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382351 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 01:37:08.383646 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382354 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 01:37:08.383646 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382356 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 01:37:08.383646 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382358 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 01:37:08.383646 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382361 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 01:37:08.384133 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382364 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 01:37:08.384133 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382366 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 01:37:08.384133 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382369 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 01:37:08.384133 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382371 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 01:37:08.384133 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382375 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 01:37:08.384133 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382377 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 01:37:08.384133 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382380 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 01:37:08.384133 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382383 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 01:37:08.384133 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382385 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 01:37:08.384133 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382387 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 01:37:08.384133 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382390 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 01:37:08.384133 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382393 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 01:37:08.384133 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382396 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 01:37:08.384133 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382398 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 01:37:08.384133 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382401 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 01:37:08.384133 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382403 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 01:37:08.384133 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382406 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 01:37:08.384133 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382408 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 01:37:08.384133 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.382411 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 01:37:08.384133 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383119 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 01:37:08.384640 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383125 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 01:37:08.384640 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383128 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 01:37:08.384640 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383131 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 01:37:08.384640 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383133 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 01:37:08.384640 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383136 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 01:37:08.384640 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383138 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 01:37:08.384640 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383141 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 01:37:08.384640 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383143 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 01:37:08.384640 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383146 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 01:37:08.384640 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383149 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 01:37:08.384640 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383151 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 01:37:08.384640 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383154 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 01:37:08.384640 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383157 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 01:37:08.384640 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383159 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 01:37:08.384640 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383164 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 01:37:08.384640 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383168 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 01:37:08.384640 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383171 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 01:37:08.384640 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383174 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 01:37:08.384640 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383178 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 01:37:08.384640 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383181 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 01:37:08.385144 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383183 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 01:37:08.385144 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383186 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 01:37:08.385144 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383189 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 01:37:08.385144 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383192 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 01:37:08.385144 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383195 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 01:37:08.385144 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383197 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 01:37:08.385144 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383199 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 01:37:08.385144 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383202 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 01:37:08.385144 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383204 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 01:37:08.385144 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383207 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 01:37:08.385144 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383210 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 01:37:08.385144 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383212 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 01:37:08.385144 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383214 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 01:37:08.385144 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383217 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 01:37:08.385144 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383219 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 01:37:08.385144 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383222 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 01:37:08.385144 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383225 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 01:37:08.385144 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383227 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 01:37:08.385144 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383230 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 01:37:08.385635 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383233 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 01:37:08.385635 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383235 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 01:37:08.385635 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383239 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 01:37:08.385635 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383242 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 01:37:08.385635 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383245 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 01:37:08.385635 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383248 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 01:37:08.385635 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383250 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 01:37:08.385635 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383253 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 01:37:08.385635 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383256 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 01:37:08.385635 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383258 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 01:37:08.385635 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383261 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 01:37:08.385635 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383264 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 01:37:08.385635 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383266 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 01:37:08.385635 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383269 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 01:37:08.385635 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383271 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 01:37:08.385635 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383274 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 01:37:08.385635 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383276 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 01:37:08.385635 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383279 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 01:37:08.385635 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383281 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 01:37:08.386111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383283 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 01:37:08.386111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383286 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 01:37:08.386111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383288 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 01:37:08.386111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383291 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 01:37:08.386111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383293 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 01:37:08.386111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383295 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 01:37:08.386111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383297 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 01:37:08.386111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383300 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 01:37:08.386111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383302 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 01:37:08.386111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383304 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 01:37:08.386111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383307 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 01:37:08.386111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383309 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 01:37:08.386111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383311 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 01:37:08.386111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383314 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 01:37:08.386111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383316 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 01:37:08.386111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383319 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 01:37:08.386111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383321 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 01:37:08.386111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383323 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 01:37:08.386111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383326 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 01:37:08.386111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383329 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 01:37:08.386598 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383331 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 01:37:08.386598 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383334 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 01:37:08.386598 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383337 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 01:37:08.386598 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383340 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 01:37:08.386598 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383342 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 01:37:08.386598 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383344 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 01:37:08.386598 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383347 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 01:37:08.386598 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383351 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 01:37:08.386598 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383355 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 01:37:08.386598 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383358 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 01:37:08.386598 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383361 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 01:37:08.386598 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383364 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 01:37:08.386598 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383367 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 01:37:08.386598 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383369 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 01:37:08.386598 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383372 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 01:37:08.386598 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383375 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 01:37:08.386598 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383378 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 01:37:08.386598 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383380 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 01:37:08.386598 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383383 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 01:37:08.386598 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383386 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 22 01:37:08.387111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383732 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 01:37:08.387111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383737 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 01:37:08.387111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383740 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 01:37:08.387111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383743 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 01:37:08.387111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383745 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 01:37:08.387111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383748 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 01:37:08.387111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383750 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 01:37:08.387111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383752 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 01:37:08.387111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383755 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 01:37:08.387111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383757 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 01:37:08.387111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383762 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 01:37:08.387111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383764 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 01:37:08.387111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383767 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 01:37:08.387111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383770 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 01:37:08.387111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383772 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 01:37:08.387111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383775 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 01:37:08.387111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383777 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 01:37:08.387111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383780 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 01:37:08.387111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383782 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 01:37:08.387111 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383785 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 01:37:08.387658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383787 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 01:37:08.387658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383790 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 01:37:08.387658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383792 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 01:37:08.387658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383795 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 01:37:08.387658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383797 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 01:37:08.387658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383800 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 01:37:08.387658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383802 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 01:37:08.387658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383805 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 01:37:08.387658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383807 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 01:37:08.387658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383811 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 01:37:08.387658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383815 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 01:37:08.387658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383817 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 01:37:08.387658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383820 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 01:37:08.387658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383823 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 01:37:08.387658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383826 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 01:37:08.387658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383829 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 01:37:08.387658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383832 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 01:37:08.387658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383835 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 01:37:08.387658 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383838 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 01:37:08.388127 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383841 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 01:37:08.388127 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383843 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 01:37:08.388127 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383846 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 01:37:08.388127 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383850 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 01:37:08.388127 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383853 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 01:37:08.388127 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383855 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 01:37:08.388127 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383857 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 01:37:08.388127 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383860 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 01:37:08.388127 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383863 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 01:37:08.388127 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383866 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 01:37:08.388127 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383868 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 01:37:08.388127 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383871 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 01:37:08.388127 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383873 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 01:37:08.388127 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383876 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 01:37:08.388127 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383878 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 01:37:08.388127 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383880 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 01:37:08.388127 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383883 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 01:37:08.388127 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383885 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 01:37:08.388127 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383888 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 01:37:08.388127 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383890 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 01:37:08.388611 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383892 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 01:37:08.388611 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383895 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 01:37:08.388611 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383897 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 01:37:08.388611 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383900 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 01:37:08.388611 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383903 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 01:37:08.388611 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383906 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 01:37:08.388611 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383910 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 01:37:08.388611 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383912 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 01:37:08.388611 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383914 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 01:37:08.388611 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383917 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 22 01:37:08.388611 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383919 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 01:37:08.388611 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383921 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 01:37:08.388611 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383924 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 01:37:08.388611 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383926 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 01:37:08.388611 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383929 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 01:37:08.388611 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383932 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 01:37:08.388611 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383935 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 01:37:08.388611 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383937 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 01:37:08.388611 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383940 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 01:37:08.388611 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383942 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 01:37:08.389104 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383945 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 01:37:08.389104 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383950 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 01:37:08.389104 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383953 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 01:37:08.389104 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383956 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 01:37:08.389104 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383959 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 01:37:08.389104 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383961 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 01:37:08.389104 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383964 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 01:37:08.389104 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383967 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 01:37:08.389104 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383969 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 01:37:08.389104 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383972 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 01:37:08.389104 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383975 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 01:37:08.389104 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383978 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 01:37:08.389104 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383980 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 01:37:08.389104 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383983 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 01:37:08.389104 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383986 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 01:37:08.389104 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383989 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 01:37:08.389104 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383992 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 01:37:08.389104 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383994 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 01:37:08.389104 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383997 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 01:37:08.389552 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.383999 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 01:37:08.389552 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384089 2561 flags.go:64] FLAG: --address="0.0.0.0" Jul 22 01:37:08.389552 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384099 2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 22 01:37:08.389552 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384108 2561 flags.go:64] FLAG: --anonymous-auth="true" Jul 22 01:37:08.389552 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384114 2561 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 22 01:37:08.389552 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384120 2561 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 22 01:37:08.389552 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384124 2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 22 01:37:08.389552 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384130 2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 22 01:37:08.389552 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384139 2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 22 01:37:08.389552 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384143 2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 22 01:37:08.389552 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384146 2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 22 01:37:08.389552 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384149 2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 22 01:37:08.389552 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384152 2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 22 01:37:08.389552 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384155 2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 22 01:37:08.389552 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384160 2561 flags.go:64] FLAG: --cgroup-root="" Jul 22 01:37:08.389552 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384162 2561 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 22 01:37:08.389552 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384166 2561 flags.go:64] FLAG: --client-ca-file="" Jul 22 01:37:08.389552 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384168 2561 flags.go:64] FLAG: --cloud-provider="external" Jul 22 01:37:08.389552 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384171 2561 flags.go:64] FLAG: --cluster-dns="[]" Jul 22 01:37:08.389552 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384175 2561 flags.go:64] FLAG: --cluster-domain="" Jul 22 01:37:08.389552 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384178 2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 22 01:37:08.389552 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384181 2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 22 01:37:08.389552 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384184 2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 22 01:37:08.389552 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384188 2561 flags.go:64] FLAG: --container-log-max-files="5" Jul 22 01:37:08.390136 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384191 2561 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 22 01:37:08.390136 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384194 2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 22 01:37:08.390136 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384197 2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 22 01:37:08.390136 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384200 2561 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 22 01:37:08.390136 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384204 2561 flags.go:64] FLAG: --contention-profiling="false" Jul 22 01:37:08.390136 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384207 2561 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 22 01:37:08.390136 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384210 2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 22 01:37:08.390136 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384213 2561 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 22 01:37:08.390136 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384216 2561 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 22 01:37:08.390136 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384220 2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 22 01:37:08.390136 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384223 2561 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 22 01:37:08.390136 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384226 2561 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 22 01:37:08.390136 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384228 2561 flags.go:64] FLAG: --enable-load-reader="false" Jul 22 01:37:08.390136 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384231 2561 flags.go:64] FLAG: --enable-server="true" Jul 22 01:37:08.390136 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384235 2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 22 01:37:08.390136 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384238 2561 flags.go:64] FLAG: --event-burst="100" Jul 22 01:37:08.390136 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384244 2561 flags.go:64] FLAG: --event-qps="50" Jul 22 01:37:08.390136 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384246 2561 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 22 01:37:08.390136 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384249 2561 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 22 01:37:08.390136 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384252 2561 flags.go:64] FLAG: --eviction-hard="" Jul 22 01:37:08.390136 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384255 2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 22 01:37:08.390136 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384258 2561 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 22 01:37:08.390136 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384261 2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 22 01:37:08.390136 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384265 2561 flags.go:64] FLAG: --eviction-soft="" Jul 22 01:37:08.390136 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384268 2561 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 22 01:37:08.390793 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384270 2561 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 22 01:37:08.390793 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384273 2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 22 01:37:08.390793 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384276 2561 flags.go:64] FLAG: --experimental-mounter-path="" Jul 22 01:37:08.390793 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384278 2561 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 22 01:37:08.390793 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384281 2561 flags.go:64] FLAG: --fail-swap-on="true" Jul 22 01:37:08.390793 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384284 2561 flags.go:64] FLAG: --feature-gates="" Jul 22 01:37:08.390793 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384287 2561 flags.go:64] FLAG: --file-check-frequency="20s" Jul 22 01:37:08.390793 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384290 2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 22 01:37:08.390793 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384293 2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 22 01:37:08.390793 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384296 2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 22 01:37:08.390793 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384299 2561 flags.go:64] FLAG: --healthz-port="10248" Jul 22 01:37:08.390793 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384302 2561 flags.go:64] FLAG: --help="false" Jul 22 01:37:08.390793 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384305 2561 flags.go:64] FLAG: --hostname-override="ip-10-0-131-56.ec2.internal" Jul 22 01:37:08.390793 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384308 2561 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 22 01:37:08.390793 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384311 2561 flags.go:64] FLAG: --http-check-frequency="20s" Jul 22 01:37:08.390793 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384314 2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 22 01:37:08.390793 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384317 2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 22 01:37:08.390793 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384320 2561 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 22 01:37:08.390793 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384323 2561 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 22 01:37:08.390793 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384326 2561 flags.go:64] FLAG: --image-service-endpoint="" Jul 22 01:37:08.390793 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384329 2561 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 22 01:37:08.390793 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384331 2561 flags.go:64] FLAG: --kube-api-burst="100" Jul 22 01:37:08.390793 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384334 2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 22 01:37:08.391407 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384340 2561 flags.go:64] FLAG: --kube-api-qps="50" Jul 22 01:37:08.391407 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384343 2561 flags.go:64] FLAG: --kube-reserved="" Jul 22 01:37:08.391407 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384346 2561 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 22 01:37:08.391407 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384348 2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 22 01:37:08.391407 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384351 2561 flags.go:64] FLAG: --kubelet-cgroups="" Jul 22 01:37:08.391407 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384353 2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 22 01:37:08.391407 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384356 2561 flags.go:64] FLAG: --lock-file="" Jul 22 01:37:08.391407 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384359 2561 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 22 01:37:08.391407 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384362 2561 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 22 01:37:08.391407 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384365 2561 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 22 01:37:08.391407 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384370 2561 flags.go:64] FLAG: --log-json-split-stream="false" Jul 22 01:37:08.391407 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384372 2561 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 22 01:37:08.391407 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384375 2561 flags.go:64] FLAG: --log-text-split-stream="false" Jul 22 01:37:08.391407 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384378 2561 flags.go:64] FLAG: --logging-format="text" Jul 22 01:37:08.391407 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384380 2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 22 01:37:08.391407 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384383 2561 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 22 01:37:08.391407 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384386 2561 flags.go:64] FLAG: --manifest-url="" Jul 22 01:37:08.391407 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384389 2561 flags.go:64] FLAG: --manifest-url-header="" Jul 22 01:37:08.391407 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384393 2561 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 22 01:37:08.391407 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384396 2561 flags.go:64] FLAG: --max-open-files="1000000" Jul 22 01:37:08.391407 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384401 2561 flags.go:64] FLAG: --max-pods="110" Jul 22 01:37:08.391407 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384404 2561 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 22 01:37:08.391407 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384406 2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 22 01:37:08.391407 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384409 2561 flags.go:64] FLAG: --memory-manager-policy="None" Jul 22 01:37:08.391407 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384412 2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 22 01:37:08.392015 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384415 2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 22 01:37:08.392015 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384417 2561 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 22 01:37:08.392015 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384420 2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 22 01:37:08.392015 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384426 2561 flags.go:64] FLAG: --node-status-max-images="50" Jul 22 01:37:08.392015 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384429 2561 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 22 01:37:08.392015 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384432 2561 flags.go:64] FLAG: --oom-score-adj="-999" Jul 22 01:37:08.392015 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384435 2561 flags.go:64] FLAG: --pod-cidr="" Jul 22 01:37:08.392015 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384439 2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 22 01:37:08.392015 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384444 2561 flags.go:64] FLAG: --pod-manifest-path="" Jul 22 01:37:08.392015 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384447 2561 flags.go:64] FLAG: --pod-max-pids="-1" Jul 22 01:37:08.392015 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384450 2561 flags.go:64] FLAG: --pods-per-core="0" Jul 22 01:37:08.392015 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384453 2561 flags.go:64] FLAG: --port="10250" Jul 22 01:37:08.392015 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384456 2561 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 22 01:37:08.392015 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384459 2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07a1f84d98e816096" Jul 22 01:37:08.392015 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384462 2561 flags.go:64] FLAG: --qos-reserved="" Jul 22 01:37:08.392015 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384464 2561 flags.go:64] FLAG: --read-only-port="10255" Jul 22 01:37:08.392015 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384468 2561 flags.go:64] FLAG: --register-node="true" Jul 22 01:37:08.392015 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384471 2561 flags.go:64] FLAG: --register-with-taints="" Jul 22 01:37:08.392015 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384474 2561 flags.go:64] FLAG: --registry-burst="10" Jul 22 01:37:08.392015 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384477 2561 flags.go:64] FLAG: --registry-qps="5" Jul 22 01:37:08.392015 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384480 2561 flags.go:64] FLAG: --reserved-cpus="" Jul 22 01:37:08.392015 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384482 2561 flags.go:64] FLAG: --reserved-memory="" Jul 22 01:37:08.392015 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384486 2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 22 01:37:08.392015 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384488 2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 22 01:37:08.392015 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384491 2561 flags.go:64] FLAG: --rotate-certificates="false" Jul 22 01:37:08.392640 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384493 2561 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 22 01:37:08.392640 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384497 2561 flags.go:64] FLAG: --runonce="false" Jul 22 01:37:08.392640 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384501 2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 22 01:37:08.392640 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384504 2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 22 01:37:08.392640 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384506 2561 flags.go:64] FLAG: --seccomp-default="false" Jul 22 01:37:08.392640 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384509 2561 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 22 01:37:08.392640 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384512 2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 22 01:37:08.392640 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384515 2561 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 22 01:37:08.392640 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384518 2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 22 01:37:08.392640 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384521 2561 flags.go:64] FLAG: --storage-driver-password="root" Jul 22 01:37:08.392640 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384523 2561 flags.go:64] FLAG: --storage-driver-secure="false" Jul 22 01:37:08.392640 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384526 2561 flags.go:64] FLAG: --storage-driver-table="stats" Jul 22 01:37:08.392640 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384528 2561 flags.go:64] FLAG: --storage-driver-user="root" Jul 22 01:37:08.392640 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384531 2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 22 01:37:08.392640 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384534 2561 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 22 01:37:08.392640 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384539 2561 flags.go:64] FLAG: --system-cgroups="" Jul 22 01:37:08.392640 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384541 2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 22 01:37:08.392640 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384546 2561 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 22 01:37:08.392640 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384549 2561 flags.go:64] FLAG: --tls-cert-file="" Jul 22 01:37:08.392640 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384551 2561 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 22 01:37:08.392640 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384556 2561 flags.go:64] FLAG: --tls-min-version="" Jul 22 01:37:08.392640 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384558 2561 flags.go:64] FLAG: --tls-private-key-file="" Jul 22 01:37:08.392640 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384561 2561 flags.go:64] FLAG: --topology-manager-policy="none" Jul 22 01:37:08.392640 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384564 2561 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 22 01:37:08.392640 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384567 2561 flags.go:64] FLAG: --topology-manager-scope="container" Jul 22 01:37:08.393265 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384570 2561 flags.go:64] FLAG: --v="2" Jul 22 01:37:08.393265 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384573 2561 flags.go:64] FLAG: --version="false" Jul 22 01:37:08.393265 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384577 2561 flags.go:64] FLAG: --vmodule="" Jul 22 01:37:08.393265 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384580 2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 22 01:37:08.393265 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.384583 2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 22 01:37:08.393265 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384698 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 01:37:08.393265 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384702 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 01:37:08.393265 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384705 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 01:37:08.393265 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384708 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 01:37:08.393265 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384713 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 01:37:08.393265 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384716 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 01:37:08.393265 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384718 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 01:37:08.393265 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384721 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 01:37:08.393265 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384723 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 01:37:08.393265 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384726 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 01:37:08.393265 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384728 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 01:37:08.393265 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384731 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 01:37:08.393265 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384734 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 01:37:08.393265 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384736 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 01:37:08.393265 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384738 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 01:37:08.393265 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384741 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 01:37:08.393772 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384743 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 01:37:08.393772 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384747 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 01:37:08.393772 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384749 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 01:37:08.393772 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384752 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 01:37:08.393772 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384755 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 01:37:08.393772 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384757 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 01:37:08.393772 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384760 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 01:37:08.393772 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384762 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 01:37:08.393772 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384764 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 01:37:08.393772 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384767 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 01:37:08.393772 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384769 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 01:37:08.393772 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384773 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 01:37:08.393772 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384775 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 01:37:08.393772 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384778 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 01:37:08.393772 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384781 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 01:37:08.393772 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384784 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 01:37:08.393772 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384786 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 01:37:08.393772 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384789 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 01:37:08.393772 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384791 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 22 01:37:08.393772 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384794 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 01:37:08.394308 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384798 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 01:37:08.394308 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384800 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 01:37:08.394308 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384802 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 01:37:08.394308 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384805 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 01:37:08.394308 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384808 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 01:37:08.394308 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384810 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 01:37:08.394308 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384813 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 01:37:08.394308 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384815 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 01:37:08.394308 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384817 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 01:37:08.394308 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384820 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 01:37:08.394308 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384822 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 01:37:08.394308 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384824 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 01:37:08.394308 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384827 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 01:37:08.394308 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384830 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 01:37:08.394308 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384833 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 01:37:08.394308 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384836 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 01:37:08.394308 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384838 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 01:37:08.394308 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384840 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 01:37:08.394308 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384843 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 01:37:08.394756 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384845 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 01:37:08.394756 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384847 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 01:37:08.394756 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384851 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 01:37:08.394756 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384855 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 01:37:08.394756 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384857 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 01:37:08.394756 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384860 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 01:37:08.394756 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384863 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 01:37:08.394756 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384866 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 01:37:08.394756 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384870 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 01:37:08.394756 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384872 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 01:37:08.394756 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384875 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 01:37:08.394756 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384878 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 01:37:08.394756 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384880 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 01:37:08.394756 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384884 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 01:37:08.394756 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384886 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 01:37:08.394756 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384889 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 01:37:08.394756 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384891 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 01:37:08.394756 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384894 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 01:37:08.394756 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384896 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 01:37:08.394756 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384898 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 01:37:08.395275 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384901 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 01:37:08.395275 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384903 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 01:37:08.395275 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384907 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 01:37:08.395275 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384910 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 01:37:08.395275 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384913 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 01:37:08.395275 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384915 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 01:37:08.395275 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384919 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 01:37:08.395275 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384922 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 01:37:08.395275 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384925 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 01:37:08.395275 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384927 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 01:37:08.395275 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384930 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 01:37:08.395275 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384932 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 01:37:08.395275 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384935 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 01:37:08.395275 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384938 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 01:37:08.395275 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384940 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 01:37:08.395275 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384942 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 01:37:08.395275 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384945 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 01:37:08.395275 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384948 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 01:37:08.395275 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384950 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 01:37:08.395737 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384953 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 01:37:08.395737 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384955 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 01:37:08.395737 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384958 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 01:37:08.395737 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384961 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 01:37:08.395737 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.384963 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 01:37:08.395737 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.385499 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 01:37:08.395737 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.392490 2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 22 01:37:08.395737 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.392503 2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 22 01:37:08.395737 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.392519 2561 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 22 01:37:08.395737 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.392527 2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 22 01:37:08.395737 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392596 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 01:37:08.395737 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392601 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 01:37:08.395737 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392604 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 01:37:08.395737 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392607 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 01:37:08.395737 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392610 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 01:37:08.396130 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392613 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 01:37:08.396130 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392615 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 01:37:08.396130 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392618 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 01:37:08.396130 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392621 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 01:37:08.396130 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392624 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 22 01:37:08.396130 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392626 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 01:37:08.396130 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392629 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 01:37:08.396130 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392631 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 01:37:08.396130 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392634 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 01:37:08.396130 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392636 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 01:37:08.396130 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392639 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 01:37:08.396130 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392641 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 01:37:08.396130 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392644 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 01:37:08.396130 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392647 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 01:37:08.396130 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392649 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 01:37:08.396130 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392652 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 01:37:08.396130 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392655 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 01:37:08.396130 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392657 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 01:37:08.396130 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392661 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 01:37:08.396130 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392664 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 01:37:08.396623 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392667 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 01:37:08.396623 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392670 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 01:37:08.396623 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392673 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 01:37:08.396623 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392675 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 01:37:08.396623 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392679 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 01:37:08.396623 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392682 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 01:37:08.396623 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392685 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 01:37:08.396623 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392687 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 01:37:08.396623 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392689 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 01:37:08.396623 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392692 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 01:37:08.396623 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392695 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 01:37:08.396623 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392697 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 01:37:08.396623 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392699 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 01:37:08.396623 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392702 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 01:37:08.396623 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392704 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 01:37:08.396623 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392707 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 01:37:08.396623 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392709 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 01:37:08.396623 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392711 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 01:37:08.396623 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392714 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 01:37:08.396623 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392716 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 01:37:08.397121 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392719 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 01:37:08.397121 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392721 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 01:37:08.397121 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392723 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 01:37:08.397121 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392726 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 01:37:08.397121 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392728 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 01:37:08.397121 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392731 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 01:37:08.397121 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392733 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 01:37:08.397121 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392735 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 01:37:08.397121 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392737 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 01:37:08.397121 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392740 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 01:37:08.397121 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392742 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 01:37:08.397121 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392744 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 01:37:08.397121 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392747 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 01:37:08.397121 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392749 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 01:37:08.397121 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392752 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 01:37:08.397121 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392754 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 01:37:08.397121 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392758 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 01:37:08.397121 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392763 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 01:37:08.397121 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392766 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 01:37:08.397579 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392768 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 01:37:08.397579 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392771 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 01:37:08.397579 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392774 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 01:37:08.397579 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392776 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 01:37:08.397579 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392778 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 01:37:08.397579 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392781 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 01:37:08.397579 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392783 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 01:37:08.397579 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392785 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 01:37:08.397579 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392788 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 01:37:08.397579 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392790 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 01:37:08.397579 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392792 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 01:37:08.397579 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392795 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 01:37:08.397579 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392798 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 01:37:08.397579 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392800 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 01:37:08.397579 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392802 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 01:37:08.397579 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392805 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 01:37:08.397579 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392807 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 01:37:08.397579 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392810 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 01:37:08.397579 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392814 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 01:37:08.398089 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392817 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 01:37:08.398089 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392819 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 01:37:08.398089 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392823 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 01:37:08.398089 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392825 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 01:37:08.398089 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392828 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 01:37:08.398089 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392831 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 01:37:08.398089 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392834 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 01:37:08.398089 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392836 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 01:37:08.398089 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392839 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 01:37:08.398089 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392842 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 01:37:08.398089 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392845 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 01:37:08.398089 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392848 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 01:37:08.398089 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392851 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 01:37:08.398089 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392854 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 01:37:08.398089 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392856 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 01:37:08.398089 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392859 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 01:37:08.398481 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.392864 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 01:37:08.398481 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392989 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 01:37:08.398481 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392995 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 01:37:08.398481 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.392998 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 01:37:08.398481 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393000 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 01:37:08.398481 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393003 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 01:37:08.398481 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393006 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 01:37:08.398481 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393009 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 01:37:08.398481 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393011 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 01:37:08.398481 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393014 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 01:37:08.398481 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393016 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 01:37:08.398481 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393020 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 01:37:08.398481 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393042 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 01:37:08.398481 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393047 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 01:37:08.398481 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393050 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 01:37:08.398848 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393052 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 01:37:08.398848 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393055 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 01:37:08.398848 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393058 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 01:37:08.398848 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393060 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 01:37:08.398848 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393063 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 01:37:08.398848 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393065 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 01:37:08.398848 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393068 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 01:37:08.398848 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393070 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 01:37:08.398848 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393073 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 01:37:08.398848 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393077 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 01:37:08.398848 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393079 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 01:37:08.398848 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393083 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 01:37:08.398848 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393086 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 01:37:08.398848 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393089 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 22 01:37:08.398848 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393092 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 01:37:08.398848 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393094 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 01:37:08.398848 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393097 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 01:37:08.398848 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393099 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 01:37:08.398848 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393102 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 01:37:08.399329 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393104 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 01:37:08.399329 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393106 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 01:37:08.399329 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393109 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 01:37:08.399329 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393111 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 01:37:08.399329 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393115 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 01:37:08.399329 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393118 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 01:37:08.399329 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393121 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 01:37:08.399329 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393124 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 01:37:08.399329 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393126 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 01:37:08.399329 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393128 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 01:37:08.399329 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393131 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 01:37:08.399329 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393133 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 01:37:08.399329 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393136 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 01:37:08.399329 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393138 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 01:37:08.399329 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393140 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 01:37:08.399329 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393143 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 01:37:08.399329 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393145 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 01:37:08.399329 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393147 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 01:37:08.399329 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393150 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 01:37:08.399329 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393152 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 01:37:08.399843 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393154 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 01:37:08.399843 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393157 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 01:37:08.399843 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393159 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 01:37:08.399843 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393162 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 01:37:08.399843 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393165 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 01:37:08.399843 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393168 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 01:37:08.399843 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393170 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 01:37:08.399843 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393172 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 01:37:08.399843 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393175 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 01:37:08.399843 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393178 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 01:37:08.399843 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393180 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 01:37:08.399843 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393183 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 01:37:08.399843 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393185 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 01:37:08.399843 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393187 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 01:37:08.399843 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393190 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 01:37:08.399843 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393192 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 01:37:08.399843 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393194 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 01:37:08.399843 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393197 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 01:37:08.399843 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393199 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 01:37:08.399843 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393201 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 01:37:08.400445 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393204 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 01:37:08.400445 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393206 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 01:37:08.400445 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393208 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 01:37:08.400445 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393211 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 01:37:08.400445 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393213 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 01:37:08.400445 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393215 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 01:37:08.400445 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393218 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 01:37:08.400445 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393220 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 01:37:08.400445 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393222 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 01:37:08.400445 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393224 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 01:37:08.400445 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393227 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 01:37:08.400445 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393229 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 01:37:08.400445 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393231 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 01:37:08.400445 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393234 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 01:37:08.400445 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393236 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 01:37:08.400445 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393239 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 01:37:08.400445 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393241 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 01:37:08.400445 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393243 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 01:37:08.400445 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393246 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 01:37:08.400445 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393248 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 01:37:08.400915 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393250 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 01:37:08.400915 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393253 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 01:37:08.400915 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393255 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 01:37:08.400915 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393258 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 01:37:08.400915 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393260 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 01:37:08.400915 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:08.393263 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 01:37:08.400915 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.393267 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 01:37:08.400915 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.393369 2561 server.go:962] "Client rotation is on, will bootstrap in background" Jul 22 01:37:08.400915 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.395351 2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 22 01:37:08.400915 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.396879 2561 server.go:1019] "Starting client certificate rotation" Jul 22 01:37:08.400915 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.396982 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 01:37:08.400915 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.397036 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 01:37:08.400915 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.400348 2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 01:37:08.402097 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.402056 2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 01:37:08.414081 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.414060 2561 log.go:25] "Validated CRI v1 runtime API" Jul 22 01:37:08.418296 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.418280 2561 log.go:25] "Validated CRI v1 image API" Jul 22 01:37:08.419195 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.419179 2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 22 01:37:08.423698 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.423671 2561 fs.go:135] Filesystem UUIDs: map[56bf91d1-e52b-4454-8a14-4cc933713cc8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b96051b8-b56d-4b8a-909a-f2fb1e95f2f0:/dev/nvme0n1p3] Jul 22 01:37:08.423782 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.423698 2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 22 01:37:08.428600 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.428487 2561 manager.go:216] Machine: {Timestamp:2026-07-22 01:37:08.427232875 +0000 UTC m=+0.341704904 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108684 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2046b062a53ca139e70b57f8b6673b SystemUUID:ec2046b0-62a5-3ca1-39e7-0b57f8b6673b BootID:50ed3b1a-9a09-47b5-9f6f-660074f6a6e4 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0e:ba:ad:68:33 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0e:ba:ad:68:33 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:35:77:fc:c1:cd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 22 01:37:08.428600 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.428594 2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 22 01:37:08.428760 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.428690 2561 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 22 01:37:08.432538 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.432510 2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 22 01:37:08.432670 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.432543 2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-56.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 22 01:37:08.433805 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.433795 2561 topology_manager.go:138] "Creating topology manager with none policy" Jul 22 01:37:08.433850 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.433808 2561 container_manager_linux.go:309] "Creating device plugin manager" Jul 22 01:37:08.433850 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.433816 2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 01:37:08.434534 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.434524 2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 01:37:08.434868 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.434852 2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 01:37:08.435866 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.435855 2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 22 01:37:08.438745 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.438733 2561 state_mem.go:36] "Initialized new in-memory state store" Jul 22 01:37:08.438910 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.438900 2561 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 22 01:37:08.438952 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.438929 2561 kubelet.go:486] "Attempting to sync node with API server" Jul 22 01:37:08.438952 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.438944 2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 22 01:37:08.439045 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.438958 2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 22 01:37:08.439045 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.438967 2561 kubelet.go:398] "Adding apiserver pod source" Jul 22 01:37:08.439045 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.438976 2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 22 01:37:08.439364 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.439353 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 01:37:08.439401 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.439371 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 01:37:08.441732 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.441717 2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 22 01:37:08.443632 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.443618 2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 22 01:37:08.443689 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.443649 2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 22 01:37:08.444767 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.444756 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 22 01:37:08.444811 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.444772 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 22 01:37:08.444811 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.444778 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 22 01:37:08.444811 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.444785 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 22 01:37:08.444811 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.444797 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 22 01:37:08.444811 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.444805 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 22 01:37:08.444811 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.444813 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 22 01:37:08.444955 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.444818 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 22 01:37:08.444955 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.444827 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 22 01:37:08.444955 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.444832 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 22 01:37:08.444955 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.444845 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 22 01:37:08.444955 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.444850 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 22 01:37:08.446353 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.446338 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 22 01:37:08.446353 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.446351 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 22 01:37:08.447921 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:08.447901 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 22 01:37:08.447989 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.447971 2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-56.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 22 01:37:08.448125 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:08.448106 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-56.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 22 01:37:08.449394 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.449382 2561 server.go:1268] "Started kubelet" Jul 22 01:37:08.449505 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.449481 2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 22 01:37:08.449623 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.449565 2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 22 01:37:08.449683 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.449640 2561 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 22 01:37:08.450266 ip-10-0-131-56 systemd[1]: Started Kubernetes Kubelet. Jul 22 01:37:08.451189 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.451129 2561 server.go:310] "Adding debug handlers to kubelet server" Jul 22 01:37:08.455727 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.455700 2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 22 01:37:08.456869 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.456844 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d6pnl" Jul 22 01:37:08.457900 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:08.457872 2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 22 01:37:08.458790 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:08.457947 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-56.ec2.internal.18c478d9a0f7458d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-56.ec2.internal,UID:ip-10-0-131-56.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-56.ec2.internal,},FirstTimestamp:2026-07-22 01:37:08.449355149 +0000 UTC m=+0.363827160,LastTimestamp:2026-07-22 01:37:08.449355149 +0000 UTC m=+0.363827160,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-56.ec2.internal,}" Jul 22 01:37:08.462113 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.462098 2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 22 01:37:08.462195 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.462105 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 22 01:37:08.462515 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:08.462483 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-56.ec2.internal\" not found" Jul 22 01:37:08.463145 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.463124 2561 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 22 01:37:08.463145 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.463145 2561 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 22 01:37:08.463273 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.463218 2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 22 01:37:08.463273 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.463226 2561 reconstruct.go:97] "Volume reconstruction finished" Jul 22 01:37:08.463273 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.463237 2561 reconciler.go:29] "Reconciler: start to sync state" Jul 22 01:37:08.463273 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.463253 2561 factory.go:55] Registering systemd factory Jul 22 01:37:08.463477 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.463293 2561 factory.go:223] Registration of the systemd container factory successfully Jul 22 01:37:08.463550 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.463537 2561 factory.go:153] Registering CRI-O factory Jul 22 01:37:08.463593 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.463556 2561 factory.go:223] Registration of the crio container factory successfully Jul 22 01:37:08.463642 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.463627 2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 22 01:37:08.463707 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.463655 2561 factory.go:103] Registering Raw factory Jul 22 01:37:08.463707 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.463671 2561 manager.go:1196] Started watching for new ooms in manager Jul 22 01:37:08.464198 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.464183 2561 manager.go:349] Starting recovery of all containers Jul 22 01:37:08.468306 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.468283 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d6pnl" Jul 22 01:37:08.473942 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:08.473911 2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-56.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 22 01:37:08.473942 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:08.473921 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 22 01:37:08.474205 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.474192 2561 manager.go:354] Recovery completed Jul 22 01:37:08.478005 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.477905 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 01:37:08.479725 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.479705 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-56.ec2.internal" event="NodeHasSufficientMemory" Jul 22 01:37:08.479804 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.479743 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-56.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 01:37:08.479804 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.479760 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-56.ec2.internal" event="NodeHasSufficientPID" Jul 22 01:37:08.480379 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.480365 2561 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 22 01:37:08.480379 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.480378 2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 22 01:37:08.480479 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.480392 2561 state_mem.go:36] "Initialized new in-memory state store" Jul 22 01:37:08.480479 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.480379 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 01:37:08.481248 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.481229 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-56.ec2.internal" event="NodeHasSufficientMemory" Jul 22 01:37:08.481299 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.481266 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-56.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 01:37:08.481299 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.481288 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-56.ec2.internal" event="NodeHasSufficientPID" Jul 22 01:37:08.481660 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.481644 2561 policy_none.go:49] "None policy: Start" Jul 22 01:37:08.481660 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.481662 2561 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 22 01:37:08.481771 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.481672 2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 22 01:37:08.482640 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.482630 2561 policy_none.go:47] "Start" Jul 22 01:37:08.514616 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.514601 2561 manager.go:337] "Starting Device Plugin manager" Jul 22 01:37:08.514702 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:08.514637 2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 22 01:37:08.514702 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.514649 2561 server.go:87] "Starting device plugin registration server" Jul 22 01:37:08.514879 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.514866 2561 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 22 01:37:08.514939 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.514881 2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 22 01:37:08.514989 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.514978 2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 22 01:37:08.515110 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.515094 2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 22 01:37:08.515170 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.515115 2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 22 01:37:08.516071 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:08.515961 2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 22 01:37:08.516071 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:08.516063 2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-56.ec2.internal\" not found" Jul 22 01:37:08.595231 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.595207 2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 22 01:37:08.596478 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.596462 2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 22 01:37:08.596533 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.596506 2561 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 22 01:37:08.596533 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.596526 2561 kubelet.go:2448] "Starting kubelet main sync loop" Jul 22 01:37:08.596620 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:08.596560 2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 22 01:37:08.599042 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.599008 2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 01:37:08.615968 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.615943 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 01:37:08.617976 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.617960 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-56.ec2.internal" event="NodeHasSufficientMemory" Jul 22 01:37:08.618063 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.617988 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-56.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 01:37:08.618063 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.617998 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-56.ec2.internal" event="NodeHasSufficientPID" Jul 22 01:37:08.618063 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.618019 2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-56.ec2.internal" Jul 22 01:37:08.624572 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.624558 2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-56.ec2.internal" Jul 22 01:37:08.624654 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:08.624580 2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-56.ec2.internal\": node \"ip-10-0-131-56.ec2.internal\" not found" Jul 22 01:37:08.640899 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:08.640852 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-56.ec2.internal\" not found" Jul 22 01:37:08.697667 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.697642 2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-56.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-56.ec2.internal"] Jul 22 01:37:08.697761 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.697702 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 01:37:08.698520 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.698505 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-56.ec2.internal" event="NodeHasSufficientMemory" Jul 22 01:37:08.698575 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.698536 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-56.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 01:37:08.698575 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.698546 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-56.ec2.internal" event="NodeHasSufficientPID" Jul 22 01:37:08.699927 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.699913 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 01:37:08.700085 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.700070 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-56.ec2.internal" Jul 22 01:37:08.700151 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.700107 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 01:37:08.700615 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.700600 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-56.ec2.internal" event="NodeHasSufficientMemory" Jul 22 01:37:08.700698 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.700630 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-56.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 01:37:08.700698 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.700644 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-56.ec2.internal" event="NodeHasSufficientPID" Jul 22 01:37:08.700698 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.700600 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-56.ec2.internal" event="NodeHasSufficientMemory" Jul 22 01:37:08.700798 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.700705 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-56.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 01:37:08.700798 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.700719 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-56.ec2.internal" event="NodeHasSufficientPID" Jul 22 01:37:08.702708 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.702691 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-56.ec2.internal" Jul 22 01:37:08.702771 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.702714 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 01:37:08.703347 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.703332 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-56.ec2.internal" event="NodeHasSufficientMemory" Jul 22 01:37:08.703409 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.703366 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-56.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 01:37:08.703409 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.703381 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-56.ec2.internal" event="NodeHasSufficientPID" Jul 22 01:37:08.732621 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:08.732599 2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-56.ec2.internal\" not found" node="ip-10-0-131-56.ec2.internal" Jul 22 01:37:08.735958 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:08.735944 2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-56.ec2.internal\" not found" node="ip-10-0-131-56.ec2.internal" Jul 22 01:37:08.741383 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:08.741369 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-56.ec2.internal\" not found" Jul 22 01:37:08.764016 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.763986 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4419f2dab53b7f8620a2511c1a87c5c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-56.ec2.internal\" (UID: \"4419f2dab53b7f8620a2511c1a87c5c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-56.ec2.internal" Jul 22 01:37:08.764107 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.764043 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4419f2dab53b7f8620a2511c1a87c5c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-56.ec2.internal\" (UID: \"4419f2dab53b7f8620a2511c1a87c5c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-56.ec2.internal" Jul 22 01:37:08.764107 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.764068 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8f30de953d78a535da92683887f7e533-config\") pod \"kube-apiserver-proxy-ip-10-0-131-56.ec2.internal\" (UID: \"8f30de953d78a535da92683887f7e533\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-56.ec2.internal" Jul 22 01:37:08.841442 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:08.841422 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-56.ec2.internal\" not found" Jul 22 01:37:08.864765 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.864737 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4419f2dab53b7f8620a2511c1a87c5c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-56.ec2.internal\" (UID: \"4419f2dab53b7f8620a2511c1a87c5c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-56.ec2.internal" Jul 22 01:37:08.864828 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.864774 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4419f2dab53b7f8620a2511c1a87c5c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-56.ec2.internal\" (UID: \"4419f2dab53b7f8620a2511c1a87c5c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-56.ec2.internal" Jul 22 01:37:08.864828 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.864790 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8f30de953d78a535da92683887f7e533-config\") pod \"kube-apiserver-proxy-ip-10-0-131-56.ec2.internal\" (UID: \"8f30de953d78a535da92683887f7e533\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-56.ec2.internal" Jul 22 01:37:08.864901 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.864837 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4419f2dab53b7f8620a2511c1a87c5c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-56.ec2.internal\" (UID: \"4419f2dab53b7f8620a2511c1a87c5c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-56.ec2.internal" Jul 22 01:37:08.864901 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.864859 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8f30de953d78a535da92683887f7e533-config\") pod \"kube-apiserver-proxy-ip-10-0-131-56.ec2.internal\" (UID: \"8f30de953d78a535da92683887f7e533\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-56.ec2.internal" Jul 22 01:37:08.864901 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:08.864836 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4419f2dab53b7f8620a2511c1a87c5c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-56.ec2.internal\" (UID: \"4419f2dab53b7f8620a2511c1a87c5c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-56.ec2.internal" Jul 22 01:37:08.942170 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:08.942120 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-56.ec2.internal\" not found" Jul 22 01:37:09.037599 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.037574 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-56.ec2.internal" Jul 22 01:37:09.038695 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.038669 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-56.ec2.internal" Jul 22 01:37:09.042442 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:09.042428 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-56.ec2.internal\" not found" Jul 22 01:37:09.142797 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:09.142770 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-56.ec2.internal\" not found" Jul 22 01:37:09.243371 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:09.243309 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-56.ec2.internal\" not found" Jul 22 01:37:09.277116 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.277098 2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 01:37:09.335114 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.335092 2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 01:37:09.363065 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.363041 2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-56.ec2.internal" Jul 22 01:37:09.374262 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.374244 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 01:37:09.376335 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.376323 2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-56.ec2.internal" Jul 22 01:37:09.388739 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.388722 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 01:37:09.395912 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.395900 2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 22 01:37:09.396002 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.395987 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 01:37:09.396074 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.396043 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 01:37:09.396074 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.396051 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 01:37:09.439163 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.439148 2561 apiserver.go:52] "Watching apiserver" Jul 22 01:37:09.452747 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.452724 2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 22 01:37:09.453079 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.453061 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-cpk88","openshift-multus/multus-c4c5n","openshift-dns/node-resolver-92dh5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-56.ec2.internal","openshift-multus/network-metrics-daemon-j4s4b","openshift-network-diagnostics/network-check-target-fxfmm","openshift-network-operator/iptables-alerter-7qn5n","openshift-ovn-kubernetes/ovnkube-node-lcjb6","kube-system/global-pull-secret-syncer-ghl8b","kube-system/konnectivity-agent-gzgpx","kube-system/kube-apiserver-proxy-ip-10-0-131-56.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk","openshift-cluster-node-tuning-operator/tuned-bbflh","openshift-image-registry/node-ca-j78lg"] Jul 22 01:37:09.454391 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.454378 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7qn5n" Jul 22 01:37:09.455633 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.455610 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.456508 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.456493 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-29tgn\"" Jul 22 01:37:09.456994 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.456973 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.457111 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.457055 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 22 01:37:09.457111 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.457058 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 22 01:37:09.457230 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.457061 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 22 01:37:09.457568 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.457554 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 22 01:37:09.457861 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.457847 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 22 01:37:09.457901 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.457884 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 22 01:37:09.457935 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.457919 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-s8r7s\"" Jul 22 01:37:09.458149 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.458132 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 22 01:37:09.458302 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.458290 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cpk88" Jul 22 01:37:09.459496 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.459456 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:37:09.459580 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:09.459556 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxfmm" podUID="271b4b39-8156-44e0-a9fb-c937501425fa" Jul 22 01:37:09.460976 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.460944 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:09.461095 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:09.461060 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j4s4b" podUID="f18aa922-9c77-4466-aad7-6e2f5fe12c59" Jul 22 01:37:09.462849 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.462827 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 22 01:37:09.464131 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.463244 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 22 01:37:09.464131 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.463423 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f9mlk\"" Jul 22 01:37:09.464131 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.463561 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 22 01:37:09.464131 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.463569 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 22 01:37:09.464131 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.463617 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-92dh5" Jul 22 01:37:09.464131 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.463687 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 22 01:37:09.464131 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.464020 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 22 01:37:09.464574 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.463632 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f2jvm\"" Jul 22 01:37:09.465136 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.465119 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 22 01:37:09.465364 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.465345 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 22 01:37:09.465453 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.465367 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 22 01:37:09.465565 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.465550 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:09.465628 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:09.465609 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghl8b" podUID="448e76a1-df97-4a26-8433-7a51cdeb4a5a" Jul 22 01:37:09.466421 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.466407 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 22 01:37:09.466911 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.466896 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gzgpx" Jul 22 01:37:09.467141 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467119 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e6ceea52-1124-417d-92ff-53d3f054bded-ovnkube-script-lib\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.467211 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467158 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d9fa62a-a81a-4bdc-893e-5928340b007b-system-cni-dir\") pod \"multus-additional-cni-plugins-cpk88\" (UID: \"4d9fa62a-a81a-4bdc-893e-5928340b007b\") " pod="openshift-multus/multus-additional-cni-plugins-cpk88" Jul 22 01:37:09.467211 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467185 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jglhg\" (UniqueName: \"kubernetes.io/projected/271b4b39-8156-44e0-a9fb-c937501425fa-kube-api-access-jglhg\") pod \"network-check-target-fxfmm\" (UID: \"271b4b39-8156-44e0-a9fb-c937501425fa\") " pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:37:09.467299 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467210 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-cnibin\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.467299 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467232 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-multus-conf-dir\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.467299 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467276 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-run-ovn\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.467441 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467310 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/448e76a1-df97-4a26-8433-7a51cdeb4a5a-kubelet-config\") pod \"global-pull-secret-syncer-ghl8b\" (UID: \"448e76a1-df97-4a26-8433-7a51cdeb4a5a\") " pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:09.467441 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467339 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/448e76a1-df97-4a26-8433-7a51cdeb4a5a-original-pull-secret\") pod \"global-pull-secret-syncer-ghl8b\" (UID: \"448e76a1-df97-4a26-8433-7a51cdeb4a5a\") " pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:09.467441 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467362 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-host-run-k8s-cni-cncf-io\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.467441 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467386 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-host-run-multus-certs\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.467441 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467421 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-systemd-units\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.467643 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467451 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d9fa62a-a81a-4bdc-893e-5928340b007b-cnibin\") pod \"multus-additional-cni-plugins-cpk88\" (UID: \"4d9fa62a-a81a-4bdc-893e-5928340b007b\") " pod="openshift-multus/multus-additional-cni-plugins-cpk88" Jul 22 01:37:09.467643 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467479 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjwfk\" (UniqueName: \"kubernetes.io/projected/6a4a5252-c595-42f9-834a-0c26e27adaf8-kube-api-access-sjwfk\") pod \"node-resolver-92dh5\" (UID: \"6a4a5252-c595-42f9-834a-0c26e27adaf8\") " pod="openshift-dns/node-resolver-92dh5" Jul 22 01:37:09.467643 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467500 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-multus-socket-dir-parent\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.467643 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467515 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-var-lib-openvswitch\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.467643 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467539 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-host-slash\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.467643 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467572 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6a4a5252-c595-42f9-834a-0c26e27adaf8-hosts-file\") pod \"node-resolver-92dh5\" (UID: \"6a4a5252-c595-42f9-834a-0c26e27adaf8\") " pod="openshift-dns/node-resolver-92dh5" Jul 22 01:37:09.467643 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467582 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 22 01:37:09.467643 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467609 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/deed9faf-5432-4522-aa0d-362817d03ad1-iptables-alerter-script\") pod \"iptables-alerter-7qn5n\" (UID: \"deed9faf-5432-4522-aa0d-362817d03ad1\") " pod="openshift-network-operator/iptables-alerter-7qn5n" Jul 22 01:37:09.467936 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467652 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f8bfk\"" Jul 22 01:37:09.467936 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467678 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-system-cni-dir\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.467936 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467717 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d9fa62a-a81a-4bdc-893e-5928340b007b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cpk88\" (UID: \"4d9fa62a-a81a-4bdc-893e-5928340b007b\") " pod="openshift-multus/multus-additional-cni-plugins-cpk88" Jul 22 01:37:09.467936 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467741 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e6ceea52-1124-417d-92ff-53d3f054bded-ovn-node-metrics-cert\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.467936 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467766 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fphnt\" (UniqueName: \"kubernetes.io/projected/4d9fa62a-a81a-4bdc-893e-5928340b007b-kube-api-access-fphnt\") pod \"multus-additional-cni-plugins-cpk88\" (UID: \"4d9fa62a-a81a-4bdc-893e-5928340b007b\") " pod="openshift-multus/multus-additional-cni-plugins-cpk88" Jul 22 01:37:09.467936 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467791 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxsf6\" (UniqueName: \"kubernetes.io/projected/f18aa922-9c77-4466-aad7-6e2f5fe12c59-kube-api-access-jxsf6\") pod \"network-metrics-daemon-j4s4b\" (UID: \"f18aa922-9c77-4466-aad7-6e2f5fe12c59\") " pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:09.467936 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467818 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a4a5252-c595-42f9-834a-0c26e27adaf8-tmp-dir\") pod \"node-resolver-92dh5\" (UID: \"6a4a5252-c595-42f9-834a-0c26e27adaf8\") " pod="openshift-dns/node-resolver-92dh5" Jul 22 01:37:09.467936 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467832 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/448e76a1-df97-4a26-8433-7a51cdeb4a5a-dbus\") pod \"global-pull-secret-syncer-ghl8b\" (UID: \"448e76a1-df97-4a26-8433-7a51cdeb4a5a\") " pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:09.467936 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467845 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-host-var-lib-kubelet\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.467936 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467861 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-etc-openvswitch\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.467936 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467893 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4d9fa62a-a81a-4bdc-893e-5928340b007b-os-release\") pod \"multus-additional-cni-plugins-cpk88\" (UID: \"4d9fa62a-a81a-4bdc-893e-5928340b007b\") " pod="openshift-multus/multus-additional-cni-plugins-cpk88" Jul 22 01:37:09.467936 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467915 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d9fa62a-a81a-4bdc-893e-5928340b007b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cpk88\" (UID: \"4d9fa62a-a81a-4bdc-893e-5928340b007b\") " pod="openshift-multus/multus-additional-cni-plugins-cpk88" Jul 22 01:37:09.467936 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467933 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23698c81-8626-4b0e-963b-e20fde992ed9-cni-binary-copy\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.468509 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467954 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-run-openvswitch\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.468509 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.467977 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-log-socket\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.468509 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.468001 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e6ceea52-1124-417d-92ff-53d3f054bded-env-overrides\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.468509 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.468055 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/deed9faf-5432-4522-aa0d-362817d03ad1-host-slash\") pod \"iptables-alerter-7qn5n\" (UID: \"deed9faf-5432-4522-aa0d-362817d03ad1\") " pod="openshift-network-operator/iptables-alerter-7qn5n" Jul 22 01:37:09.468509 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.468083 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-multus-cni-dir\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.468509 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.468111 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-os-release\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.468509 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.468133 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-host-var-lib-cni-bin\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.468509 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.468160 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-node-log\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.468509 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.468183 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d9fa62a-a81a-4bdc-893e-5928340b007b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cpk88\" (UID: \"4d9fa62a-a81a-4bdc-893e-5928340b007b\") " pod="openshift-multus/multus-additional-cni-plugins-cpk88" Jul 22 01:37:09.468509 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.468206 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/23698c81-8626-4b0e-963b-e20fde992ed9-multus-daemon-config\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.468509 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.468234 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-etc-kubernetes\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.468509 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.468255 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-host-cni-bin\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.468509 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.468279 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxfgh\" (UniqueName: \"kubernetes.io/projected/e6ceea52-1124-417d-92ff-53d3f054bded-kube-api-access-cxfgh\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.468509 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.468349 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d9fa62a-a81a-4bdc-893e-5928340b007b-cni-binary-copy\") pod \"multus-additional-cni-plugins-cpk88\" (UID: \"4d9fa62a-a81a-4bdc-893e-5928340b007b\") " pod="openshift-multus/multus-additional-cni-plugins-cpk88" Jul 22 01:37:09.468509 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.468376 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f18aa922-9c77-4466-aad7-6e2f5fe12c59-metrics-certs\") pod \"network-metrics-daemon-j4s4b\" (UID: \"f18aa922-9c77-4466-aad7-6e2f5fe12c59\") " pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:09.468509 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.468390 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk" Jul 22 01:37:09.468509 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.468403 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-host-var-lib-cni-multus\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.469159 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.468426 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-host-kubelet\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.469159 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.468449 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-run-systemd\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.469159 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.468476 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-host-run-ovn-kubernetes\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.469159 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.468515 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.469159 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.468547 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e6ceea52-1124-417d-92ff-53d3f054bded-ovnkube-config\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.469159 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.468573 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdc72\" (UniqueName: \"kubernetes.io/projected/23698c81-8626-4b0e-963b-e20fde992ed9-kube-api-access-tdc72\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.469159 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.468595 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59965\" (UniqueName: \"kubernetes.io/projected/deed9faf-5432-4522-aa0d-362817d03ad1-kube-api-access-59965\") pod \"iptables-alerter-7qn5n\" (UID: \"deed9faf-5432-4522-aa0d-362817d03ad1\") " pod="openshift-network-operator/iptables-alerter-7qn5n" Jul 22 01:37:09.469159 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.468625 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-host-run-netns\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.469159 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.468647 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-hostroot\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.469159 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.468663 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-host-run-netns\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.469159 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.468676 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-host-cni-netd\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.469159 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.468843 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 22 01:37:09.469159 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.469165 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 22 01:37:09.469587 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.469207 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vhfhg\"" Jul 22 01:37:09.469924 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.469909 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:09.470633 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.470552 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 22 01:37:09.470633 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.470624 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 22 01:37:09.471442 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.470647 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qmr5c\"" Jul 22 01:37:09.471442 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.470929 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 22 01:37:09.471520 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.471478 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j78lg" Jul 22 01:37:09.471853 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.471809 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 01:32:09 +0000 UTC" deadline="2028-04-24 06:36:48.509780946 +0000 UTC" Jul 22 01:37:09.471853 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.471850 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15412h59m39.037934755s" Jul 22 01:37:09.472060 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.472045 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 22 01:37:09.472107 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.472045 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6hgjf\"" Jul 22 01:37:09.472139 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.472111 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 22 01:37:09.473472 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.473457 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 22 01:37:09.473649 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.473636 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 22 01:37:09.473649 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.473643 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mgwrh\"" Jul 22 01:37:09.473928 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.473913 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 22 01:37:09.477926 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.477911 2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 01:37:09.498792 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.498743 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-l9whc" Jul 22 01:37:09.506387 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.506373 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-l9whc" Jul 22 01:37:09.566876 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.565767 2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 22 01:37:09.568869 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.568852 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-host-slash\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.568949 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.568878 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6a4a5252-c595-42f9-834a-0c26e27adaf8-hosts-file\") pod \"node-resolver-92dh5\" (UID: \"6a4a5252-c595-42f9-834a-0c26e27adaf8\") " pod="openshift-dns/node-resolver-92dh5" Jul 22 01:37:09.568949 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.568898 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/deed9faf-5432-4522-aa0d-362817d03ad1-iptables-alerter-script\") pod \"iptables-alerter-7qn5n\" (UID: \"deed9faf-5432-4522-aa0d-362817d03ad1\") " pod="openshift-network-operator/iptables-alerter-7qn5n" Jul 22 01:37:09.568949 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.568920 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/16abf4b8-914b-4259-b8f5-e6e51f112359-etc-tuned\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:09.568949 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.568934 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16abf4b8-914b-4259-b8f5-e6e51f112359-tmp\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:09.568949 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.568949 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a184b54b-d2a6-4c78-aea6-714b9f0a6a1f-serviceca\") pod \"node-ca-j78lg\" (UID: \"a184b54b-d2a6-4c78-aea6-714b9f0a6a1f\") " pod="openshift-image-registry/node-ca-j78lg" Jul 22 01:37:09.569171 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.568973 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-system-cni-dir\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.569171 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.568983 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-host-slash\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.569171 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.568990 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6a4a5252-c595-42f9-834a-0c26e27adaf8-hosts-file\") pod \"node-resolver-92dh5\" (UID: \"6a4a5252-c595-42f9-834a-0c26e27adaf8\") " pod="openshift-dns/node-resolver-92dh5" Jul 22 01:37:09.569171 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569012 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d9fa62a-a81a-4bdc-893e-5928340b007b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cpk88\" (UID: \"4d9fa62a-a81a-4bdc-893e-5928340b007b\") " pod="openshift-multus/multus-additional-cni-plugins-cpk88" Jul 22 01:37:09.569171 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569045 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-system-cni-dir\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.569171 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569059 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e-device-dir\") pod \"aws-ebs-csi-driver-node-mghsk\" (UID: \"fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk" Jul 22 01:37:09.569171 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569082 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e6ceea52-1124-417d-92ff-53d3f054bded-ovn-node-metrics-cert\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.569171 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569098 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fphnt\" (UniqueName: \"kubernetes.io/projected/4d9fa62a-a81a-4bdc-893e-5928340b007b-kube-api-access-fphnt\") pod \"multus-additional-cni-plugins-cpk88\" (UID: \"4d9fa62a-a81a-4bdc-893e-5928340b007b\") " pod="openshift-multus/multus-additional-cni-plugins-cpk88" Jul 22 01:37:09.569171 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569114 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxsf6\" (UniqueName: \"kubernetes.io/projected/f18aa922-9c77-4466-aad7-6e2f5fe12c59-kube-api-access-jxsf6\") pod \"network-metrics-daemon-j4s4b\" (UID: \"f18aa922-9c77-4466-aad7-6e2f5fe12c59\") " pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:09.569171 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569135 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a4a5252-c595-42f9-834a-0c26e27adaf8-tmp-dir\") pod \"node-resolver-92dh5\" (UID: \"6a4a5252-c595-42f9-834a-0c26e27adaf8\") " pod="openshift-dns/node-resolver-92dh5" Jul 22 01:37:09.569171 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569160 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/448e76a1-df97-4a26-8433-7a51cdeb4a5a-dbus\") pod \"global-pull-secret-syncer-ghl8b\" (UID: \"448e76a1-df97-4a26-8433-7a51cdeb4a5a\") " pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:09.569659 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569181 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-run\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:09.569659 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569206 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-var-lib-kubelet\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:09.569659 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569228 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-host-var-lib-kubelet\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.569659 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569258 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-etc-openvswitch\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.569659 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569298 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4d9fa62a-a81a-4bdc-893e-5928340b007b-os-release\") pod \"multus-additional-cni-plugins-cpk88\" (UID: \"4d9fa62a-a81a-4bdc-893e-5928340b007b\") " pod="openshift-multus/multus-additional-cni-plugins-cpk88" Jul 22 01:37:09.569659 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569322 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d9fa62a-a81a-4bdc-893e-5928340b007b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cpk88\" (UID: \"4d9fa62a-a81a-4bdc-893e-5928340b007b\") " pod="openshift-multus/multus-additional-cni-plugins-cpk88" Jul 22 01:37:09.569659 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569324 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-host-var-lib-kubelet\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.569659 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569345 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23698c81-8626-4b0e-963b-e20fde992ed9-cni-binary-copy\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.569659 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569364 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/448e76a1-df97-4a26-8433-7a51cdeb4a5a-dbus\") pod \"global-pull-secret-syncer-ghl8b\" (UID: \"448e76a1-df97-4a26-8433-7a51cdeb4a5a\") " pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:09.569659 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569362 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-etc-openvswitch\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.569659 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569390 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4d9fa62a-a81a-4bdc-893e-5928340b007b-os-release\") pod \"multus-additional-cni-plugins-cpk88\" (UID: \"4d9fa62a-a81a-4bdc-893e-5928340b007b\") " pod="openshift-multus/multus-additional-cni-plugins-cpk88" Jul 22 01:37:09.569659 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569367 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-run-openvswitch\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.569659 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569416 2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 22 01:37:09.569659 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569425 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a4a5252-c595-42f9-834a-0c26e27adaf8-tmp-dir\") pod \"node-resolver-92dh5\" (UID: \"6a4a5252-c595-42f9-834a-0c26e27adaf8\") " pod="openshift-dns/node-resolver-92dh5" Jul 22 01:37:09.569659 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569438 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-log-socket\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.569659 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569465 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e6ceea52-1124-417d-92ff-53d3f054bded-env-overrides\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.569659 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569434 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-run-openvswitch\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.569659 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569500 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d9fa62a-a81a-4bdc-893e-5928340b007b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cpk88\" (UID: \"4d9fa62a-a81a-4bdc-893e-5928340b007b\") " pod="openshift-multus/multus-additional-cni-plugins-cpk88" Jul 22 01:37:09.570545 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569488 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/deed9faf-5432-4522-aa0d-362817d03ad1-host-slash\") pod \"iptables-alerter-7qn5n\" (UID: \"deed9faf-5432-4522-aa0d-362817d03ad1\") " pod="openshift-network-operator/iptables-alerter-7qn5n" Jul 22 01:37:09.570545 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569531 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/deed9faf-5432-4522-aa0d-362817d03ad1-host-slash\") pod \"iptables-alerter-7qn5n\" (UID: \"deed9faf-5432-4522-aa0d-362817d03ad1\") " pod="openshift-network-operator/iptables-alerter-7qn5n" Jul 22 01:37:09.570545 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569535 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/50a4581a-21ba-4900-aee7-648e18e7abe5-konnectivity-ca\") pod \"konnectivity-agent-gzgpx\" (UID: \"50a4581a-21ba-4900-aee7-648e18e7abe5\") " pod="kube-system/konnectivity-agent-gzgpx" Jul 22 01:37:09.570545 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569572 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-multus-cni-dir\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.570545 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569625 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-log-socket\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.570545 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569678 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-os-release\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.570545 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569690 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-multus-cni-dir\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.570545 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569702 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-host-var-lib-cni-bin\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.570545 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569729 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-node-log\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.570545 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569749 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-os-release\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.570545 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569755 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d9fa62a-a81a-4bdc-893e-5928340b007b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cpk88\" (UID: \"4d9fa62a-a81a-4bdc-893e-5928340b007b\") " pod="openshift-multus/multus-additional-cni-plugins-cpk88" Jul 22 01:37:09.570545 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569784 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmhn5\" (UniqueName: \"kubernetes.io/projected/fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e-kube-api-access-lmhn5\") pod \"aws-ebs-csi-driver-node-mghsk\" (UID: \"fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk" Jul 22 01:37:09.570545 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569788 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-host-var-lib-cni-bin\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.570545 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569808 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-etc-sysconfig\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:09.570545 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569814 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-node-log\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.570545 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569836 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-etc-sysctl-conf\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:09.570545 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569860 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/23698c81-8626-4b0e-963b-e20fde992ed9-multus-daemon-config\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.570545 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569889 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23698c81-8626-4b0e-963b-e20fde992ed9-cni-binary-copy\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.571348 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569907 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e6ceea52-1124-417d-92ff-53d3f054bded-env-overrides\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.571348 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569891 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e-registration-dir\") pod \"aws-ebs-csi-driver-node-mghsk\" (UID: \"fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk" Jul 22 01:37:09.571348 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569950 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e-sys-fs\") pod \"aws-ebs-csi-driver-node-mghsk\" (UID: \"fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk" Jul 22 01:37:09.571348 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569969 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-etc-modprobe-d\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:09.571348 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.569985 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-etc-kubernetes\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.571348 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570050 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-host-cni-bin\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.571348 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570067 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxfgh\" (UniqueName: \"kubernetes.io/projected/e6ceea52-1124-417d-92ff-53d3f054bded-kube-api-access-cxfgh\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.571348 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570076 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-etc-kubernetes\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.571348 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570084 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d9fa62a-a81a-4bdc-893e-5928340b007b-cni-binary-copy\") pod \"multus-additional-cni-plugins-cpk88\" (UID: \"4d9fa62a-a81a-4bdc-893e-5928340b007b\") " pod="openshift-multus/multus-additional-cni-plugins-cpk88" Jul 22 01:37:09.571348 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570118 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f18aa922-9c77-4466-aad7-6e2f5fe12c59-metrics-certs\") pod \"network-metrics-daemon-j4s4b\" (UID: \"f18aa922-9c77-4466-aad7-6e2f5fe12c59\") " pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:09.571348 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570136 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-host-cni-bin\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.571348 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570145 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/50a4581a-21ba-4900-aee7-648e18e7abe5-agent-certs\") pod \"konnectivity-agent-gzgpx\" (UID: \"50a4581a-21ba-4900-aee7-648e18e7abe5\") " pod="kube-system/konnectivity-agent-gzgpx" Jul 22 01:37:09.571348 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570186 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-etc-sysctl-d\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:09.571348 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570215 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-host-var-lib-cni-multus\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.571348 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570239 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-host-kubelet\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.571348 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570262 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-run-systemd\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.571348 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570299 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-host-run-ovn-kubernetes\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.571348 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:09.570325 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:37:09.571978 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570370 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.571978 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570374 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d9fa62a-a81a-4bdc-893e-5928340b007b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cpk88\" (UID: \"4d9fa62a-a81a-4bdc-893e-5928340b007b\") " pod="openshift-multus/multus-additional-cni-plugins-cpk88" Jul 22 01:37:09.571978 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570384 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/deed9faf-5432-4522-aa0d-362817d03ad1-iptables-alerter-script\") pod \"iptables-alerter-7qn5n\" (UID: \"deed9faf-5432-4522-aa0d-362817d03ad1\") " pod="openshift-network-operator/iptables-alerter-7qn5n" Jul 22 01:37:09.571978 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:09.570397 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f18aa922-9c77-4466-aad7-6e2f5fe12c59-metrics-certs podName:f18aa922-9c77-4466-aad7-6e2f5fe12c59 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:10.070379961 +0000 UTC m=+1.984851981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f18aa922-9c77-4466-aad7-6e2f5fe12c59-metrics-certs") pod "network-metrics-daemon-j4s4b" (UID: "f18aa922-9c77-4466-aad7-6e2f5fe12c59") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:37:09.571978 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570323 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.571978 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570421 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e-socket-dir\") pod \"aws-ebs-csi-driver-node-mghsk\" (UID: \"fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk" Jul 22 01:37:09.571978 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570417 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-host-var-lib-cni-multus\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.571978 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570439 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-sys\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:09.571978 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570442 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-run-systemd\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.571978 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570473 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a184b54b-d2a6-4c78-aea6-714b9f0a6a1f-host\") pod \"node-ca-j78lg\" (UID: \"a184b54b-d2a6-4c78-aea6-714b9f0a6a1f\") " pod="openshift-image-registry/node-ca-j78lg" Jul 22 01:37:09.571978 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570482 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-host-run-ovn-kubernetes\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.571978 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570483 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-host-kubelet\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.571978 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570503 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbmz5\" (UniqueName: \"kubernetes.io/projected/a184b54b-d2a6-4c78-aea6-714b9f0a6a1f-kube-api-access-vbmz5\") pod \"node-ca-j78lg\" (UID: \"a184b54b-d2a6-4c78-aea6-714b9f0a6a1f\") " pod="openshift-image-registry/node-ca-j78lg" Jul 22 01:37:09.571978 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570504 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/23698c81-8626-4b0e-963b-e20fde992ed9-multus-daemon-config\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.571978 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570531 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e6ceea52-1124-417d-92ff-53d3f054bded-ovnkube-config\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.571978 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570555 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-etc-systemd\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:09.572476 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570578 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdc72\" (UniqueName: \"kubernetes.io/projected/23698c81-8626-4b0e-963b-e20fde992ed9-kube-api-access-tdc72\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.572476 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570603 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59965\" (UniqueName: \"kubernetes.io/projected/deed9faf-5432-4522-aa0d-362817d03ad1-kube-api-access-59965\") pod \"iptables-alerter-7qn5n\" (UID: \"deed9faf-5432-4522-aa0d-362817d03ad1\") " pod="openshift-network-operator/iptables-alerter-7qn5n" Jul 22 01:37:09.572476 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570624 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-host\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:09.572476 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570646 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7wfk\" (UniqueName: \"kubernetes.io/projected/16abf4b8-914b-4259-b8f5-e6e51f112359-kube-api-access-b7wfk\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:09.572476 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570670 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-host-run-netns\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.572476 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570695 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-hostroot\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.572476 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570718 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-host-run-netns\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.572476 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570742 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-host-cni-netd\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.572476 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570764 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e6ceea52-1124-417d-92ff-53d3f054bded-ovnkube-script-lib\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.572476 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570791 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d9fa62a-a81a-4bdc-893e-5928340b007b-system-cni-dir\") pod \"multus-additional-cni-plugins-cpk88\" (UID: \"4d9fa62a-a81a-4bdc-893e-5928340b007b\") " pod="openshift-multus/multus-additional-cni-plugins-cpk88" Jul 22 01:37:09.572476 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570816 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jglhg\" (UniqueName: \"kubernetes.io/projected/271b4b39-8156-44e0-a9fb-c937501425fa-kube-api-access-jglhg\") pod \"network-check-target-fxfmm\" (UID: \"271b4b39-8156-44e0-a9fb-c937501425fa\") " pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:37:09.572476 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570830 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-host-run-netns\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.572476 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570840 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-cnibin\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.572476 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570858 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d9fa62a-a81a-4bdc-893e-5928340b007b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cpk88\" (UID: \"4d9fa62a-a81a-4bdc-893e-5928340b007b\") " pod="openshift-multus/multus-additional-cni-plugins-cpk88" Jul 22 01:37:09.572476 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570865 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-multus-conf-dir\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.572476 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570897 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-run-ovn\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.572476 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570920 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d9fa62a-a81a-4bdc-893e-5928340b007b-cni-binary-copy\") pod \"multus-additional-cni-plugins-cpk88\" (UID: \"4d9fa62a-a81a-4bdc-893e-5928340b007b\") " pod="openshift-multus/multus-additional-cni-plugins-cpk88" Jul 22 01:37:09.572968 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570921 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/448e76a1-df97-4a26-8433-7a51cdeb4a5a-kubelet-config\") pod \"global-pull-secret-syncer-ghl8b\" (UID: \"448e76a1-df97-4a26-8433-7a51cdeb4a5a\") " pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:09.572968 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570965 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/448e76a1-df97-4a26-8433-7a51cdeb4a5a-original-pull-secret\") pod \"global-pull-secret-syncer-ghl8b\" (UID: \"448e76a1-df97-4a26-8433-7a51cdeb4a5a\") " pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:09.572968 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570969 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/448e76a1-df97-4a26-8433-7a51cdeb4a5a-kubelet-config\") pod \"global-pull-secret-syncer-ghl8b\" (UID: \"448e76a1-df97-4a26-8433-7a51cdeb4a5a\") " pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:09.572968 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.570991 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-lib-modules\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:09.572968 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.571008 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e6ceea52-1124-417d-92ff-53d3f054bded-ovnkube-config\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.572968 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.571020 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d9fa62a-a81a-4bdc-893e-5928340b007b-system-cni-dir\") pod \"multus-additional-cni-plugins-cpk88\" (UID: \"4d9fa62a-a81a-4bdc-893e-5928340b007b\") " pod="openshift-multus/multus-additional-cni-plugins-cpk88" Jul 22 01:37:09.572968 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.571044 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-host-run-k8s-cni-cncf-io\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.572968 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.571070 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-host-run-multus-certs\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.572968 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.571100 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-systemd-units\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.572968 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.571117 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-run-ovn\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.572968 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:09.571121 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 01:37:09.572968 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.571137 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-host-run-k8s-cni-cncf-io\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.572968 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.571153 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-multus-conf-dir\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.572968 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.571176 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-cnibin\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.572968 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:09.571188 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448e76a1-df97-4a26-8433-7a51cdeb4a5a-original-pull-secret podName:448e76a1-df97-4a26-8433-7a51cdeb4a5a nodeName:}" failed. No retries permitted until 2026-07-22 01:37:10.071173106 +0000 UTC m=+1.985645102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/448e76a1-df97-4a26-8433-7a51cdeb4a5a-original-pull-secret") pod "global-pull-secret-syncer-ghl8b" (UID: "448e76a1-df97-4a26-8433-7a51cdeb4a5a") : object "kube-system"/"original-pull-secret" not registered Jul 22 01:37:09.572968 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.571132 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-systemd-units\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.572968 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.571201 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-host-run-multus-certs\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.573455 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.571230 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-host-run-netns\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.573455 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.571232 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d9fa62a-a81a-4bdc-893e-5928340b007b-cnibin\") pod \"multus-additional-cni-plugins-cpk88\" (UID: \"4d9fa62a-a81a-4bdc-893e-5928340b007b\") " pod="openshift-multus/multus-additional-cni-plugins-cpk88" Jul 22 01:37:09.573455 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.571235 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e6ceea52-1124-417d-92ff-53d3f054bded-ovnkube-script-lib\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.573455 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.571270 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjwfk\" (UniqueName: \"kubernetes.io/projected/6a4a5252-c595-42f9-834a-0c26e27adaf8-kube-api-access-sjwfk\") pod \"node-resolver-92dh5\" (UID: \"6a4a5252-c595-42f9-834a-0c26e27adaf8\") " pod="openshift-dns/node-resolver-92dh5" Jul 22 01:37:09.573455 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.571275 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-host-cni-netd\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.573455 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.571289 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d9fa62a-a81a-4bdc-893e-5928340b007b-cnibin\") pod \"multus-additional-cni-plugins-cpk88\" (UID: \"4d9fa62a-a81a-4bdc-893e-5928340b007b\") " pod="openshift-multus/multus-additional-cni-plugins-cpk88" Jul 22 01:37:09.573455 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.571305 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-hostroot\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.573455 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.571319 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e-etc-selinux\") pod \"aws-ebs-csi-driver-node-mghsk\" (UID: \"fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk" Jul 22 01:37:09.573455 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.571442 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-etc-kubernetes\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:09.573455 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.571474 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-multus-socket-dir-parent\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.573455 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.571498 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-var-lib-openvswitch\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.573455 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.571549 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6ceea52-1124-417d-92ff-53d3f054bded-var-lib-openvswitch\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:09.573455 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.571551 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/23698c81-8626-4b0e-963b-e20fde992ed9-multus-socket-dir-parent\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:09.573455 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.571565 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mghsk\" (UID: \"fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk" Jul 22 01:37:09.573455 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:09.572709 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e6ceea52-1124-417d-92ff-53d3f054bded-ovn-node-metrics-cert\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:10.899618 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:10.899592 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 01:37:10.899683 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:10.899620 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 01:37:10.899683 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:10.899632 2561 projected.go:196] Error preparing data for projected volume kube-api-access-jglhg for pod openshift-network-diagnostics/network-check-target-fxfmm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:37:10.899683 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:10.899677 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/271b4b39-8156-44e0-a9fb-c937501425fa-kube-api-access-jglhg podName:271b4b39-8156-44e0-a9fb-c937501425fa nodeName:}" failed. No retries permitted until 2026-07-22 01:37:11.399661703 +0000 UTC m=+1.992466064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jglhg" (UniqueName: "kubernetes.io/projected/271b4b39-8156-44e0-a9fb-c937501425fa-kube-api-access-jglhg") pod "network-check-target-fxfmm" (UID: "271b4b39-8156-44e0-a9fb-c937501425fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:37:10.901688 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.901662 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fphnt\" (UniqueName: \"kubernetes.io/projected/4d9fa62a-a81a-4bdc-893e-5928340b007b-kube-api-access-fphnt\") pod \"multus-additional-cni-plugins-cpk88\" (UID: \"4d9fa62a-a81a-4bdc-893e-5928340b007b\") " pod="openshift-multus/multus-additional-cni-plugins-cpk88" Jul 22 01:37:10.902292 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.902271 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdc72\" (UniqueName: \"kubernetes.io/projected/23698c81-8626-4b0e-963b-e20fde992ed9-kube-api-access-tdc72\") pod \"multus-c4c5n\" (UID: \"23698c81-8626-4b0e-963b-e20fde992ed9\") " pod="openshift-multus/multus-c4c5n" Jul 22 01:37:10.902952 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.902931 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxsf6\" (UniqueName: \"kubernetes.io/projected/f18aa922-9c77-4466-aad7-6e2f5fe12c59-kube-api-access-jxsf6\") pod \"network-metrics-daemon-j4s4b\" (UID: \"f18aa922-9c77-4466-aad7-6e2f5fe12c59\") " pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:10.903113 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.903095 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59965\" (UniqueName: \"kubernetes.io/projected/deed9faf-5432-4522-aa0d-362817d03ad1-kube-api-access-59965\") pod \"iptables-alerter-7qn5n\" (UID: \"deed9faf-5432-4522-aa0d-362817d03ad1\") " pod="openshift-network-operator/iptables-alerter-7qn5n" Jul 22 01:37:10.903153 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.903126 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxfgh\" (UniqueName: \"kubernetes.io/projected/e6ceea52-1124-417d-92ff-53d3f054bded-kube-api-access-cxfgh\") pod \"ovnkube-node-lcjb6\" (UID: \"e6ceea52-1124-417d-92ff-53d3f054bded\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:10.903183 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.903150 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjwfk\" (UniqueName: \"kubernetes.io/projected/6a4a5252-c595-42f9-834a-0c26e27adaf8-kube-api-access-sjwfk\") pod \"node-resolver-92dh5\" (UID: \"6a4a5252-c595-42f9-834a-0c26e27adaf8\") " pod="openshift-dns/node-resolver-92dh5" Jul 22 01:37:10.993912 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.993882 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-host\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:10.993912 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.993910 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7wfk\" (UniqueName: \"kubernetes.io/projected/16abf4b8-914b-4259-b8f5-e6e51f112359-kube-api-access-b7wfk\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:10.994115 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.993942 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-lib-modules\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:10.994115 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.993960 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e-etc-selinux\") pod \"aws-ebs-csi-driver-node-mghsk\" (UID: \"fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk" Jul 22 01:37:10.994115 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.993987 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-etc-kubernetes\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:10.994115 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994007 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mghsk\" (UID: \"fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk" Jul 22 01:37:10.994115 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994037 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-host\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:10.994115 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994044 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/16abf4b8-914b-4259-b8f5-e6e51f112359-etc-tuned\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:10.994115 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994072 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-lib-modules\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:10.994115 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994081 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16abf4b8-914b-4259-b8f5-e6e51f112359-tmp\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:10.994115 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994085 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e-etc-selinux\") pod \"aws-ebs-csi-driver-node-mghsk\" (UID: \"fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk" Jul 22 01:37:10.994115 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994104 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a184b54b-d2a6-4c78-aea6-714b9f0a6a1f-serviceca\") pod \"node-ca-j78lg\" (UID: \"a184b54b-d2a6-4c78-aea6-714b9f0a6a1f\") " pod="openshift-image-registry/node-ca-j78lg" Jul 22 01:37:10.994115 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994111 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-etc-kubernetes\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:10.994628 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994129 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e-device-dir\") pod \"aws-ebs-csi-driver-node-mghsk\" (UID: \"fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk" Jul 22 01:37:10.994628 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994139 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mghsk\" (UID: \"fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk" Jul 22 01:37:10.994628 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994154 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-run\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:10.994628 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994191 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-var-lib-kubelet\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:10.994628 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994196 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-run\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:10.994628 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994214 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e-device-dir\") pod \"aws-ebs-csi-driver-node-mghsk\" (UID: \"fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk" Jul 22 01:37:10.994628 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994233 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/50a4581a-21ba-4900-aee7-648e18e7abe5-konnectivity-ca\") pod \"konnectivity-agent-gzgpx\" (UID: \"50a4581a-21ba-4900-aee7-648e18e7abe5\") " pod="kube-system/konnectivity-agent-gzgpx" Jul 22 01:37:10.994628 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994302 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-var-lib-kubelet\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:10.994628 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994315 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmhn5\" (UniqueName: \"kubernetes.io/projected/fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e-kube-api-access-lmhn5\") pod \"aws-ebs-csi-driver-node-mghsk\" (UID: \"fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk" Jul 22 01:37:10.994628 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994372 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-etc-sysconfig\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:10.994628 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994398 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-etc-sysctl-conf\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:10.994628 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994424 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e-registration-dir\") pod \"aws-ebs-csi-driver-node-mghsk\" (UID: \"fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk" Jul 22 01:37:10.994628 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994449 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e-sys-fs\") pod \"aws-ebs-csi-driver-node-mghsk\" (UID: \"fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk" Jul 22 01:37:10.994628 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994459 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-etc-sysconfig\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:10.994628 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994492 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-etc-modprobe-d\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:10.994628 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994518 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e-registration-dir\") pod \"aws-ebs-csi-driver-node-mghsk\" (UID: \"fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk" Jul 22 01:37:10.994628 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994530 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-etc-sysctl-conf\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:10.995218 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994537 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/50a4581a-21ba-4900-aee7-648e18e7abe5-agent-certs\") pod \"konnectivity-agent-gzgpx\" (UID: \"50a4581a-21ba-4900-aee7-648e18e7abe5\") " pod="kube-system/konnectivity-agent-gzgpx" Jul 22 01:37:10.995218 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994562 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a184b54b-d2a6-4c78-aea6-714b9f0a6a1f-serviceca\") pod \"node-ca-j78lg\" (UID: \"a184b54b-d2a6-4c78-aea6-714b9f0a6a1f\") " pod="openshift-image-registry/node-ca-j78lg" Jul 22 01:37:10.995218 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994570 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e-sys-fs\") pod \"aws-ebs-csi-driver-node-mghsk\" (UID: \"fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk" Jul 22 01:37:10.995218 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994579 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-etc-sysctl-d\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:10.995218 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994610 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e-socket-dir\") pod \"aws-ebs-csi-driver-node-mghsk\" (UID: \"fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk" Jul 22 01:37:10.995218 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994609 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-etc-modprobe-d\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:10.995218 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994683 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-sys\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:10.995218 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994689 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-etc-sysctl-d\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:10.995218 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994727 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a184b54b-d2a6-4c78-aea6-714b9f0a6a1f-host\") pod \"node-ca-j78lg\" (UID: \"a184b54b-d2a6-4c78-aea6-714b9f0a6a1f\") " pod="openshift-image-registry/node-ca-j78lg" Jul 22 01:37:10.995218 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994744 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-sys\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:10.995218 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994753 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbmz5\" (UniqueName: \"kubernetes.io/projected/a184b54b-d2a6-4c78-aea6-714b9f0a6a1f-kube-api-access-vbmz5\") pod \"node-ca-j78lg\" (UID: \"a184b54b-d2a6-4c78-aea6-714b9f0a6a1f\") " pod="openshift-image-registry/node-ca-j78lg" Jul 22 01:37:10.995218 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994779 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-etc-systemd\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:10.995218 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994789 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a184b54b-d2a6-4c78-aea6-714b9f0a6a1f-host\") pod \"node-ca-j78lg\" (UID: \"a184b54b-d2a6-4c78-aea6-714b9f0a6a1f\") " pod="openshift-image-registry/node-ca-j78lg" Jul 22 01:37:10.995218 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994798 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e-socket-dir\") pod \"aws-ebs-csi-driver-node-mghsk\" (UID: \"fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk" Jul 22 01:37:10.995218 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994828 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/50a4581a-21ba-4900-aee7-648e18e7abe5-konnectivity-ca\") pod \"konnectivity-agent-gzgpx\" (UID: \"50a4581a-21ba-4900-aee7-648e18e7abe5\") " pod="kube-system/konnectivity-agent-gzgpx" Jul 22 01:37:10.995218 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.994915 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/16abf4b8-914b-4259-b8f5-e6e51f112359-etc-systemd\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:10.996306 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.996286 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/16abf4b8-914b-4259-b8f5-e6e51f112359-etc-tuned\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:10.996640 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.996620 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16abf4b8-914b-4259-b8f5-e6e51f112359-tmp\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:10.996915 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:10.996898 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/50a4581a-21ba-4900-aee7-648e18e7abe5-agent-certs\") pod \"konnectivity-agent-gzgpx\" (UID: \"50a4581a-21ba-4900-aee7-648e18e7abe5\") " pod="kube-system/konnectivity-agent-gzgpx" Jul 22 01:37:11.007899 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.007877 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbmz5\" (UniqueName: \"kubernetes.io/projected/a184b54b-d2a6-4c78-aea6-714b9f0a6a1f-kube-api-access-vbmz5\") pod \"node-ca-j78lg\" (UID: \"a184b54b-d2a6-4c78-aea6-714b9f0a6a1f\") " pod="openshift-image-registry/node-ca-j78lg" Jul 22 01:37:11.008028 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.008009 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7wfk\" (UniqueName: \"kubernetes.io/projected/16abf4b8-914b-4259-b8f5-e6e51f112359-kube-api-access-b7wfk\") pod \"tuned-bbflh\" (UID: \"16abf4b8-914b-4259-b8f5-e6e51f112359\") " pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:11.008079 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.008020 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmhn5\" (UniqueName: \"kubernetes.io/projected/fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e-kube-api-access-lmhn5\") pod \"aws-ebs-csi-driver-node-mghsk\" (UID: \"fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk" Jul 22 01:37:11.021599 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:11.021575 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f30de953d78a535da92683887f7e533.slice/crio-807501849ffe2498dbde869640232e132e518175e7caa8d39dcb394a82576033 WatchSource:0}: Error finding container 807501849ffe2498dbde869640232e132e518175e7caa8d39dcb394a82576033: Status 404 returned error can't find the container with id 807501849ffe2498dbde869640232e132e518175e7caa8d39dcb394a82576033 Jul 22 01:37:11.028294 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.028278 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 01:37:11.108350 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.108299 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7qn5n" Jul 22 01:37:11.115603 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:11.115585 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeed9faf_5432_4522_aa0d_362817d03ad1.slice/crio-924723900e064fd8b80f42c53b36f6772c73922f4b0799174b70762228b10373 WatchSource:0}: Error finding container 924723900e064fd8b80f42c53b36f6772c73922f4b0799174b70762228b10373: Status 404 returned error can't find the container with id 924723900e064fd8b80f42c53b36f6772c73922f4b0799174b70762228b10373 Jul 22 01:37:11.132000 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.131984 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c4c5n" Jul 22 01:37:11.138372 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:11.138353 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23698c81_8626_4b0e_963b_e20fde992ed9.slice/crio-7bb974c8938bbfc92feb61a113d7346bc8095b6232d440f7172cc207ba58d2cf WatchSource:0}: Error finding container 7bb974c8938bbfc92feb61a113d7346bc8095b6232d440f7172cc207ba58d2cf: Status 404 returned error can't find the container with id 7bb974c8938bbfc92feb61a113d7346bc8095b6232d440f7172cc207ba58d2cf Jul 22 01:37:11.152114 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.152091 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:11.158443 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:11.158425 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ceea52_1124_417d_92ff_53d3f054bded.slice/crio-6af91c2aa957955b53d0e7090998db44c0c93f3966c3dba0c8606ee1567d1395 WatchSource:0}: Error finding container 6af91c2aa957955b53d0e7090998db44c0c93f3966c3dba0c8606ee1567d1395: Status 404 returned error can't find the container with id 6af91c2aa957955b53d0e7090998db44c0c93f3966c3dba0c8606ee1567d1395 Jul 22 01:37:11.166940 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.166925 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cpk88" Jul 22 01:37:11.173516 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:11.173496 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9fa62a_a81a_4bdc_893e_5928340b007b.slice/crio-dda2b198dddb1415576e57c7eda0763fe47bd7d6c8ae91e69662cf8a5a46effe WatchSource:0}: Error finding container dda2b198dddb1415576e57c7eda0763fe47bd7d6c8ae91e69662cf8a5a46effe: Status 404 returned error can't find the container with id dda2b198dddb1415576e57c7eda0763fe47bd7d6c8ae91e69662cf8a5a46effe Jul 22 01:37:11.187282 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.187258 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-92dh5" Jul 22 01:37:11.193520 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:11.193503 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a4a5252_c595_42f9_834a_0c26e27adaf8.slice/crio-7b8d6bd235edef52ee5065eeda6185c9433c8f45cb1a0a19d4773bf1d3cc7eee WatchSource:0}: Error finding container 7b8d6bd235edef52ee5065eeda6185c9433c8f45cb1a0a19d4773bf1d3cc7eee: Status 404 returned error can't find the container with id 7b8d6bd235edef52ee5065eeda6185c9433c8f45cb1a0a19d4773bf1d3cc7eee Jul 22 01:37:11.203406 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.203386 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gzgpx" Jul 22 01:37:11.209672 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:11.209652 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a4581a_21ba_4900_aee7_648e18e7abe5.slice/crio-d260a7429588558daa3a8f9cafda232422df4fec5b1cf7f4c3d302f6c037dd6b WatchSource:0}: Error finding container d260a7429588558daa3a8f9cafda232422df4fec5b1cf7f4c3d302f6c037dd6b: Status 404 returned error can't find the container with id d260a7429588558daa3a8f9cafda232422df4fec5b1cf7f4c3d302f6c037dd6b Jul 22 01:37:11.228391 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.228373 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk" Jul 22 01:37:11.235553 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:11.235536 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe79ae2d_bfd2_433d_ad58_1cf7cf2e274e.slice/crio-61fffff5bec229e8a92a9d82f39fe6e4961ea706dfc3595e148366566a3c6fec WatchSource:0}: Error finding container 61fffff5bec229e8a92a9d82f39fe6e4961ea706dfc3595e148366566a3c6fec: Status 404 returned error can't find the container with id 61fffff5bec229e8a92a9d82f39fe6e4961ea706dfc3595e148366566a3c6fec Jul 22 01:37:11.239445 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.239431 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bbflh" Jul 22 01:37:11.243986 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.243968 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j78lg" Jul 22 01:37:11.246570 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:11.246549 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16abf4b8_914b_4259_b8f5_e6e51f112359.slice/crio-84612a4116b655582b22f026d83639d1a7f5899043d79b8c2752fa075c82823f WatchSource:0}: Error finding container 84612a4116b655582b22f026d83639d1a7f5899043d79b8c2752fa075c82823f: Status 404 returned error can't find the container with id 84612a4116b655582b22f026d83639d1a7f5899043d79b8c2752fa075c82823f Jul 22 01:37:11.251297 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:11.251276 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda184b54b_d2a6_4c78_aea6_714b9f0a6a1f.slice/crio-d9a250fd855c536bcfbf20906e052227a1aeb87720105b988f9783486b95c03b WatchSource:0}: Error finding container d9a250fd855c536bcfbf20906e052227a1aeb87720105b988f9783486b95c03b: Status 404 returned error can't find the container with id d9a250fd855c536bcfbf20906e052227a1aeb87720105b988f9783486b95c03b Jul 22 01:37:11.257716 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.257682 2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 01:37:11.398332 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.398287 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f18aa922-9c77-4466-aad7-6e2f5fe12c59-metrics-certs\") pod \"network-metrics-daemon-j4s4b\" (UID: \"f18aa922-9c77-4466-aad7-6e2f5fe12c59\") " pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:11.398332 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.398328 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/448e76a1-df97-4a26-8433-7a51cdeb4a5a-original-pull-secret\") pod \"global-pull-secret-syncer-ghl8b\" (UID: \"448e76a1-df97-4a26-8433-7a51cdeb4a5a\") " pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:11.398461 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:11.398416 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 01:37:11.398461 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:11.398420 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:37:11.398522 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:11.398464 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448e76a1-df97-4a26-8433-7a51cdeb4a5a-original-pull-secret podName:448e76a1-df97-4a26-8433-7a51cdeb4a5a nodeName:}" failed. No retries permitted until 2026-07-22 01:37:12.398446058 +0000 UTC m=+2.991250398 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/448e76a1-df97-4a26-8433-7a51cdeb4a5a-original-pull-secret") pod "global-pull-secret-syncer-ghl8b" (UID: "448e76a1-df97-4a26-8433-7a51cdeb4a5a") : object "kube-system"/"original-pull-secret" not registered Jul 22 01:37:11.398522 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:11.398478 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f18aa922-9c77-4466-aad7-6e2f5fe12c59-metrics-certs podName:f18aa922-9c77-4466-aad7-6e2f5fe12c59 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:12.398472384 +0000 UTC m=+2.991276725 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f18aa922-9c77-4466-aad7-6e2f5fe12c59-metrics-certs") pod "network-metrics-daemon-j4s4b" (UID: "f18aa922-9c77-4466-aad7-6e2f5fe12c59") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:37:11.422016 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:11.421995 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4419f2dab53b7f8620a2511c1a87c5c9.slice/crio-66f21bc8e3ed3984f1776c92457e684a87e1ea68331d91d4285a7b2bee794096 WatchSource:0}: Error finding container 66f21bc8e3ed3984f1776c92457e684a87e1ea68331d91d4285a7b2bee794096: Status 404 returned error can't find the container with id 66f21bc8e3ed3984f1776c92457e684a87e1ea68331d91d4285a7b2bee794096 Jul 22 01:37:11.498903 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.498883 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jglhg\" (UniqueName: \"kubernetes.io/projected/271b4b39-8156-44e0-a9fb-c937501425fa-kube-api-access-jglhg\") pod \"network-check-target-fxfmm\" (UID: \"271b4b39-8156-44e0-a9fb-c937501425fa\") " pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:37:11.499019 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:11.499005 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 01:37:11.499085 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:11.499024 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 01:37:11.499085 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:11.499035 2561 projected.go:196] Error preparing data for projected volume kube-api-access-jglhg for pod openshift-network-diagnostics/network-check-target-fxfmm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:37:11.499085 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:11.499085 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/271b4b39-8156-44e0-a9fb-c937501425fa-kube-api-access-jglhg podName:271b4b39-8156-44e0-a9fb-c937501425fa nodeName:}" failed. No retries permitted until 2026-07-22 01:37:12.499069481 +0000 UTC m=+3.091873824 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jglhg" (UniqueName: "kubernetes.io/projected/271b4b39-8156-44e0-a9fb-c937501425fa-kube-api-access-jglhg") pod "network-check-target-fxfmm" (UID: "271b4b39-8156-44e0-a9fb-c937501425fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:37:11.829888 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.829773 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 01:32:10 +0000 UTC" deadline="2028-04-09 18:10:47.425141906 +0000 UTC" Jul 22 01:37:11.829888 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.829842 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15064h33m35.595303957s" Jul 22 01:37:11.921008 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.920977 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:11.921177 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:11.921111 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghl8b" podUID="448e76a1-df97-4a26-8433-7a51cdeb4a5a" Jul 22 01:37:11.933639 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.933584 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j78lg" event={"ID":"a184b54b-d2a6-4c78-aea6-714b9f0a6a1f","Type":"ContainerStarted","Data":"d9a250fd855c536bcfbf20906e052227a1aeb87720105b988f9783486b95c03b"} Jul 22 01:37:11.935237 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.935212 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bbflh" event={"ID":"16abf4b8-914b-4259-b8f5-e6e51f112359","Type":"ContainerStarted","Data":"84612a4116b655582b22f026d83639d1a7f5899043d79b8c2752fa075c82823f"} Jul 22 01:37:11.938667 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.938583 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk" event={"ID":"fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e","Type":"ContainerStarted","Data":"61fffff5bec229e8a92a9d82f39fe6e4961ea706dfc3595e148366566a3c6fec"} Jul 22 01:37:11.943390 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.943028 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gzgpx" event={"ID":"50a4581a-21ba-4900-aee7-648e18e7abe5","Type":"ContainerStarted","Data":"d260a7429588558daa3a8f9cafda232422df4fec5b1cf7f4c3d302f6c037dd6b"} Jul 22 01:37:11.944411 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.944387 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-92dh5" event={"ID":"6a4a5252-c595-42f9-834a-0c26e27adaf8","Type":"ContainerStarted","Data":"7b8d6bd235edef52ee5065eeda6185c9433c8f45cb1a0a19d4773bf1d3cc7eee"} Jul 22 01:37:11.946156 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.946094 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7qn5n" event={"ID":"deed9faf-5432-4522-aa0d-362817d03ad1","Type":"ContainerStarted","Data":"924723900e064fd8b80f42c53b36f6772c73922f4b0799174b70762228b10373"} Jul 22 01:37:11.948839 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.948777 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-56.ec2.internal" event={"ID":"8f30de953d78a535da92683887f7e533","Type":"ContainerStarted","Data":"807501849ffe2498dbde869640232e132e518175e7caa8d39dcb394a82576033"} Jul 22 01:37:11.950921 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.950891 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-56.ec2.internal" event={"ID":"4419f2dab53b7f8620a2511c1a87c5c9","Type":"ContainerStarted","Data":"66f21bc8e3ed3984f1776c92457e684a87e1ea68331d91d4285a7b2bee794096"} Jul 22 01:37:11.953483 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.953460 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cpk88" event={"ID":"4d9fa62a-a81a-4bdc-893e-5928340b007b","Type":"ContainerStarted","Data":"dda2b198dddb1415576e57c7eda0763fe47bd7d6c8ae91e69662cf8a5a46effe"} Jul 22 01:37:11.957168 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.957134 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" event={"ID":"e6ceea52-1124-417d-92ff-53d3f054bded","Type":"ContainerStarted","Data":"6af91c2aa957955b53d0e7090998db44c0c93f3966c3dba0c8606ee1567d1395"} Jul 22 01:37:11.961232 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:11.961207 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c4c5n" event={"ID":"23698c81-8626-4b0e-963b-e20fde992ed9","Type":"ContainerStarted","Data":"7bb974c8938bbfc92feb61a113d7346bc8095b6232d440f7172cc207ba58d2cf"} Jul 22 01:37:12.067675 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:12.067473 2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 01:37:12.405175 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:12.405143 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f18aa922-9c77-4466-aad7-6e2f5fe12c59-metrics-certs\") pod \"network-metrics-daemon-j4s4b\" (UID: \"f18aa922-9c77-4466-aad7-6e2f5fe12c59\") " pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:12.405344 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:12.405215 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/448e76a1-df97-4a26-8433-7a51cdeb4a5a-original-pull-secret\") pod \"global-pull-secret-syncer-ghl8b\" (UID: \"448e76a1-df97-4a26-8433-7a51cdeb4a5a\") " pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:12.405407 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:12.405341 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 01:37:12.405407 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:12.405397 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448e76a1-df97-4a26-8433-7a51cdeb4a5a-original-pull-secret podName:448e76a1-df97-4a26-8433-7a51cdeb4a5a nodeName:}" failed. No retries permitted until 2026-07-22 01:37:14.40538111 +0000 UTC m=+4.998185455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/448e76a1-df97-4a26-8433-7a51cdeb4a5a-original-pull-secret") pod "global-pull-secret-syncer-ghl8b" (UID: "448e76a1-df97-4a26-8433-7a51cdeb4a5a") : object "kube-system"/"original-pull-secret" not registered Jul 22 01:37:12.406186 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:12.405809 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:37:12.406186 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:12.405861 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f18aa922-9c77-4466-aad7-6e2f5fe12c59-metrics-certs podName:f18aa922-9c77-4466-aad7-6e2f5fe12c59 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:14.405844786 +0000 UTC m=+4.998649143 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f18aa922-9c77-4466-aad7-6e2f5fe12c59-metrics-certs") pod "network-metrics-daemon-j4s4b" (UID: "f18aa922-9c77-4466-aad7-6e2f5fe12c59") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:37:12.411724 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:12.411467 2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 01:37:12.506539 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:12.506456 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jglhg\" (UniqueName: \"kubernetes.io/projected/271b4b39-8156-44e0-a9fb-c937501425fa-kube-api-access-jglhg\") pod \"network-check-target-fxfmm\" (UID: \"271b4b39-8156-44e0-a9fb-c937501425fa\") " pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:37:12.506720 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:12.506624 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 01:37:12.506720 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:12.506662 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 01:37:12.506720 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:12.506674 2561 projected.go:196] Error preparing data for projected volume kube-api-access-jglhg for pod openshift-network-diagnostics/network-check-target-fxfmm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:37:12.506879 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:12.506743 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/271b4b39-8156-44e0-a9fb-c937501425fa-kube-api-access-jglhg podName:271b4b39-8156-44e0-a9fb-c937501425fa nodeName:}" failed. No retries permitted until 2026-07-22 01:37:14.50672366 +0000 UTC m=+5.099528004 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jglhg" (UniqueName: "kubernetes.io/projected/271b4b39-8156-44e0-a9fb-c937501425fa-kube-api-access-jglhg") pod "network-check-target-fxfmm" (UID: "271b4b39-8156-44e0-a9fb-c937501425fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:37:12.830989 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:12.830949 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 01:32:10 +0000 UTC" deadline="2028-02-02 09:20:11.472270942 +0000 UTC" Jul 22 01:37:12.830989 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:12.830987 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13447h42m58.641288072s" Jul 22 01:37:12.919346 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:12.919315 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:12.919521 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:12.919460 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j4s4b" podUID="f18aa922-9c77-4466-aad7-6e2f5fe12c59" Jul 22 01:37:12.919901 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:12.919876 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:37:12.920039 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:12.919979 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxfmm" podUID="271b4b39-8156-44e0-a9fb-c937501425fa" Jul 22 01:37:13.688145 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:13.687928 2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 01:37:13.925110 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:13.925079 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:13.925539 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:13.925220 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghl8b" podUID="448e76a1-df97-4a26-8433-7a51cdeb4a5a" Jul 22 01:37:14.426652 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:14.425896 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f18aa922-9c77-4466-aad7-6e2f5fe12c59-metrics-certs\") pod \"network-metrics-daemon-j4s4b\" (UID: \"f18aa922-9c77-4466-aad7-6e2f5fe12c59\") " pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:14.426652 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:14.425972 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/448e76a1-df97-4a26-8433-7a51cdeb4a5a-original-pull-secret\") pod \"global-pull-secret-syncer-ghl8b\" (UID: \"448e76a1-df97-4a26-8433-7a51cdeb4a5a\") " pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:14.426652 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:14.426120 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 01:37:14.426652 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:14.426183 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448e76a1-df97-4a26-8433-7a51cdeb4a5a-original-pull-secret podName:448e76a1-df97-4a26-8433-7a51cdeb4a5a nodeName:}" failed. No retries permitted until 2026-07-22 01:37:18.426162703 +0000 UTC m=+9.018967058 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/448e76a1-df97-4a26-8433-7a51cdeb4a5a-original-pull-secret") pod "global-pull-secret-syncer-ghl8b" (UID: "448e76a1-df97-4a26-8433-7a51cdeb4a5a") : object "kube-system"/"original-pull-secret" not registered Jul 22 01:37:14.426652 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:14.426557 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:37:14.426652 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:14.426604 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f18aa922-9c77-4466-aad7-6e2f5fe12c59-metrics-certs podName:f18aa922-9c77-4466-aad7-6e2f5fe12c59 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:18.426588136 +0000 UTC m=+9.019392480 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f18aa922-9c77-4466-aad7-6e2f5fe12c59-metrics-certs") pod "network-metrics-daemon-j4s4b" (UID: "f18aa922-9c77-4466-aad7-6e2f5fe12c59") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:37:14.527834 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:14.527207 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jglhg\" (UniqueName: \"kubernetes.io/projected/271b4b39-8156-44e0-a9fb-c937501425fa-kube-api-access-jglhg\") pod \"network-check-target-fxfmm\" (UID: \"271b4b39-8156-44e0-a9fb-c937501425fa\") " pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:37:14.527834 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:14.527415 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 01:37:14.527834 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:14.527434 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 01:37:14.527834 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:14.527446 2561 projected.go:196] Error preparing data for projected volume kube-api-access-jglhg for pod openshift-network-diagnostics/network-check-target-fxfmm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:37:14.527834 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:14.527503 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/271b4b39-8156-44e0-a9fb-c937501425fa-kube-api-access-jglhg podName:271b4b39-8156-44e0-a9fb-c937501425fa nodeName:}" failed. No retries permitted until 2026-07-22 01:37:18.527484633 +0000 UTC m=+9.120288976 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jglhg" (UniqueName: "kubernetes.io/projected/271b4b39-8156-44e0-a9fb-c937501425fa-kube-api-access-jglhg") pod "network-check-target-fxfmm" (UID: "271b4b39-8156-44e0-a9fb-c937501425fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:37:14.919338 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:14.919260 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:37:14.919494 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:14.919381 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxfmm" podUID="271b4b39-8156-44e0-a9fb-c937501425fa" Jul 22 01:37:14.919877 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:14.919854 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:14.919993 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:14.919963 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j4s4b" podUID="f18aa922-9c77-4466-aad7-6e2f5fe12c59" Jul 22 01:37:15.919891 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:15.919862 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:15.920327 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:15.919991 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghl8b" podUID="448e76a1-df97-4a26-8433-7a51cdeb4a5a" Jul 22 01:37:16.919589 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:16.919109 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:16.919589 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:16.919128 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:37:16.919589 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:16.919258 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j4s4b" podUID="f18aa922-9c77-4466-aad7-6e2f5fe12c59" Jul 22 01:37:16.919589 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:16.919270 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxfmm" podUID="271b4b39-8156-44e0-a9fb-c937501425fa" Jul 22 01:37:17.918825 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:17.918790 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:17.918977 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:17.918916 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghl8b" podUID="448e76a1-df97-4a26-8433-7a51cdeb4a5a" Jul 22 01:37:18.466114 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:18.465360 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/448e76a1-df97-4a26-8433-7a51cdeb4a5a-original-pull-secret\") pod \"global-pull-secret-syncer-ghl8b\" (UID: \"448e76a1-df97-4a26-8433-7a51cdeb4a5a\") " pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:18.466114 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:18.465452 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f18aa922-9c77-4466-aad7-6e2f5fe12c59-metrics-certs\") pod \"network-metrics-daemon-j4s4b\" (UID: \"f18aa922-9c77-4466-aad7-6e2f5fe12c59\") " pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:18.466114 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:18.465596 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:37:18.466114 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:18.465666 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f18aa922-9c77-4466-aad7-6e2f5fe12c59-metrics-certs podName:f18aa922-9c77-4466-aad7-6e2f5fe12c59 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:26.465644773 +0000 UTC m=+17.058449115 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f18aa922-9c77-4466-aad7-6e2f5fe12c59-metrics-certs") pod "network-metrics-daemon-j4s4b" (UID: "f18aa922-9c77-4466-aad7-6e2f5fe12c59") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:37:18.466114 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:18.466052 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 01:37:18.466114 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:18.466091 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448e76a1-df97-4a26-8433-7a51cdeb4a5a-original-pull-secret podName:448e76a1-df97-4a26-8433-7a51cdeb4a5a nodeName:}" failed. No retries permitted until 2026-07-22 01:37:26.466080423 +0000 UTC m=+17.058884764 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/448e76a1-df97-4a26-8433-7a51cdeb4a5a-original-pull-secret") pod "global-pull-secret-syncer-ghl8b" (UID: "448e76a1-df97-4a26-8433-7a51cdeb4a5a") : object "kube-system"/"original-pull-secret" not registered Jul 22 01:37:18.566270 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:18.566193 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jglhg\" (UniqueName: \"kubernetes.io/projected/271b4b39-8156-44e0-a9fb-c937501425fa-kube-api-access-jglhg\") pod \"network-check-target-fxfmm\" (UID: \"271b4b39-8156-44e0-a9fb-c937501425fa\") " pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:37:18.566452 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:18.566351 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 01:37:18.566452 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:18.566381 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 01:37:18.566452 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:18.566394 2561 projected.go:196] Error preparing data for projected volume kube-api-access-jglhg for pod openshift-network-diagnostics/network-check-target-fxfmm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:37:18.566608 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:18.566462 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/271b4b39-8156-44e0-a9fb-c937501425fa-kube-api-access-jglhg podName:271b4b39-8156-44e0-a9fb-c937501425fa nodeName:}" failed. No retries permitted until 2026-07-22 01:37:26.566440614 +0000 UTC m=+17.159244958 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jglhg" (UniqueName: "kubernetes.io/projected/271b4b39-8156-44e0-a9fb-c937501425fa-kube-api-access-jglhg") pod "network-check-target-fxfmm" (UID: "271b4b39-8156-44e0-a9fb-c937501425fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:37:18.919683 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:18.919232 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:18.919683 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:18.919266 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:37:18.919683 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:18.919353 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j4s4b" podUID="f18aa922-9c77-4466-aad7-6e2f5fe12c59" Jul 22 01:37:18.919683 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:18.919464 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxfmm" podUID="271b4b39-8156-44e0-a9fb-c937501425fa" Jul 22 01:37:19.920501 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:19.920465 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:19.920955 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:19.920579 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghl8b" podUID="448e76a1-df97-4a26-8433-7a51cdeb4a5a" Jul 22 01:37:20.918642 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:20.918606 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:37:20.918834 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:20.918653 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:20.918834 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:20.918763 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxfmm" podUID="271b4b39-8156-44e0-a9fb-c937501425fa" Jul 22 01:37:20.918945 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:20.918853 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j4s4b" podUID="f18aa922-9c77-4466-aad7-6e2f5fe12c59" Jul 22 01:37:21.919015 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:21.918978 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:21.919406 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:21.919125 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghl8b" podUID="448e76a1-df97-4a26-8433-7a51cdeb4a5a" Jul 22 01:37:22.919190 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:22.919115 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:37:22.919190 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:22.919162 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:22.919666 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:22.919251 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxfmm" podUID="271b4b39-8156-44e0-a9fb-c937501425fa" Jul 22 01:37:22.919666 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:22.919389 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j4s4b" podUID="f18aa922-9c77-4466-aad7-6e2f5fe12c59" Jul 22 01:37:23.918900 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:23.918864 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:23.919081 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:23.918995 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghl8b" podUID="448e76a1-df97-4a26-8433-7a51cdeb4a5a" Jul 22 01:37:24.918595 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:24.918558 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:24.919039 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:24.918708 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j4s4b" podUID="f18aa922-9c77-4466-aad7-6e2f5fe12c59" Jul 22 01:37:24.919039 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:24.918735 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:37:24.919039 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:24.918843 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxfmm" podUID="271b4b39-8156-44e0-a9fb-c937501425fa" Jul 22 01:37:25.918747 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:25.918718 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:25.919173 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:25.918834 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghl8b" podUID="448e76a1-df97-4a26-8433-7a51cdeb4a5a" Jul 22 01:37:26.531158 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:26.531118 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/448e76a1-df97-4a26-8433-7a51cdeb4a5a-original-pull-secret\") pod \"global-pull-secret-syncer-ghl8b\" (UID: \"448e76a1-df97-4a26-8433-7a51cdeb4a5a\") " pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:26.531344 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:26.531184 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f18aa922-9c77-4466-aad7-6e2f5fe12c59-metrics-certs\") pod \"network-metrics-daemon-j4s4b\" (UID: \"f18aa922-9c77-4466-aad7-6e2f5fe12c59\") " pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:26.531344 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:26.531243 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 01:37:26.531344 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:26.531265 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:37:26.531344 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:26.531318 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448e76a1-df97-4a26-8433-7a51cdeb4a5a-original-pull-secret podName:448e76a1-df97-4a26-8433-7a51cdeb4a5a nodeName:}" failed. No retries permitted until 2026-07-22 01:37:42.531297028 +0000 UTC m=+33.124101374 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/448e76a1-df97-4a26-8433-7a51cdeb4a5a-original-pull-secret") pod "global-pull-secret-syncer-ghl8b" (UID: "448e76a1-df97-4a26-8433-7a51cdeb4a5a") : object "kube-system"/"original-pull-secret" not registered Jul 22 01:37:26.531344 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:26.531336 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f18aa922-9c77-4466-aad7-6e2f5fe12c59-metrics-certs podName:f18aa922-9c77-4466-aad7-6e2f5fe12c59 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:42.531327394 +0000 UTC m=+33.124131736 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f18aa922-9c77-4466-aad7-6e2f5fe12c59-metrics-certs") pod "network-metrics-daemon-j4s4b" (UID: "f18aa922-9c77-4466-aad7-6e2f5fe12c59") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:37:26.631752 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:26.631669 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jglhg\" (UniqueName: \"kubernetes.io/projected/271b4b39-8156-44e0-a9fb-c937501425fa-kube-api-access-jglhg\") pod \"network-check-target-fxfmm\" (UID: \"271b4b39-8156-44e0-a9fb-c937501425fa\") " pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:37:26.631937 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:26.631854 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 01:37:26.631937 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:26.631880 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 01:37:26.631937 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:26.631892 2561 projected.go:196] Error preparing data for projected volume kube-api-access-jglhg for pod openshift-network-diagnostics/network-check-target-fxfmm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:37:26.632089 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:26.631948 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/271b4b39-8156-44e0-a9fb-c937501425fa-kube-api-access-jglhg podName:271b4b39-8156-44e0-a9fb-c937501425fa nodeName:}" failed. No retries permitted until 2026-07-22 01:37:42.631931595 +0000 UTC m=+33.224735938 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jglhg" (UniqueName: "kubernetes.io/projected/271b4b39-8156-44e0-a9fb-c937501425fa-kube-api-access-jglhg") pod "network-check-target-fxfmm" (UID: "271b4b39-8156-44e0-a9fb-c937501425fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:37:26.918575 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:26.918497 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:26.918798 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:26.918610 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j4s4b" podUID="f18aa922-9c77-4466-aad7-6e2f5fe12c59" Jul 22 01:37:26.918798 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:26.918681 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:37:26.919163 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:26.918795 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxfmm" podUID="271b4b39-8156-44e0-a9fb-c937501425fa" Jul 22 01:37:27.918625 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:27.918595 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:27.918824 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:27.918740 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghl8b" podUID="448e76a1-df97-4a26-8433-7a51cdeb4a5a" Jul 22 01:37:28.919366 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:28.919332 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:28.919824 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:28.919332 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:37:28.919824 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:28.919442 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j4s4b" podUID="f18aa922-9c77-4466-aad7-6e2f5fe12c59" Jul 22 01:37:28.919824 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:28.919508 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxfmm" podUID="271b4b39-8156-44e0-a9fb-c937501425fa" Jul 22 01:37:29.920237 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:29.920208 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:29.920639 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:29.920326 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghl8b" podUID="448e76a1-df97-4a26-8433-7a51cdeb4a5a" Jul 22 01:37:30.919173 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:30.919139 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:30.919301 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:30.919281 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j4s4b" podUID="f18aa922-9c77-4466-aad7-6e2f5fe12c59" Jul 22 01:37:30.919373 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:30.919333 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:37:30.919438 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:30.919420 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxfmm" podUID="271b4b39-8156-44e0-a9fb-c937501425fa" Jul 22 01:37:31.919442 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:31.919269 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:31.920015 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:31.919521 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghl8b" podUID="448e76a1-df97-4a26-8433-7a51cdeb4a5a" Jul 22 01:37:32.005125 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:32.005099 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-56.ec2.internal" event={"ID":"8f30de953d78a535da92683887f7e533","Type":"ContainerStarted","Data":"ced6c9b91b62bf7a5ff4840c0041008be85b380956f0f5faee2c58c8561668c1"} Jul 22 01:37:32.006514 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:32.006486 2561 generic.go:356] "Generic (PLEG): container finished" podID="4419f2dab53b7f8620a2511c1a87c5c9" containerID="dd345f93d363ea3e6ad9c24d1171f2dd33dfd5b95c293fd939ee976cae609263" exitCode=0 Jul 22 01:37:32.006625 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:32.006572 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-56.ec2.internal" event={"ID":"4419f2dab53b7f8620a2511c1a87c5c9","Type":"ContainerDied","Data":"dd345f93d363ea3e6ad9c24d1171f2dd33dfd5b95c293fd939ee976cae609263"} Jul 22 01:37:32.008095 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:32.008069 2561 generic.go:356] "Generic (PLEG): container finished" podID="4d9fa62a-a81a-4bdc-893e-5928340b007b" containerID="3db845c46717a7d589d4eee6e1a45db1588cf8b954da3b922fce4e9aedf43cd3" exitCode=0 Jul 22 01:37:32.008179 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:32.008162 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cpk88" event={"ID":"4d9fa62a-a81a-4bdc-893e-5928340b007b","Type":"ContainerDied","Data":"3db845c46717a7d589d4eee6e1a45db1588cf8b954da3b922fce4e9aedf43cd3"} Jul 22 01:37:32.011741 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:32.011722 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" event={"ID":"e6ceea52-1124-417d-92ff-53d3f054bded","Type":"ContainerStarted","Data":"dc4b8f46f8c56b76056f5e0dbded1761686cbbe8c2f331cfe58714a0ef9139be"} Jul 22 01:37:32.011818 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:32.011780 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" event={"ID":"e6ceea52-1124-417d-92ff-53d3f054bded","Type":"ContainerStarted","Data":"fd4d137079657b9d9a88922f5fd173a4edc09b9d4e5266c6cc41774e0f729c31"} Jul 22 01:37:32.011866 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:32.011816 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" event={"ID":"e6ceea52-1124-417d-92ff-53d3f054bded","Type":"ContainerStarted","Data":"340ca31f186a00dc790ab8d08184a1081c66b8a972ef72f18fb6e648ef3059b1"} Jul 22 01:37:32.011866 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:32.011858 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" event={"ID":"e6ceea52-1124-417d-92ff-53d3f054bded","Type":"ContainerStarted","Data":"e5cb7ff8e21be197661a8ba635a866879f5126468ea5601b006d4a00e8358236"} Jul 22 01:37:32.011950 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:32.011890 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" event={"ID":"e6ceea52-1124-417d-92ff-53d3f054bded","Type":"ContainerStarted","Data":"c665ae720321d1695cc6a9b8736595dd8dac43b0e0b5bbfda7dbb8b0d81abafc"} Jul 22 01:37:32.013052 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:32.013033 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c4c5n" event={"ID":"23698c81-8626-4b0e-963b-e20fde992ed9","Type":"ContainerStarted","Data":"209ef0f2b218bd87954897897f44de04bdcfb1da9634232cb93cdde4548da2e9"} Jul 22 01:37:32.014807 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:32.014781 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j78lg" event={"ID":"a184b54b-d2a6-4c78-aea6-714b9f0a6a1f","Type":"ContainerStarted","Data":"f5d5a694fe762b1dae44d9ad320f24f1498716b36dec1e9355e4739a9aee13ff"} Jul 22 01:37:32.016492 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:32.016465 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bbflh" event={"ID":"16abf4b8-914b-4259-b8f5-e6e51f112359","Type":"ContainerStarted","Data":"e2bda08e4d9c57a33a11cfab750f160fccc31a854374de0f748924fbcdcd1d06"} Jul 22 01:37:32.017742 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:32.017722 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk" event={"ID":"fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e","Type":"ContainerStarted","Data":"2e23c245c7c2988eef7e137038d96baa00a14ab014a41f792e98932502a1a29e"} Jul 22 01:37:32.019322 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:32.019301 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gzgpx" event={"ID":"50a4581a-21ba-4900-aee7-648e18e7abe5","Type":"ContainerStarted","Data":"7635131e63c464871b6ab67b5335697ac538720ac1b7ec14bd7ad6b9d2f92be1"} Jul 22 01:37:32.020872 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:32.020836 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-56.ec2.internal" podStartSLOduration=22.02081944 podStartE2EDuration="22.02081944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:37:32.020027054 +0000 UTC m=+22.612831418" watchObservedRunningTime="2026-07-22 01:37:32.02081944 +0000 UTC m=+22.613623822" Jul 22 01:37:32.021130 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:32.021104 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-92dh5" event={"ID":"6a4a5252-c595-42f9-834a-0c26e27adaf8","Type":"ContainerStarted","Data":"8531c2aad987e6d1032f1dd2383bf002e73f1e9f905f6da7761007908f6dcdf9"} Jul 22 01:37:32.034421 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:32.034381 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-j78lg" podStartSLOduration=2.049456963 podStartE2EDuration="22.034367489s" totalImagesPullingTime="19.984910526s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:37:32.034030565 +0000 UTC m=+22.626834929" watchObservedRunningTime="2026-07-22 01:37:32.034367489 +0000 UTC m=+22.627171852" Jul 22 01:37:32.056171 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:32.056127 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-c4c5n" podStartSLOduration=2.767723701 podStartE2EDuration="23.056108325s" totalImagesPullingTime="20.288384624s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:37:32.055216109 +0000 UTC m=+22.648020473" watchObservedRunningTime="2026-07-22 01:37:32.056108325 +0000 UTC m=+22.648912691" Jul 22 01:37:32.115208 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:32.114959 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bbflh" podStartSLOduration=2.086790809 podStartE2EDuration="22.114940028s" totalImagesPullingTime="20.028149219s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:37:32.098288288 +0000 UTC m=+22.691092645" watchObservedRunningTime="2026-07-22 01:37:32.114940028 +0000 UTC m=+22.707744393" Jul 22 01:37:32.132931 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:32.132876 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gzgpx" podStartSLOduration=7.047469431 podStartE2EDuration="22.132854457s" totalImagesPullingTime="15.085385026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:37:32.132597348 +0000 UTC m=+22.725401711" watchObservedRunningTime="2026-07-22 01:37:32.132854457 +0000 UTC m=+22.725658820" Jul 22 01:37:32.152220 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:32.152169 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-92dh5" podStartSLOduration=2.068455241 podStartE2EDuration="22.152151153s" totalImagesPullingTime="20.083695912s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:37:32.151544189 +0000 UTC m=+22.744348544" watchObservedRunningTime="2026-07-22 01:37:32.152151153 +0000 UTC m=+22.744955517" Jul 22 01:37:32.388526 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:32.388494 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gzgpx" Jul 22 01:37:32.389126 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:32.389105 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gzgpx" Jul 22 01:37:32.919462 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:32.919398 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:32.919462 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:32.919452 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:37:32.919986 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:32.919539 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j4s4b" podUID="f18aa922-9c77-4466-aad7-6e2f5fe12c59" Jul 22 01:37:32.919986 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:32.919673 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxfmm" podUID="271b4b39-8156-44e0-a9fb-c937501425fa" Jul 22 01:37:33.024649 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:33.024615 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7qn5n" event={"ID":"deed9faf-5432-4522-aa0d-362817d03ad1","Type":"ContainerStarted","Data":"2ed1da2f0546c9a708372ada12d5e98650f2a54a3663ca2477ef201476b1b68f"} Jul 22 01:37:33.026856 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:33.026832 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-56.ec2.internal" event={"ID":"4419f2dab53b7f8620a2511c1a87c5c9","Type":"ContainerStarted","Data":"cdcb3b0180f84fd6bb9bc5e197dca57164ce764c8e8f6ee89c55db1ab9a9417b"} Jul 22 01:37:33.030575 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:33.030444 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" event={"ID":"e6ceea52-1124-417d-92ff-53d3f054bded","Type":"ContainerStarted","Data":"0be21bc3de5f03cc53b431543b2c42756ae443b7868f1f99312d63ecba2b0df4"} Jul 22 01:37:33.030955 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:33.030836 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gzgpx" Jul 22 01:37:33.031240 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:33.031221 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gzgpx" Jul 22 01:37:33.058831 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:33.058784 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7qn5n" podStartSLOduration=4.339275456 podStartE2EDuration="24.058764763s" totalImagesPullingTime="19.719489307s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:37:33.043005606 +0000 UTC m=+23.635809970" watchObservedRunningTime="2026-07-22 01:37:33.058764763 +0000 UTC m=+23.651569128" Jul 22 01:37:33.074640 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:33.074588 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-56.ec2.internal" podStartSLOduration=23.074573032 podStartE2EDuration="23.074573032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:37:33.073951533 +0000 UTC m=+23.666755908" watchObservedRunningTime="2026-07-22 01:37:33.074573032 +0000 UTC m=+23.667377396" Jul 22 01:37:33.137491 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:33.137469 2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 22 01:37:33.849175 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:33.849067 2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T01:37:33.137490609Z","UUID":"4dce8168-79fe-42af-b6ee-5a95533508c5","Handler":null,"Name":"","Endpoint":""} Jul 22 01:37:33.850872 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:33.850846 2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 22 01:37:33.851008 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:33.850893 2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 22 01:37:33.918642 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:33.918611 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:33.918857 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:33.918741 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghl8b" podUID="448e76a1-df97-4a26-8433-7a51cdeb4a5a" Jul 22 01:37:34.034328 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:34.034283 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk" event={"ID":"fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e","Type":"ContainerStarted","Data":"aa29ff0a2a9888a5c4987878d71208d242d5af0b50dc3e430f807a53b422e02d"} Jul 22 01:37:34.919485 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:34.919455 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:34.919790 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:34.919581 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j4s4b" podUID="f18aa922-9c77-4466-aad7-6e2f5fe12c59" Jul 22 01:37:34.919790 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:34.919649 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:37:34.919790 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:34.919772 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxfmm" podUID="271b4b39-8156-44e0-a9fb-c937501425fa" Jul 22 01:37:35.039961 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:35.039925 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" event={"ID":"e6ceea52-1124-417d-92ff-53d3f054bded","Type":"ContainerStarted","Data":"109c8e36626b631f03e49a101894fd2355fdd0f67406065b25bbd91cb787e100"} Jul 22 01:37:35.042423 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:35.042394 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk" event={"ID":"fe79ae2d-bfd2-433d-ad58-1cf7cf2e274e","Type":"ContainerStarted","Data":"12d53a9a2f1d6026001c968ea7351070d66fd1076f3162aa1c0170b1f8f0ba53"} Jul 22 01:37:35.065630 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:35.065586 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mghsk" podStartSLOduration=2.9277818570000003 podStartE2EDuration="25.065571586s" totalImagesPullingTime="22.137789729s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:37:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:37:35.0648652 +0000 UTC m=+25.657669566" watchObservedRunningTime="2026-07-22 01:37:35.065571586 +0000 UTC m=+25.658375948" Jul 22 01:37:35.918495 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:35.918465 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:35.918677 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:35.918581 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghl8b" podUID="448e76a1-df97-4a26-8433-7a51cdeb4a5a" Jul 22 01:37:36.919298 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:36.919135 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:37:36.919883 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:36.919141 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:36.919883 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:36.919370 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxfmm" podUID="271b4b39-8156-44e0-a9fb-c937501425fa" Jul 22 01:37:36.919883 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:36.919442 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j4s4b" podUID="f18aa922-9c77-4466-aad7-6e2f5fe12c59" Jul 22 01:37:37.047792 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:37.047761 2561 generic.go:356] "Generic (PLEG): container finished" podID="4d9fa62a-a81a-4bdc-893e-5928340b007b" containerID="f16aaee02b6cbf09325d7f75aa6dddcff686e5f2d5874b542821ee443797652f" exitCode=0 Jul 22 01:37:37.047979 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:37.047850 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cpk88" event={"ID":"4d9fa62a-a81a-4bdc-893e-5928340b007b","Type":"ContainerDied","Data":"f16aaee02b6cbf09325d7f75aa6dddcff686e5f2d5874b542821ee443797652f"} Jul 22 01:37:37.051460 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:37.051439 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" event={"ID":"e6ceea52-1124-417d-92ff-53d3f054bded","Type":"ContainerStarted","Data":"a6394b75da91afbd6f923fd17f1054a0854cad794980b368f7dd1fccf59eba6d"} Jul 22 01:37:37.051706 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:37.051677 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:37.051777 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:37.051740 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:37.051777 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:37.051774 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:37.070102 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:37.070081 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:37.070238 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:37.070223 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:37:37.097727 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:37.097666 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" podStartSLOduration=7.934412912 podStartE2EDuration="28.09765236s" totalImagesPullingTime="20.163239448s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:37:37.097136695 +0000 UTC m=+27.689941073" watchObservedRunningTime="2026-07-22 01:37:37.09765236 +0000 UTC m=+27.690456722" Jul 22 01:37:37.919405 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:37.919377 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:37.920101 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:37.919508 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghl8b" podUID="448e76a1-df97-4a26-8433-7a51cdeb4a5a" Jul 22 01:37:38.274408 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:38.274222 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-j4s4b"] Jul 22 01:37:38.274539 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:38.274528 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:38.274672 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:38.274649 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j4s4b" podUID="f18aa922-9c77-4466-aad7-6e2f5fe12c59" Jul 22 01:37:38.277377 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:38.277323 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fxfmm"] Jul 22 01:37:38.277511 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:38.277464 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:37:38.277583 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:38.277561 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxfmm" podUID="271b4b39-8156-44e0-a9fb-c937501425fa" Jul 22 01:37:38.277946 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:38.277923 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ghl8b"] Jul 22 01:37:38.278157 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:38.278000 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:38.278214 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:38.278167 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghl8b" podUID="448e76a1-df97-4a26-8433-7a51cdeb4a5a" Jul 22 01:37:39.057761 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:39.057728 2561 generic.go:356] "Generic (PLEG): container finished" podID="4d9fa62a-a81a-4bdc-893e-5928340b007b" containerID="9843fd34a250749d17e1cefd6fbb8d230bb3e689b2d76b27d9d3b71065c23228" exitCode=0 Jul 22 01:37:39.058196 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:39.057817 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cpk88" event={"ID":"4d9fa62a-a81a-4bdc-893e-5928340b007b","Type":"ContainerDied","Data":"9843fd34a250749d17e1cefd6fbb8d230bb3e689b2d76b27d9d3b71065c23228"} Jul 22 01:37:39.919270 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:39.919235 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:37:39.919462 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:39.919310 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxfmm" podUID="271b4b39-8156-44e0-a9fb-c937501425fa" Jul 22 01:37:39.919462 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:39.919356 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:39.919462 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:39.919401 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghl8b" podUID="448e76a1-df97-4a26-8433-7a51cdeb4a5a" Jul 22 01:37:39.919462 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:39.919431 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:39.919663 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:39.919478 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j4s4b" podUID="f18aa922-9c77-4466-aad7-6e2f5fe12c59" Jul 22 01:37:41.066063 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:41.066030 2561 generic.go:356] "Generic (PLEG): container finished" podID="4d9fa62a-a81a-4bdc-893e-5928340b007b" containerID="7f3e0b7ed1bf13433712fc44d57c0267bdb0f03aac10125d23ee33e5e70c9026" exitCode=0 Jul 22 01:37:41.066414 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:41.066108 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cpk88" event={"ID":"4d9fa62a-a81a-4bdc-893e-5928340b007b","Type":"ContainerDied","Data":"7f3e0b7ed1bf13433712fc44d57c0267bdb0f03aac10125d23ee33e5e70c9026"} Jul 22 01:37:41.918955 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:41.918921 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:37:41.919116 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:41.918931 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:41.919116 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:41.919039 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxfmm" podUID="271b4b39-8156-44e0-a9fb-c937501425fa" Jul 22 01:37:41.919116 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:41.918933 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:41.919264 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:41.919115 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j4s4b" podUID="f18aa922-9c77-4466-aad7-6e2f5fe12c59" Jul 22 01:37:41.919264 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:41.919186 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghl8b" podUID="448e76a1-df97-4a26-8433-7a51cdeb4a5a" Jul 22 01:37:42.554300 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:42.554270 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/448e76a1-df97-4a26-8433-7a51cdeb4a5a-original-pull-secret\") pod \"global-pull-secret-syncer-ghl8b\" (UID: \"448e76a1-df97-4a26-8433-7a51cdeb4a5a\") " pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:42.554713 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:42.554318 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f18aa922-9c77-4466-aad7-6e2f5fe12c59-metrics-certs\") pod \"network-metrics-daemon-j4s4b\" (UID: \"f18aa922-9c77-4466-aad7-6e2f5fe12c59\") " pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:42.554713 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:42.554432 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:37:42.554713 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:42.554436 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 01:37:42.554713 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:42.554490 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f18aa922-9c77-4466-aad7-6e2f5fe12c59-metrics-certs podName:f18aa922-9c77-4466-aad7-6e2f5fe12c59 nodeName:}" failed. No retries permitted until 2026-07-22 01:38:14.554471039 +0000 UTC m=+65.147275457 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f18aa922-9c77-4466-aad7-6e2f5fe12c59-metrics-certs") pod "network-metrics-daemon-j4s4b" (UID: "f18aa922-9c77-4466-aad7-6e2f5fe12c59") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 01:37:42.554713 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:42.554504 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448e76a1-df97-4a26-8433-7a51cdeb4a5a-original-pull-secret podName:448e76a1-df97-4a26-8433-7a51cdeb4a5a nodeName:}" failed. No retries permitted until 2026-07-22 01:38:14.554497956 +0000 UTC m=+65.147302296 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/448e76a1-df97-4a26-8433-7a51cdeb4a5a-original-pull-secret") pod "global-pull-secret-syncer-ghl8b" (UID: "448e76a1-df97-4a26-8433-7a51cdeb4a5a") : object "kube-system"/"original-pull-secret" not registered Jul 22 01:37:42.654847 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:42.654817 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jglhg\" (UniqueName: \"kubernetes.io/projected/271b4b39-8156-44e0-a9fb-c937501425fa-kube-api-access-jglhg\") pod \"network-check-target-fxfmm\" (UID: \"271b4b39-8156-44e0-a9fb-c937501425fa\") " pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:37:42.655006 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:42.654967 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 01:37:42.655006 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:42.654985 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 01:37:42.655006 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:42.654993 2561 projected.go:196] Error preparing data for projected volume kube-api-access-jglhg for pod openshift-network-diagnostics/network-check-target-fxfmm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:37:42.655153 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:42.655043 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/271b4b39-8156-44e0-a9fb-c937501425fa-kube-api-access-jglhg podName:271b4b39-8156-44e0-a9fb-c937501425fa nodeName:}" failed. No retries permitted until 2026-07-22 01:38:14.655024528 +0000 UTC m=+65.247828872 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jglhg" (UniqueName: "kubernetes.io/projected/271b4b39-8156-44e0-a9fb-c937501425fa-kube-api-access-jglhg") pod "network-check-target-fxfmm" (UID: "271b4b39-8156-44e0-a9fb-c937501425fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 01:37:43.918906 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:43.918883 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:37:43.918906 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:43.918900 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:43.919351 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:43.918884 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:43.919351 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:43.919056 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ghl8b" podUID="448e76a1-df97-4a26-8433-7a51cdeb4a5a" Jul 22 01:37:43.919351 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:43.919061 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j4s4b" podUID="f18aa922-9c77-4466-aad7-6e2f5fe12c59" Jul 22 01:37:43.919351 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:43.919112 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxfmm" podUID="271b4b39-8156-44e0-a9fb-c937501425fa" Jul 22 01:37:44.274026 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.273994 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-56.ec2.internal" event="NodeReady" Jul 22 01:37:44.274213 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.274137 2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 22 01:37:44.319464 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.319434 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-drsdb"] Jul 22 01:37:44.322193 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.322161 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ddbnn"] Jul 22 01:37:44.322904 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.322168 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-drsdb" Jul 22 01:37:44.324043 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.324024 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ddbnn" Jul 22 01:37:44.325211 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.325186 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 22 01:37:44.325373 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.325190 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fhl28\"" Jul 22 01:37:44.325568 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.325549 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 22 01:37:44.326603 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.326288 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 22 01:37:44.326603 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.326340 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 22 01:37:44.326603 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.326507 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 22 01:37:44.326828 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.326767 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-v8ltf\"" Jul 22 01:37:44.332284 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.331408 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-drsdb"] Jul 22 01:37:44.338839 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.338814 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ddbnn"] Jul 22 01:37:44.467203 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.467173 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8510529c-89e5-48b1-bbc0-3f3ccd912896-config-volume\") pod \"dns-default-drsdb\" (UID: \"8510529c-89e5-48b1-bbc0-3f3ccd912896\") " pod="openshift-dns/dns-default-drsdb" Jul 22 01:37:44.467203 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.467207 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmldm\" (UniqueName: \"kubernetes.io/projected/8f8e967e-e9b7-47c7-83b0-9fedc09ce1d5-kube-api-access-zmldm\") pod \"ingress-canary-ddbnn\" (UID: \"8f8e967e-e9b7-47c7-83b0-9fedc09ce1d5\") " pod="openshift-ingress-canary/ingress-canary-ddbnn" Jul 22 01:37:44.467440 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.467241 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8510529c-89e5-48b1-bbc0-3f3ccd912896-metrics-tls\") pod \"dns-default-drsdb\" (UID: \"8510529c-89e5-48b1-bbc0-3f3ccd912896\") " pod="openshift-dns/dns-default-drsdb" Jul 22 01:37:44.467440 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.467266 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8510529c-89e5-48b1-bbc0-3f3ccd912896-tmp-dir\") pod \"dns-default-drsdb\" (UID: \"8510529c-89e5-48b1-bbc0-3f3ccd912896\") " pod="openshift-dns/dns-default-drsdb" Jul 22 01:37:44.467440 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.467301 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-676k2\" (UniqueName: \"kubernetes.io/projected/8510529c-89e5-48b1-bbc0-3f3ccd912896-kube-api-access-676k2\") pod \"dns-default-drsdb\" (UID: \"8510529c-89e5-48b1-bbc0-3f3ccd912896\") " pod="openshift-dns/dns-default-drsdb" Jul 22 01:37:44.467440 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.467326 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f8e967e-e9b7-47c7-83b0-9fedc09ce1d5-cert\") pod \"ingress-canary-ddbnn\" (UID: \"8f8e967e-e9b7-47c7-83b0-9fedc09ce1d5\") " pod="openshift-ingress-canary/ingress-canary-ddbnn" Jul 22 01:37:44.568623 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.568535 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-676k2\" (UniqueName: \"kubernetes.io/projected/8510529c-89e5-48b1-bbc0-3f3ccd912896-kube-api-access-676k2\") pod \"dns-default-drsdb\" (UID: \"8510529c-89e5-48b1-bbc0-3f3ccd912896\") " pod="openshift-dns/dns-default-drsdb" Jul 22 01:37:44.568623 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.568586 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f8e967e-e9b7-47c7-83b0-9fedc09ce1d5-cert\") pod \"ingress-canary-ddbnn\" (UID: \"8f8e967e-e9b7-47c7-83b0-9fedc09ce1d5\") " pod="openshift-ingress-canary/ingress-canary-ddbnn" Jul 22 01:37:44.568870 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.568667 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8510529c-89e5-48b1-bbc0-3f3ccd912896-config-volume\") pod \"dns-default-drsdb\" (UID: \"8510529c-89e5-48b1-bbc0-3f3ccd912896\") " pod="openshift-dns/dns-default-drsdb" Jul 22 01:37:44.568870 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.568810 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmldm\" (UniqueName: \"kubernetes.io/projected/8f8e967e-e9b7-47c7-83b0-9fedc09ce1d5-kube-api-access-zmldm\") pod \"ingress-canary-ddbnn\" (UID: \"8f8e967e-e9b7-47c7-83b0-9fedc09ce1d5\") " pod="openshift-ingress-canary/ingress-canary-ddbnn" Jul 22 01:37:44.568870 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.568863 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8510529c-89e5-48b1-bbc0-3f3ccd912896-metrics-tls\") pod \"dns-default-drsdb\" (UID: \"8510529c-89e5-48b1-bbc0-3f3ccd912896\") " pod="openshift-dns/dns-default-drsdb" Jul 22 01:37:44.569015 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.568886 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8510529c-89e5-48b1-bbc0-3f3ccd912896-tmp-dir\") pod \"dns-default-drsdb\" (UID: \"8510529c-89e5-48b1-bbc0-3f3ccd912896\") " pod="openshift-dns/dns-default-drsdb" Jul 22 01:37:44.569175 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.569147 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8510529c-89e5-48b1-bbc0-3f3ccd912896-tmp-dir\") pod \"dns-default-drsdb\" (UID: \"8510529c-89e5-48b1-bbc0-3f3ccd912896\") " pod="openshift-dns/dns-default-drsdb" Jul 22 01:37:44.569301 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:44.569175 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 01:37:44.569301 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.569192 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8510529c-89e5-48b1-bbc0-3f3ccd912896-config-volume\") pod \"dns-default-drsdb\" (UID: \"8510529c-89e5-48b1-bbc0-3f3ccd912896\") " pod="openshift-dns/dns-default-drsdb" Jul 22 01:37:44.569301 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:44.569239 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8510529c-89e5-48b1-bbc0-3f3ccd912896-metrics-tls podName:8510529c-89e5-48b1-bbc0-3f3ccd912896 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:45.069221905 +0000 UTC m=+35.662026245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8510529c-89e5-48b1-bbc0-3f3ccd912896-metrics-tls") pod "dns-default-drsdb" (UID: "8510529c-89e5-48b1-bbc0-3f3ccd912896") : secret "dns-default-metrics-tls" not found Jul 22 01:37:44.573715 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.573584 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f8e967e-e9b7-47c7-83b0-9fedc09ce1d5-cert\") pod \"ingress-canary-ddbnn\" (UID: \"8f8e967e-e9b7-47c7-83b0-9fedc09ce1d5\") " pod="openshift-ingress-canary/ingress-canary-ddbnn" Jul 22 01:37:44.580153 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.580133 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-676k2\" (UniqueName: \"kubernetes.io/projected/8510529c-89e5-48b1-bbc0-3f3ccd912896-kube-api-access-676k2\") pod \"dns-default-drsdb\" (UID: \"8510529c-89e5-48b1-bbc0-3f3ccd912896\") " pod="openshift-dns/dns-default-drsdb" Jul 22 01:37:44.580558 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.580541 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmldm\" (UniqueName: \"kubernetes.io/projected/8f8e967e-e9b7-47c7-83b0-9fedc09ce1d5-kube-api-access-zmldm\") pod \"ingress-canary-ddbnn\" (UID: \"8f8e967e-e9b7-47c7-83b0-9fedc09ce1d5\") " pod="openshift-ingress-canary/ingress-canary-ddbnn" Jul 22 01:37:44.648552 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:44.648520 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ddbnn" Jul 22 01:37:45.073343 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:45.073307 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8510529c-89e5-48b1-bbc0-3f3ccd912896-metrics-tls\") pod \"dns-default-drsdb\" (UID: \"8510529c-89e5-48b1-bbc0-3f3ccd912896\") " pod="openshift-dns/dns-default-drsdb" Jul 22 01:37:45.073760 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:45.073444 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 01:37:45.073760 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:45.073503 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8510529c-89e5-48b1-bbc0-3f3ccd912896-metrics-tls podName:8510529c-89e5-48b1-bbc0-3f3ccd912896 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:46.073484918 +0000 UTC m=+36.666289259 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8510529c-89e5-48b1-bbc0-3f3ccd912896-metrics-tls") pod "dns-default-drsdb" (UID: "8510529c-89e5-48b1-bbc0-3f3ccd912896") : secret "dns-default-metrics-tls" not found Jul 22 01:37:45.789226 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:45.789201 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ddbnn"] Jul 22 01:37:45.792681 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:45.792659 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8e967e_e9b7_47c7_83b0_9fedc09ce1d5.slice/crio-35675c6b53ecef1e2a686f2af1ce385dc28c8853bd2998095357c883918b16d5 WatchSource:0}: Error finding container 35675c6b53ecef1e2a686f2af1ce385dc28c8853bd2998095357c883918b16d5: Status 404 returned error can't find the container with id 35675c6b53ecef1e2a686f2af1ce385dc28c8853bd2998095357c883918b16d5 Jul 22 01:37:45.919183 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:45.919129 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:37:45.919275 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:45.919129 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:37:45.919338 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:45.919269 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:37:45.922847 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:45.922805 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 01:37:45.922847 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:45.922825 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 01:37:45.923025 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:45.922851 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 01:37:45.923025 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:45.922883 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 01:37:45.923025 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:45.922825 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tqzqp\"" Jul 22 01:37:45.923025 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:45.922825 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fbrqb\"" Jul 22 01:37:46.078171 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:46.078138 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ddbnn" event={"ID":"8f8e967e-e9b7-47c7-83b0-9fedc09ce1d5","Type":"ContainerStarted","Data":"35675c6b53ecef1e2a686f2af1ce385dc28c8853bd2998095357c883918b16d5"} Jul 22 01:37:46.080943 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:46.080919 2561 generic.go:356] "Generic (PLEG): container finished" podID="4d9fa62a-a81a-4bdc-893e-5928340b007b" containerID="9604da8779c623b0d45dfd713bdf6519d19bbfe35c83f7e66f2f399963432986" exitCode=0 Jul 22 01:37:46.081067 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:46.080969 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cpk88" event={"ID":"4d9fa62a-a81a-4bdc-893e-5928340b007b","Type":"ContainerDied","Data":"9604da8779c623b0d45dfd713bdf6519d19bbfe35c83f7e66f2f399963432986"} Jul 22 01:37:46.082802 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:46.082776 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8510529c-89e5-48b1-bbc0-3f3ccd912896-metrics-tls\") pod \"dns-default-drsdb\" (UID: \"8510529c-89e5-48b1-bbc0-3f3ccd912896\") " pod="openshift-dns/dns-default-drsdb" Jul 22 01:37:46.082910 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:46.082895 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 01:37:46.082963 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:46.082954 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8510529c-89e5-48b1-bbc0-3f3ccd912896-metrics-tls podName:8510529c-89e5-48b1-bbc0-3f3ccd912896 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:48.082940711 +0000 UTC m=+38.675745053 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8510529c-89e5-48b1-bbc0-3f3ccd912896-metrics-tls") pod "dns-default-drsdb" (UID: "8510529c-89e5-48b1-bbc0-3f3ccd912896") : secret "dns-default-metrics-tls" not found Jul 22 01:37:47.086303 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:47.086116 2561 generic.go:356] "Generic (PLEG): container finished" podID="4d9fa62a-a81a-4bdc-893e-5928340b007b" containerID="969fee062d098b3de6cb87e484544fef1339767f30e95aedf156f52384a64041" exitCode=0 Jul 22 01:37:47.086657 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:47.086159 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cpk88" event={"ID":"4d9fa62a-a81a-4bdc-893e-5928340b007b","Type":"ContainerDied","Data":"969fee062d098b3de6cb87e484544fef1339767f30e95aedf156f52384a64041"} Jul 22 01:37:48.093301 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.093214 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cpk88" event={"ID":"4d9fa62a-a81a-4bdc-893e-5928340b007b","Type":"ContainerStarted","Data":"921d98c4873495b650a5cdd5c6dcaf028295b6af617c833cf68d2736152ce0bf"} Jul 22 01:37:48.098065 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.098040 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8510529c-89e5-48b1-bbc0-3f3ccd912896-metrics-tls\") pod \"dns-default-drsdb\" (UID: \"8510529c-89e5-48b1-bbc0-3f3ccd912896\") " pod="openshift-dns/dns-default-drsdb" Jul 22 01:37:48.098229 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:48.098187 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 01:37:48.098304 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:48.098286 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8510529c-89e5-48b1-bbc0-3f3ccd912896-metrics-tls podName:8510529c-89e5-48b1-bbc0-3f3ccd912896 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:52.098267008 +0000 UTC m=+42.691071349 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8510529c-89e5-48b1-bbc0-3f3ccd912896-metrics-tls") pod "dns-default-drsdb" (UID: "8510529c-89e5-48b1-bbc0-3f3ccd912896") : secret "dns-default-metrics-tls" not found Jul 22 01:37:48.115833 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.115793 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cpk88" podStartSLOduration=12.900770102 podStartE2EDuration="39.115778488s" totalImagesPullingTime="26.215008386s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:37:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:37:48.114356373 +0000 UTC m=+38.707160736" watchObservedRunningTime="2026-07-22 01:37:48.115778488 +0000 UTC m=+38.708582851" Jul 22 01:37:48.704607 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.704576 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vmk5j"] Jul 22 01:37:48.713521 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.713489 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vmk5j" Jul 22 01:37:48.716516 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.716490 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 22 01:37:48.716660 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.716518 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 22 01:37:48.716768 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.716654 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l77sq\"" Jul 22 01:37:48.717065 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.717046 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 22 01:37:48.721742 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.719885 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 22 01:37:48.721742 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.721546 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vmk5j"] Jul 22 01:37:48.803837 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.803804 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1a297949-cd40-49e4-8643-0f3610080c45-crio-socket\") pod \"insights-runtime-extractor-vmk5j\" (UID: \"1a297949-cd40-49e4-8643-0f3610080c45\") " pod="openshift-insights/insights-runtime-extractor-vmk5j" Jul 22 01:37:48.803837 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.803838 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbtvf\" (UniqueName: \"kubernetes.io/projected/1a297949-cd40-49e4-8643-0f3610080c45-kube-api-access-rbtvf\") pod \"insights-runtime-extractor-vmk5j\" (UID: \"1a297949-cd40-49e4-8643-0f3610080c45\") " pod="openshift-insights/insights-runtime-extractor-vmk5j" Jul 22 01:37:48.804070 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.803872 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a297949-cd40-49e4-8643-0f3610080c45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vmk5j\" (UID: \"1a297949-cd40-49e4-8643-0f3610080c45\") " pod="openshift-insights/insights-runtime-extractor-vmk5j" Jul 22 01:37:48.804070 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.803961 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a297949-cd40-49e4-8643-0f3610080c45-data-volume\") pod \"insights-runtime-extractor-vmk5j\" (UID: \"1a297949-cd40-49e4-8643-0f3610080c45\") " pod="openshift-insights/insights-runtime-extractor-vmk5j" Jul 22 01:37:48.804070 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.804037 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a297949-cd40-49e4-8643-0f3610080c45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmk5j\" (UID: \"1a297949-cd40-49e4-8643-0f3610080c45\") " pod="openshift-insights/insights-runtime-extractor-vmk5j" Jul 22 01:37:48.904823 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.904790 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1a297949-cd40-49e4-8643-0f3610080c45-crio-socket\") pod \"insights-runtime-extractor-vmk5j\" (UID: \"1a297949-cd40-49e4-8643-0f3610080c45\") " pod="openshift-insights/insights-runtime-extractor-vmk5j" Jul 22 01:37:48.904989 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.904832 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbtvf\" (UniqueName: \"kubernetes.io/projected/1a297949-cd40-49e4-8643-0f3610080c45-kube-api-access-rbtvf\") pod \"insights-runtime-extractor-vmk5j\" (UID: \"1a297949-cd40-49e4-8643-0f3610080c45\") " pod="openshift-insights/insights-runtime-extractor-vmk5j" Jul 22 01:37:48.904989 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.904870 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a297949-cd40-49e4-8643-0f3610080c45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vmk5j\" (UID: \"1a297949-cd40-49e4-8643-0f3610080c45\") " pod="openshift-insights/insights-runtime-extractor-vmk5j" Jul 22 01:37:48.904989 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.904903 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a297949-cd40-49e4-8643-0f3610080c45-data-volume\") pod \"insights-runtime-extractor-vmk5j\" (UID: \"1a297949-cd40-49e4-8643-0f3610080c45\") " pod="openshift-insights/insights-runtime-extractor-vmk5j" Jul 22 01:37:48.904989 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.904970 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a297949-cd40-49e4-8643-0f3610080c45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmk5j\" (UID: \"1a297949-cd40-49e4-8643-0f3610080c45\") " pod="openshift-insights/insights-runtime-extractor-vmk5j" Jul 22 01:37:48.904989 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:48.904981 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 01:37:48.905238 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:48.905053 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a297949-cd40-49e4-8643-0f3610080c45-kube-rbac-proxy-cm podName:1a297949-cd40-49e4-8643-0f3610080c45 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:49.405035658 +0000 UTC m=+39.997839999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1a297949-cd40-49e4-8643-0f3610080c45-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vmk5j" (UID: "1a297949-cd40-49e4-8643-0f3610080c45") : configmap "kube-rbac-proxy" not found Jul 22 01:37:48.905238 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.905047 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1a297949-cd40-49e4-8643-0f3610080c45-crio-socket\") pod \"insights-runtime-extractor-vmk5j\" (UID: \"1a297949-cd40-49e4-8643-0f3610080c45\") " pod="openshift-insights/insights-runtime-extractor-vmk5j" Jul 22 01:37:48.905238 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:48.905110 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 01:37:48.905238 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:48.905169 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a297949-cd40-49e4-8643-0f3610080c45-insights-runtime-extractor-tls podName:1a297949-cd40-49e4-8643-0f3610080c45 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:49.40515316 +0000 UTC m=+39.997957503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a297949-cd40-49e4-8643-0f3610080c45-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vmk5j" (UID: "1a297949-cd40-49e4-8643-0f3610080c45") : secret "insights-runtime-extractor-tls" not found Jul 22 01:37:48.905366 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.905248 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a297949-cd40-49e4-8643-0f3610080c45-data-volume\") pod \"insights-runtime-extractor-vmk5j\" (UID: \"1a297949-cd40-49e4-8643-0f3610080c45\") " pod="openshift-insights/insights-runtime-extractor-vmk5j" Jul 22 01:37:48.917072 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.917049 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbtvf\" (UniqueName: \"kubernetes.io/projected/1a297949-cd40-49e4-8643-0f3610080c45-kube-api-access-rbtvf\") pod \"insights-runtime-extractor-vmk5j\" (UID: \"1a297949-cd40-49e4-8643-0f3610080c45\") " pod="openshift-insights/insights-runtime-extractor-vmk5j" Jul 22 01:37:48.939496 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.939470 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-wbzgd"] Jul 22 01:37:48.954848 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.954735 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-wbzgd" Jul 22 01:37:48.955821 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.955800 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-wbzgd"] Jul 22 01:37:48.957219 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.957197 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 22 01:37:48.957321 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.957303 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 22 01:37:48.957872 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:48.957852 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-nn5nv\"" Jul 22 01:37:49.105949 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:49.105921 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dlwf\" (UniqueName: \"kubernetes.io/projected/d1b2434a-a884-4c5b-aff6-338544f7c60b-kube-api-access-2dlwf\") pod \"migrator-7cc9c7ffd7-wbzgd\" (UID: \"d1b2434a-a884-4c5b-aff6-338544f7c60b\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-wbzgd" Jul 22 01:37:49.206465 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:49.206395 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dlwf\" (UniqueName: \"kubernetes.io/projected/d1b2434a-a884-4c5b-aff6-338544f7c60b-kube-api-access-2dlwf\") pod \"migrator-7cc9c7ffd7-wbzgd\" (UID: \"d1b2434a-a884-4c5b-aff6-338544f7c60b\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-wbzgd" Jul 22 01:37:49.215965 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:49.215941 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dlwf\" (UniqueName: \"kubernetes.io/projected/d1b2434a-a884-4c5b-aff6-338544f7c60b-kube-api-access-2dlwf\") pod \"migrator-7cc9c7ffd7-wbzgd\" (UID: \"d1b2434a-a884-4c5b-aff6-338544f7c60b\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-wbzgd" Jul 22 01:37:49.264515 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:49.264494 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-wbzgd" Jul 22 01:37:49.407260 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:49.407236 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a297949-cd40-49e4-8643-0f3610080c45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vmk5j\" (UID: \"1a297949-cd40-49e4-8643-0f3610080c45\") " pod="openshift-insights/insights-runtime-extractor-vmk5j" Jul 22 01:37:49.407382 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:49.407305 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a297949-cd40-49e4-8643-0f3610080c45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmk5j\" (UID: \"1a297949-cd40-49e4-8643-0f3610080c45\") " pod="openshift-insights/insights-runtime-extractor-vmk5j" Jul 22 01:37:49.407382 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:49.407362 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 01:37:49.407571 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:49.407400 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 01:37:49.407571 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:49.407425 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a297949-cd40-49e4-8643-0f3610080c45-kube-rbac-proxy-cm podName:1a297949-cd40-49e4-8643-0f3610080c45 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:50.407406688 +0000 UTC m=+41.000211048 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1a297949-cd40-49e4-8643-0f3610080c45-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vmk5j" (UID: "1a297949-cd40-49e4-8643-0f3610080c45") : configmap "kube-rbac-proxy" not found Jul 22 01:37:49.407571 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:49.407443 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a297949-cd40-49e4-8643-0f3610080c45-insights-runtime-extractor-tls podName:1a297949-cd40-49e4-8643-0f3610080c45 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:50.407434557 +0000 UTC m=+41.000238901 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a297949-cd40-49e4-8643-0f3610080c45-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vmk5j" (UID: "1a297949-cd40-49e4-8643-0f3610080c45") : secret "insights-runtime-extractor-tls" not found Jul 22 01:37:49.432267 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:49.432235 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-wbzgd"] Jul 22 01:37:49.435547 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:37:49.435522 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b2434a_a884_4c5b_aff6_338544f7c60b.slice/crio-922c121995016f7d120c229262c68635654fbfdb73f24f4e28ff20ac98f07d8c WatchSource:0}: Error finding container 922c121995016f7d120c229262c68635654fbfdb73f24f4e28ff20ac98f07d8c: Status 404 returned error can't find the container with id 922c121995016f7d120c229262c68635654fbfdb73f24f4e28ff20ac98f07d8c Jul 22 01:37:50.098283 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:50.098251 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-wbzgd" event={"ID":"d1b2434a-a884-4c5b-aff6-338544f7c60b","Type":"ContainerStarted","Data":"922c121995016f7d120c229262c68635654fbfdb73f24f4e28ff20ac98f07d8c"} Jul 22 01:37:50.099431 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:50.099408 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ddbnn" event={"ID":"8f8e967e-e9b7-47c7-83b0-9fedc09ce1d5","Type":"ContainerStarted","Data":"eb6c4c6fe07e07d955de8e6fbbf9b0e20beb1f836f9483ecb019ce623e4a48a4"} Jul 22 01:37:50.116612 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:50.116568 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ddbnn" podStartSLOduration=2.610557838 podStartE2EDuration="6.116555577s" totalImagesPullingTime="3.505997739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:37:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:37:50.115833578 +0000 UTC m=+40.708637952" watchObservedRunningTime="2026-07-22 01:37:50.116555577 +0000 UTC m=+40.709359918" Jul 22 01:37:50.418952 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:50.418874 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a297949-cd40-49e4-8643-0f3610080c45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmk5j\" (UID: \"1a297949-cd40-49e4-8643-0f3610080c45\") " pod="openshift-insights/insights-runtime-extractor-vmk5j" Jul 22 01:37:50.418952 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:50.418939 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a297949-cd40-49e4-8643-0f3610080c45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vmk5j\" (UID: \"1a297949-cd40-49e4-8643-0f3610080c45\") " pod="openshift-insights/insights-runtime-extractor-vmk5j" Jul 22 01:37:50.419114 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:50.419005 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 01:37:50.419114 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:50.419020 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 01:37:50.419114 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:50.419058 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a297949-cd40-49e4-8643-0f3610080c45-kube-rbac-proxy-cm podName:1a297949-cd40-49e4-8643-0f3610080c45 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:52.419043798 +0000 UTC m=+43.011848139 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1a297949-cd40-49e4-8643-0f3610080c45-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vmk5j" (UID: "1a297949-cd40-49e4-8643-0f3610080c45") : configmap "kube-rbac-proxy" not found Jul 22 01:37:50.419114 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:50.419076 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a297949-cd40-49e4-8643-0f3610080c45-insights-runtime-extractor-tls podName:1a297949-cd40-49e4-8643-0f3610080c45 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:52.419063239 +0000 UTC m=+43.011867581 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a297949-cd40-49e4-8643-0f3610080c45-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vmk5j" (UID: "1a297949-cd40-49e4-8643-0f3610080c45") : secret "insights-runtime-extractor-tls" not found Jul 22 01:37:50.590041 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:50.590009 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-92dh5_6a4a5252-c595-42f9-834a-0c26e27adaf8/dns-node-resolver/0.log" Jul 22 01:37:51.589928 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:51.589899 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j78lg_a184b54b-d2a6-4c78-aea6-714b9f0a6a1f/node-ca/0.log" Jul 22 01:37:52.130879 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:52.130837 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8510529c-89e5-48b1-bbc0-3f3ccd912896-metrics-tls\") pod \"dns-default-drsdb\" (UID: \"8510529c-89e5-48b1-bbc0-3f3ccd912896\") " pod="openshift-dns/dns-default-drsdb" Jul 22 01:37:52.131069 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:52.131007 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 01:37:52.131155 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:52.131092 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8510529c-89e5-48b1-bbc0-3f3ccd912896-metrics-tls podName:8510529c-89e5-48b1-bbc0-3f3ccd912896 nodeName:}" failed. No retries permitted until 2026-07-22 01:38:00.131070637 +0000 UTC m=+50.723874981 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8510529c-89e5-48b1-bbc0-3f3ccd912896-metrics-tls") pod "dns-default-drsdb" (UID: "8510529c-89e5-48b1-bbc0-3f3ccd912896") : secret "dns-default-metrics-tls" not found Jul 22 01:37:52.390871 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:52.390758 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ddbnn_8f8e967e-e9b7-47c7-83b0-9fedc09ce1d5/serve-healthcheck-canary/0.log" Jul 22 01:37:52.434927 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:52.434896 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a297949-cd40-49e4-8643-0f3610080c45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vmk5j\" (UID: \"1a297949-cd40-49e4-8643-0f3610080c45\") " pod="openshift-insights/insights-runtime-extractor-vmk5j" Jul 22 01:37:52.435046 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:52.434964 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a297949-cd40-49e4-8643-0f3610080c45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmk5j\" (UID: \"1a297949-cd40-49e4-8643-0f3610080c45\") " pod="openshift-insights/insights-runtime-extractor-vmk5j" Jul 22 01:37:52.435046 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:52.435023 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 01:37:52.435125 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:52.435087 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 01:37:52.435125 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:52.435088 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a297949-cd40-49e4-8643-0f3610080c45-kube-rbac-proxy-cm podName:1a297949-cd40-49e4-8643-0f3610080c45 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:56.435071405 +0000 UTC m=+47.027875751 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1a297949-cd40-49e4-8643-0f3610080c45-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vmk5j" (UID: "1a297949-cd40-49e4-8643-0f3610080c45") : configmap "kube-rbac-proxy" not found Jul 22 01:37:52.435199 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:52.435129 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a297949-cd40-49e4-8643-0f3610080c45-insights-runtime-extractor-tls podName:1a297949-cd40-49e4-8643-0f3610080c45 nodeName:}" failed. No retries permitted until 2026-07-22 01:37:56.435119088 +0000 UTC m=+47.027923428 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a297949-cd40-49e4-8643-0f3610080c45-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vmk5j" (UID: "1a297949-cd40-49e4-8643-0f3610080c45") : secret "insights-runtime-extractor-tls" not found Jul 22 01:37:53.108378 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:53.108348 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-wbzgd" event={"ID":"d1b2434a-a884-4c5b-aff6-338544f7c60b","Type":"ContainerStarted","Data":"14eba2061f365f5936f2270a564fc52de2774cb10cd5759aa1014876c86188bf"} Jul 22 01:37:53.108745 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:53.108427 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-wbzgd" event={"ID":"d1b2434a-a884-4c5b-aff6-338544f7c60b","Type":"ContainerStarted","Data":"34aa0752bfba5c9f4237fbef9ca266b5d6ebb7c880f2d8d82c18bb97ac95e872"} Jul 22 01:37:53.124950 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:53.124907 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-wbzgd" podStartSLOduration=2.006138936 podStartE2EDuration="5.124892304s" totalImagesPullingTime="3.118753368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:37:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:37:53.124118656 +0000 UTC m=+43.716923019" watchObservedRunningTime="2026-07-22 01:37:53.124892304 +0000 UTC m=+43.717696704" Jul 22 01:37:56.464363 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:56.464322 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a297949-cd40-49e4-8643-0f3610080c45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmk5j\" (UID: \"1a297949-cd40-49e4-8643-0f3610080c45\") " pod="openshift-insights/insights-runtime-extractor-vmk5j" Jul 22 01:37:56.464861 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:37:56.464397 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a297949-cd40-49e4-8643-0f3610080c45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vmk5j\" (UID: \"1a297949-cd40-49e4-8643-0f3610080c45\") " pod="openshift-insights/insights-runtime-extractor-vmk5j" Jul 22 01:37:56.464861 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:56.464498 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 01:37:56.464861 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:56.464513 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 01:37:56.464861 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:56.464563 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a297949-cd40-49e4-8643-0f3610080c45-kube-rbac-proxy-cm podName:1a297949-cd40-49e4-8643-0f3610080c45 nodeName:}" failed. No retries permitted until 2026-07-22 01:38:04.464547695 +0000 UTC m=+55.057352036 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1a297949-cd40-49e4-8643-0f3610080c45-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vmk5j" (UID: "1a297949-cd40-49e4-8643-0f3610080c45") : configmap "kube-rbac-proxy" not found Jul 22 01:37:56.464861 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:37:56.464578 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a297949-cd40-49e4-8643-0f3610080c45-insights-runtime-extractor-tls podName:1a297949-cd40-49e4-8643-0f3610080c45 nodeName:}" failed. No retries permitted until 2026-07-22 01:38:04.464572246 +0000 UTC m=+55.057376586 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a297949-cd40-49e4-8643-0f3610080c45-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vmk5j" (UID: "1a297949-cd40-49e4-8643-0f3610080c45") : secret "insights-runtime-extractor-tls" not found Jul 22 01:38:00.193601 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:00.193564 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8510529c-89e5-48b1-bbc0-3f3ccd912896-metrics-tls\") pod \"dns-default-drsdb\" (UID: \"8510529c-89e5-48b1-bbc0-3f3ccd912896\") " pod="openshift-dns/dns-default-drsdb" Jul 22 01:38:00.195937 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:00.195919 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8510529c-89e5-48b1-bbc0-3f3ccd912896-metrics-tls\") pod \"dns-default-drsdb\" (UID: \"8510529c-89e5-48b1-bbc0-3f3ccd912896\") " pod="openshift-dns/dns-default-drsdb" Jul 22 01:38:00.242061 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:00.242041 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-drsdb" Jul 22 01:38:00.381149 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:00.381120 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-drsdb"] Jul 22 01:38:00.384456 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:38:00.384429 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8510529c_89e5_48b1_bbc0_3f3ccd912896.slice/crio-57338c12861098e0fdd40332933a8fe5ff4ba9a093beed6b0d029830e5584541 WatchSource:0}: Error finding container 57338c12861098e0fdd40332933a8fe5ff4ba9a093beed6b0d029830e5584541: Status 404 returned error can't find the container with id 57338c12861098e0fdd40332933a8fe5ff4ba9a093beed6b0d029830e5584541 Jul 22 01:38:01.126976 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:01.126940 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-drsdb" event={"ID":"8510529c-89e5-48b1-bbc0-3f3ccd912896","Type":"ContainerStarted","Data":"57338c12861098e0fdd40332933a8fe5ff4ba9a093beed6b0d029830e5584541"} Jul 22 01:38:04.135326 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:04.135288 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-drsdb" event={"ID":"8510529c-89e5-48b1-bbc0-3f3ccd912896","Type":"ContainerStarted","Data":"22d918a46196c72cea85bb202e01c69e10330c2988f564a05cb25bc516f7a64f"} Jul 22 01:38:04.135718 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:04.135349 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-drsdb" event={"ID":"8510529c-89e5-48b1-bbc0-3f3ccd912896","Type":"ContainerStarted","Data":"38e807b3a542487e8098fef94190a0d82ec2c7b9e840e31a26d2834813785d23"} Jul 22 01:38:04.135718 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:04.135410 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-drsdb" Jul 22 01:38:04.158186 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:04.158135 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-drsdb" podStartSLOduration=17.120473807 podStartE2EDuration="20.158120041s" totalImagesPullingTime="3.037646234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:37:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:38:04.157423116 +0000 UTC m=+54.750227478" watchObservedRunningTime="2026-07-22 01:38:04.158120041 +0000 UTC m=+54.750924404" Jul 22 01:38:04.524708 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:04.524668 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a297949-cd40-49e4-8643-0f3610080c45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmk5j\" (UID: \"1a297949-cd40-49e4-8643-0f3610080c45\") " pod="openshift-insights/insights-runtime-extractor-vmk5j" Jul 22 01:38:04.524872 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:04.524765 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a297949-cd40-49e4-8643-0f3610080c45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vmk5j\" (UID: \"1a297949-cd40-49e4-8643-0f3610080c45\") " pod="openshift-insights/insights-runtime-extractor-vmk5j" Jul 22 01:38:04.524872 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:38:04.524812 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 01:38:04.524959 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:38:04.524876 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a297949-cd40-49e4-8643-0f3610080c45-insights-runtime-extractor-tls podName:1a297949-cd40-49e4-8643-0f3610080c45 nodeName:}" failed. No retries permitted until 2026-07-22 01:38:20.524861237 +0000 UTC m=+71.117665580 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a297949-cd40-49e4-8643-0f3610080c45-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vmk5j" (UID: "1a297949-cd40-49e4-8643-0f3610080c45") : secret "insights-runtime-extractor-tls" not found Jul 22 01:38:04.525228 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:04.525211 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a297949-cd40-49e4-8643-0f3610080c45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vmk5j\" (UID: \"1a297949-cd40-49e4-8643-0f3610080c45\") " pod="openshift-insights/insights-runtime-extractor-vmk5j" Jul 22 01:38:09.068365 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:09.068337 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lcjb6" Jul 22 01:38:13.999424 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:13.999392 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769697667b-9wz4t"] Jul 22 01:38:14.002049 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.002033 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769697667b-9wz4t" Jul 22 01:38:14.005728 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.005687 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 22 01:38:14.005728 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.005719 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 22 01:38:14.005897 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.005685 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-62xgt\"" Jul 22 01:38:14.005897 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.005689 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 22 01:38:14.005897 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.005712 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 22 01:38:14.013200 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.013175 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769697667b-9wz4t"] Jul 22 01:38:14.086844 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.086820 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67d86b484b-dtqd8"] Jul 22 01:38:14.088744 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.088729 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-r6jqj"] Jul 22 01:38:14.088854 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.088839 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" Jul 22 01:38:14.090755 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.090737 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-r6jqj" Jul 22 01:38:14.090966 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.090944 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fnkgz\"" Jul 22 01:38:14.091101 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.091079 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 22 01:38:14.091252 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.091235 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 22 01:38:14.091307 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.091294 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 22 01:38:14.093737 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.093713 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db949\" (UniqueName: \"kubernetes.io/projected/3ee3e659-6e9c-4474-97a4-82946834605a-kube-api-access-db949\") pod \"managed-serviceaccount-addon-agent-769697667b-9wz4t\" (UID: \"3ee3e659-6e9c-4474-97a4-82946834605a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769697667b-9wz4t" Jul 22 01:38:14.093835 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.093770 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 22 01:38:14.093835 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.093791 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-gwnlc\"" Jul 22 01:38:14.093835 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.093794 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ee3e659-6e9c-4474-97a4-82946834605a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-769697667b-9wz4t\" (UID: \"3ee3e659-6e9c-4474-97a4-82946834605a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769697667b-9wz4t" Jul 22 01:38:14.096042 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.096018 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 22 01:38:14.097246 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.097227 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 22 01:38:14.105445 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.105405 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-r6jqj"] Jul 22 01:38:14.109838 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.109818 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67d86b484b-dtqd8"] Jul 22 01:38:14.140221 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.140203 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-drsdb" Jul 22 01:38:14.194589 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.194561 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac01ce0a-32b3-458f-b280-b170447dcc21-bound-sa-token\") pod \"image-registry-67d86b484b-dtqd8\" (UID: \"ac01ce0a-32b3-458f-b280-b170447dcc21\") " pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" Jul 22 01:38:14.194722 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.194613 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ee3e659-6e9c-4474-97a4-82946834605a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-769697667b-9wz4t\" (UID: \"3ee3e659-6e9c-4474-97a4-82946834605a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769697667b-9wz4t" Jul 22 01:38:14.194860 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.194837 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac01ce0a-32b3-458f-b280-b170447dcc21-image-registry-private-configuration\") pod \"image-registry-67d86b484b-dtqd8\" (UID: \"ac01ce0a-32b3-458f-b280-b170447dcc21\") " pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" Jul 22 01:38:14.194924 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.194884 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db949\" (UniqueName: \"kubernetes.io/projected/3ee3e659-6e9c-4474-97a4-82946834605a-kube-api-access-db949\") pod \"managed-serviceaccount-addon-agent-769697667b-9wz4t\" (UID: \"3ee3e659-6e9c-4474-97a4-82946834605a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769697667b-9wz4t" Jul 22 01:38:14.194924 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.194916 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac01ce0a-32b3-458f-b280-b170447dcc21-registry-tls\") pod \"image-registry-67d86b484b-dtqd8\" (UID: \"ac01ce0a-32b3-458f-b280-b170447dcc21\") " pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" Jul 22 01:38:14.195025 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.194947 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l29d9\" (UniqueName: \"kubernetes.io/projected/cd37a5f0-67e8-4a2c-ade0-528e281a7a3e-kube-api-access-l29d9\") pod \"downloads-5bc67dcd46-r6jqj\" (UID: \"cd37a5f0-67e8-4a2c-ade0-528e281a7a3e\") " pod="openshift-console/downloads-5bc67dcd46-r6jqj" Jul 22 01:38:14.195541 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.195516 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac01ce0a-32b3-458f-b280-b170447dcc21-trusted-ca\") pod \"image-registry-67d86b484b-dtqd8\" (UID: \"ac01ce0a-32b3-458f-b280-b170447dcc21\") " pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" Jul 22 01:38:14.195633 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.195570 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhvn9\" (UniqueName: \"kubernetes.io/projected/ac01ce0a-32b3-458f-b280-b170447dcc21-kube-api-access-bhvn9\") pod \"image-registry-67d86b484b-dtqd8\" (UID: \"ac01ce0a-32b3-458f-b280-b170447dcc21\") " pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" Jul 22 01:38:14.195685 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.195659 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac01ce0a-32b3-458f-b280-b170447dcc21-registry-certificates\") pod \"image-registry-67d86b484b-dtqd8\" (UID: \"ac01ce0a-32b3-458f-b280-b170447dcc21\") " pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" Jul 22 01:38:14.195787 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.195763 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac01ce0a-32b3-458f-b280-b170447dcc21-ca-trust-extracted\") pod \"image-registry-67d86b484b-dtqd8\" (UID: \"ac01ce0a-32b3-458f-b280-b170447dcc21\") " pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" Jul 22 01:38:14.195846 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.195812 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac01ce0a-32b3-458f-b280-b170447dcc21-installation-pull-secrets\") pod \"image-registry-67d86b484b-dtqd8\" (UID: \"ac01ce0a-32b3-458f-b280-b170447dcc21\") " pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" Jul 22 01:38:14.201007 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.200983 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ee3e659-6e9c-4474-97a4-82946834605a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-769697667b-9wz4t\" (UID: \"3ee3e659-6e9c-4474-97a4-82946834605a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769697667b-9wz4t" Jul 22 01:38:14.208232 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.208207 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db949\" (UniqueName: \"kubernetes.io/projected/3ee3e659-6e9c-4474-97a4-82946834605a-kube-api-access-db949\") pod \"managed-serviceaccount-addon-agent-769697667b-9wz4t\" (UID: \"3ee3e659-6e9c-4474-97a4-82946834605a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769697667b-9wz4t" Jul 22 01:38:14.297113 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.297050 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac01ce0a-32b3-458f-b280-b170447dcc21-image-registry-private-configuration\") pod \"image-registry-67d86b484b-dtqd8\" (UID: \"ac01ce0a-32b3-458f-b280-b170447dcc21\") " pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" Jul 22 01:38:14.297113 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.297091 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac01ce0a-32b3-458f-b280-b170447dcc21-registry-tls\") pod \"image-registry-67d86b484b-dtqd8\" (UID: \"ac01ce0a-32b3-458f-b280-b170447dcc21\") " pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" Jul 22 01:38:14.297113 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.297108 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l29d9\" (UniqueName: \"kubernetes.io/projected/cd37a5f0-67e8-4a2c-ade0-528e281a7a3e-kube-api-access-l29d9\") pod \"downloads-5bc67dcd46-r6jqj\" (UID: \"cd37a5f0-67e8-4a2c-ade0-528e281a7a3e\") " pod="openshift-console/downloads-5bc67dcd46-r6jqj" Jul 22 01:38:14.297325 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.297127 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac01ce0a-32b3-458f-b280-b170447dcc21-trusted-ca\") pod \"image-registry-67d86b484b-dtqd8\" (UID: \"ac01ce0a-32b3-458f-b280-b170447dcc21\") " pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" Jul 22 01:38:14.297325 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.297142 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhvn9\" (UniqueName: \"kubernetes.io/projected/ac01ce0a-32b3-458f-b280-b170447dcc21-kube-api-access-bhvn9\") pod \"image-registry-67d86b484b-dtqd8\" (UID: \"ac01ce0a-32b3-458f-b280-b170447dcc21\") " pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" Jul 22 01:38:14.297325 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.297165 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac01ce0a-32b3-458f-b280-b170447dcc21-registry-certificates\") pod \"image-registry-67d86b484b-dtqd8\" (UID: \"ac01ce0a-32b3-458f-b280-b170447dcc21\") " pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" Jul 22 01:38:14.297456 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.297383 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac01ce0a-32b3-458f-b280-b170447dcc21-ca-trust-extracted\") pod \"image-registry-67d86b484b-dtqd8\" (UID: \"ac01ce0a-32b3-458f-b280-b170447dcc21\") " pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" Jul 22 01:38:14.297456 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.297446 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac01ce0a-32b3-458f-b280-b170447dcc21-installation-pull-secrets\") pod \"image-registry-67d86b484b-dtqd8\" (UID: \"ac01ce0a-32b3-458f-b280-b170447dcc21\") " pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" Jul 22 01:38:14.297618 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.297512 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac01ce0a-32b3-458f-b280-b170447dcc21-bound-sa-token\") pod \"image-registry-67d86b484b-dtqd8\" (UID: \"ac01ce0a-32b3-458f-b280-b170447dcc21\") " pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" Jul 22 01:38:14.297818 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.297794 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac01ce0a-32b3-458f-b280-b170447dcc21-ca-trust-extracted\") pod \"image-registry-67d86b484b-dtqd8\" (UID: \"ac01ce0a-32b3-458f-b280-b170447dcc21\") " pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" Jul 22 01:38:14.298088 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.298066 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac01ce0a-32b3-458f-b280-b170447dcc21-registry-certificates\") pod \"image-registry-67d86b484b-dtqd8\" (UID: \"ac01ce0a-32b3-458f-b280-b170447dcc21\") " pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" Jul 22 01:38:14.298318 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.298300 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac01ce0a-32b3-458f-b280-b170447dcc21-trusted-ca\") pod \"image-registry-67d86b484b-dtqd8\" (UID: \"ac01ce0a-32b3-458f-b280-b170447dcc21\") " pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" Jul 22 01:38:14.299635 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.299617 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac01ce0a-32b3-458f-b280-b170447dcc21-image-registry-private-configuration\") pod \"image-registry-67d86b484b-dtqd8\" (UID: \"ac01ce0a-32b3-458f-b280-b170447dcc21\") " pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" Jul 22 01:38:14.300279 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.300254 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac01ce0a-32b3-458f-b280-b170447dcc21-installation-pull-secrets\") pod \"image-registry-67d86b484b-dtqd8\" (UID: \"ac01ce0a-32b3-458f-b280-b170447dcc21\") " pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" Jul 22 01:38:14.300403 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.300389 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac01ce0a-32b3-458f-b280-b170447dcc21-registry-tls\") pod \"image-registry-67d86b484b-dtqd8\" (UID: \"ac01ce0a-32b3-458f-b280-b170447dcc21\") " pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" Jul 22 01:38:14.313432 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.313404 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhvn9\" (UniqueName: \"kubernetes.io/projected/ac01ce0a-32b3-458f-b280-b170447dcc21-kube-api-access-bhvn9\") pod \"image-registry-67d86b484b-dtqd8\" (UID: \"ac01ce0a-32b3-458f-b280-b170447dcc21\") " pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" Jul 22 01:38:14.313512 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.313501 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac01ce0a-32b3-458f-b280-b170447dcc21-bound-sa-token\") pod \"image-registry-67d86b484b-dtqd8\" (UID: \"ac01ce0a-32b3-458f-b280-b170447dcc21\") " pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" Jul 22 01:38:14.314225 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.314202 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l29d9\" (UniqueName: \"kubernetes.io/projected/cd37a5f0-67e8-4a2c-ade0-528e281a7a3e-kube-api-access-l29d9\") pod \"downloads-5bc67dcd46-r6jqj\" (UID: \"cd37a5f0-67e8-4a2c-ade0-528e281a7a3e\") " pod="openshift-console/downloads-5bc67dcd46-r6jqj" Jul 22 01:38:14.321003 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.320987 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769697667b-9wz4t" Jul 22 01:38:14.399209 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.399168 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" Jul 22 01:38:14.404393 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.404368 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-r6jqj" Jul 22 01:38:14.450833 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.450753 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769697667b-9wz4t"] Jul 22 01:38:14.455359 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:38:14.455326 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ee3e659_6e9c_4474_97a4_82946834605a.slice/crio-bda5af4a60d9ac8e19791b21391099b800236ea109242c972f30b0e2b5aabd7a WatchSource:0}: Error finding container bda5af4a60d9ac8e19791b21391099b800236ea109242c972f30b0e2b5aabd7a: Status 404 returned error can't find the container with id bda5af4a60d9ac8e19791b21391099b800236ea109242c972f30b0e2b5aabd7a Jul 22 01:38:14.534237 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.534205 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67d86b484b-dtqd8"] Jul 22 01:38:14.537102 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:38:14.537073 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac01ce0a_32b3_458f_b280_b170447dcc21.slice/crio-757fe9ee5ab6833ce21b3771f7dd202e63f0da21c3d86b91a463f4b3abc476ab WatchSource:0}: Error finding container 757fe9ee5ab6833ce21b3771f7dd202e63f0da21c3d86b91a463f4b3abc476ab: Status 404 returned error can't find the container with id 757fe9ee5ab6833ce21b3771f7dd202e63f0da21c3d86b91a463f4b3abc476ab Jul 22 01:38:14.550494 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.550441 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-r6jqj"] Jul 22 01:38:14.552984 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:38:14.552959 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd37a5f0_67e8_4a2c_ade0_528e281a7a3e.slice/crio-7d7c9a64e31a8536c66130e5693cf138f2c1b4d8171157cc188559ac01898cdc WatchSource:0}: Error finding container 7d7c9a64e31a8536c66130e5693cf138f2c1b4d8171157cc188559ac01898cdc: Status 404 returned error can't find the container with id 7d7c9a64e31a8536c66130e5693cf138f2c1b4d8171157cc188559ac01898cdc Jul 22 01:38:14.600027 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.600001 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f18aa922-9c77-4466-aad7-6e2f5fe12c59-metrics-certs\") pod \"network-metrics-daemon-j4s4b\" (UID: \"f18aa922-9c77-4466-aad7-6e2f5fe12c59\") " pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:38:14.600132 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.600063 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/448e76a1-df97-4a26-8433-7a51cdeb4a5a-original-pull-secret\") pod \"global-pull-secret-syncer-ghl8b\" (UID: \"448e76a1-df97-4a26-8433-7a51cdeb4a5a\") " pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:38:14.602187 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.602168 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 01:38:14.602278 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.602237 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 01:38:14.613174 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.613153 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/448e76a1-df97-4a26-8433-7a51cdeb4a5a-original-pull-secret\") pod \"global-pull-secret-syncer-ghl8b\" (UID: \"448e76a1-df97-4a26-8433-7a51cdeb4a5a\") " pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:38:14.613261 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.613179 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f18aa922-9c77-4466-aad7-6e2f5fe12c59-metrics-certs\") pod \"network-metrics-daemon-j4s4b\" (UID: \"f18aa922-9c77-4466-aad7-6e2f5fe12c59\") " pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:38:14.701099 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.701074 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jglhg\" (UniqueName: \"kubernetes.io/projected/271b4b39-8156-44e0-a9fb-c937501425fa-kube-api-access-jglhg\") pod \"network-check-target-fxfmm\" (UID: \"271b4b39-8156-44e0-a9fb-c937501425fa\") " pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:38:14.703328 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.703311 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 01:38:14.714333 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.714314 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 01:38:14.725604 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.725582 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jglhg\" (UniqueName: \"kubernetes.io/projected/271b4b39-8156-44e0-a9fb-c937501425fa-kube-api-access-jglhg\") pod \"network-check-target-fxfmm\" (UID: \"271b4b39-8156-44e0-a9fb-c937501425fa\") " pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:38:14.730510 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.730488 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tqzqp\"" Jul 22 01:38:14.736959 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.736942 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fbrqb\"" Jul 22 01:38:14.738987 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.738971 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:38:14.739063 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.739011 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ghl8b" Jul 22 01:38:14.743817 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.743800 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j4s4b" Jul 22 01:38:14.748092 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.748070 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74d9dc645c-6gn9v"] Jul 22 01:38:14.750159 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.750136 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74d9dc645c-6gn9v" Jul 22 01:38:14.752486 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.752467 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 22 01:38:14.753155 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.752645 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 22 01:38:14.753266 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.752784 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 22 01:38:14.753350 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.752799 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 22 01:38:14.753403 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.752873 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 22 01:38:14.753450 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.752970 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kl75w\"" Jul 22 01:38:14.762924 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.762889 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74d9dc645c-6gn9v"] Jul 22 01:38:14.802387 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.801862 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-console-serving-cert\") pod \"console-74d9dc645c-6gn9v\" (UID: \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\") " pod="openshift-console/console-74d9dc645c-6gn9v" Jul 22 01:38:14.802387 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.801955 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9thw\" (UniqueName: \"kubernetes.io/projected/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-kube-api-access-k9thw\") pod \"console-74d9dc645c-6gn9v\" (UID: \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\") " pod="openshift-console/console-74d9dc645c-6gn9v" Jul 22 01:38:14.802387 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.801976 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-oauth-serving-cert\") pod \"console-74d9dc645c-6gn9v\" (UID: \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\") " pod="openshift-console/console-74d9dc645c-6gn9v" Jul 22 01:38:14.802387 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.802016 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-console-oauth-config\") pod \"console-74d9dc645c-6gn9v\" (UID: \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\") " pod="openshift-console/console-74d9dc645c-6gn9v" Jul 22 01:38:14.802387 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.802035 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-console-config\") pod \"console-74d9dc645c-6gn9v\" (UID: \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\") " pod="openshift-console/console-74d9dc645c-6gn9v" Jul 22 01:38:14.802387 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.802096 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-service-ca\") pod \"console-74d9dc645c-6gn9v\" (UID: \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\") " pod="openshift-console/console-74d9dc645c-6gn9v" Jul 22 01:38:14.900130 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.900087 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fxfmm"] Jul 22 01:38:14.903364 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.903339 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-console-serving-cert\") pod \"console-74d9dc645c-6gn9v\" (UID: \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\") " pod="openshift-console/console-74d9dc645c-6gn9v" Jul 22 01:38:14.903414 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.903386 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9thw\" (UniqueName: \"kubernetes.io/projected/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-kube-api-access-k9thw\") pod \"console-74d9dc645c-6gn9v\" (UID: \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\") " pod="openshift-console/console-74d9dc645c-6gn9v" Jul 22 01:38:14.903462 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.903419 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-oauth-serving-cert\") pod \"console-74d9dc645c-6gn9v\" (UID: \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\") " pod="openshift-console/console-74d9dc645c-6gn9v" Jul 22 01:38:14.903587 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.903563 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-console-oauth-config\") pod \"console-74d9dc645c-6gn9v\" (UID: \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\") " pod="openshift-console/console-74d9dc645c-6gn9v" Jul 22 01:38:14.903687 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.903612 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-console-config\") pod \"console-74d9dc645c-6gn9v\" (UID: \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\") " pod="openshift-console/console-74d9dc645c-6gn9v" Jul 22 01:38:14.903764 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.903742 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-service-ca\") pod \"console-74d9dc645c-6gn9v\" (UID: \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\") " pod="openshift-console/console-74d9dc645c-6gn9v" Jul 22 01:38:14.904483 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.904450 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-console-config\") pod \"console-74d9dc645c-6gn9v\" (UID: \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\") " pod="openshift-console/console-74d9dc645c-6gn9v" Jul 22 01:38:14.904607 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.904501 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-oauth-serving-cert\") pod \"console-74d9dc645c-6gn9v\" (UID: \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\") " pod="openshift-console/console-74d9dc645c-6gn9v" Jul 22 01:38:14.904745 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.904676 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-service-ca\") pod \"console-74d9dc645c-6gn9v\" (UID: \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\") " pod="openshift-console/console-74d9dc645c-6gn9v" Jul 22 01:38:14.905097 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:38:14.905066 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271b4b39_8156_44e0_a9fb_c937501425fa.slice/crio-6d871c81ee54981432c91c27172bc48d74e5748ea482f40a4ca26c049a637d96 WatchSource:0}: Error finding container 6d871c81ee54981432c91c27172bc48d74e5748ea482f40a4ca26c049a637d96: Status 404 returned error can't find the container with id 6d871c81ee54981432c91c27172bc48d74e5748ea482f40a4ca26c049a637d96 Jul 22 01:38:14.906800 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.906755 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-console-oauth-config\") pod \"console-74d9dc645c-6gn9v\" (UID: \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\") " pod="openshift-console/console-74d9dc645c-6gn9v" Jul 22 01:38:14.907364 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.907334 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-console-serving-cert\") pod \"console-74d9dc645c-6gn9v\" (UID: \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\") " pod="openshift-console/console-74d9dc645c-6gn9v" Jul 22 01:38:14.907654 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.907636 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-j4s4b"] Jul 22 01:38:14.910951 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:38:14.910930 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf18aa922_9c77_4466_aad7_6e2f5fe12c59.slice/crio-40d55dd19c375ea6ee1b1b4b2c37fd5668f3db02f03a93b61c2e93141ba9bc64 WatchSource:0}: Error finding container 40d55dd19c375ea6ee1b1b4b2c37fd5668f3db02f03a93b61c2e93141ba9bc64: Status 404 returned error can't find the container with id 40d55dd19c375ea6ee1b1b4b2c37fd5668f3db02f03a93b61c2e93141ba9bc64 Jul 22 01:38:14.915021 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.915001 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9thw\" (UniqueName: \"kubernetes.io/projected/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-kube-api-access-k9thw\") pod \"console-74d9dc645c-6gn9v\" (UID: \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\") " pod="openshift-console/console-74d9dc645c-6gn9v" Jul 22 01:38:14.933132 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:14.933110 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ghl8b"] Jul 22 01:38:14.935851 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:38:14.935829 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod448e76a1_df97_4a26_8433_7a51cdeb4a5a.slice/crio-d14d608f39eac2af3bea5987fc1dbf170c6e40d37b47abca140b95c101f33819 WatchSource:0}: Error finding container d14d608f39eac2af3bea5987fc1dbf170c6e40d37b47abca140b95c101f33819: Status 404 returned error can't find the container with id d14d608f39eac2af3bea5987fc1dbf170c6e40d37b47abca140b95c101f33819 Jul 22 01:38:15.064062 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:15.063984 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74d9dc645c-6gn9v" Jul 22 01:38:15.163547 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:15.163509 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" event={"ID":"ac01ce0a-32b3-458f-b280-b170447dcc21","Type":"ContainerStarted","Data":"b13ce9da84dd1962f82ea7e77f66d2a58a7fe1c4e082844f22d3614e1a829944"} Jul 22 01:38:15.163766 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:15.163566 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" event={"ID":"ac01ce0a-32b3-458f-b280-b170447dcc21","Type":"ContainerStarted","Data":"757fe9ee5ab6833ce21b3771f7dd202e63f0da21c3d86b91a463f4b3abc476ab"} Jul 22 01:38:15.163766 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:15.163594 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" Jul 22 01:38:15.164827 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:15.164801 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j4s4b" event={"ID":"f18aa922-9c77-4466-aad7-6e2f5fe12c59","Type":"ContainerStarted","Data":"40d55dd19c375ea6ee1b1b4b2c37fd5668f3db02f03a93b61c2e93141ba9bc64"} Jul 22 01:38:15.166024 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:15.166000 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ghl8b" event={"ID":"448e76a1-df97-4a26-8433-7a51cdeb4a5a","Type":"ContainerStarted","Data":"d14d608f39eac2af3bea5987fc1dbf170c6e40d37b47abca140b95c101f33819"} Jul 22 01:38:15.167147 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:15.167123 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769697667b-9wz4t" event={"ID":"3ee3e659-6e9c-4474-97a4-82946834605a","Type":"ContainerStarted","Data":"bda5af4a60d9ac8e19791b21391099b800236ea109242c972f30b0e2b5aabd7a"} Jul 22 01:38:15.168341 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:15.168315 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fxfmm" event={"ID":"271b4b39-8156-44e0-a9fb-c937501425fa","Type":"ContainerStarted","Data":"6d871c81ee54981432c91c27172bc48d74e5748ea482f40a4ca26c049a637d96"} Jul 22 01:38:15.169517 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:15.169496 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-r6jqj" event={"ID":"cd37a5f0-67e8-4a2c-ade0-528e281a7a3e","Type":"ContainerStarted","Data":"7d7c9a64e31a8536c66130e5693cf138f2c1b4d8171157cc188559ac01898cdc"} Jul 22 01:38:15.190441 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:15.190404 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" podStartSLOduration=1.1903914 podStartE2EDuration="1.1903914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:38:15.189510934 +0000 UTC m=+65.782315288" watchObservedRunningTime="2026-07-22 01:38:15.1903914 +0000 UTC m=+65.783195742" Jul 22 01:38:15.193821 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:15.193802 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74d9dc645c-6gn9v"] Jul 22 01:38:15.199604 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:38:15.199581 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a3bbe67_f37d_4d0e_bea4_9f0862d6231a.slice/crio-a53232af3c6e0a7ba292d773dd9e289981a09bf2837d98bdb6e1a0d1d7a25005 WatchSource:0}: Error finding container a53232af3c6e0a7ba292d773dd9e289981a09bf2837d98bdb6e1a0d1d7a25005: Status 404 returned error can't find the container with id a53232af3c6e0a7ba292d773dd9e289981a09bf2837d98bdb6e1a0d1d7a25005 Jul 22 01:38:15.330715 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:15.330629 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-thj9m"] Jul 22 01:38:15.333166 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:15.333146 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-thj9m" Jul 22 01:38:15.335482 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:15.335458 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 22 01:38:15.335726 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:15.335684 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-72w9v\"" Jul 22 01:38:15.345643 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:15.345618 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-thj9m"] Jul 22 01:38:15.407927 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:15.407890 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2c9c232c-a93d-47aa-88a2-90fa05b1bc47-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-thj9m\" (UID: \"2c9c232c-a93d-47aa-88a2-90fa05b1bc47\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-thj9m" Jul 22 01:38:15.509431 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:15.509388 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2c9c232c-a93d-47aa-88a2-90fa05b1bc47-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-thj9m\" (UID: \"2c9c232c-a93d-47aa-88a2-90fa05b1bc47\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-thj9m" Jul 22 01:38:15.509613 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:38:15.509524 2561 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 22 01:38:15.509613 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:38:15.509597 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c9c232c-a93d-47aa-88a2-90fa05b1bc47-tls-certificates podName:2c9c232c-a93d-47aa-88a2-90fa05b1bc47 nodeName:}" failed. No retries permitted until 2026-07-22 01:38:16.009576872 +0000 UTC m=+66.602381226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/2c9c232c-a93d-47aa-88a2-90fa05b1bc47-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-thj9m" (UID: "2c9c232c-a93d-47aa-88a2-90fa05b1bc47") : secret "prometheus-operator-admission-webhook-tls" not found Jul 22 01:38:16.013499 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:16.013456 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2c9c232c-a93d-47aa-88a2-90fa05b1bc47-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-thj9m\" (UID: \"2c9c232c-a93d-47aa-88a2-90fa05b1bc47\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-thj9m" Jul 22 01:38:16.025909 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:16.022368 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2c9c232c-a93d-47aa-88a2-90fa05b1bc47-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-thj9m\" (UID: \"2c9c232c-a93d-47aa-88a2-90fa05b1bc47\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-thj9m" Jul 22 01:38:16.189984 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:16.189948 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74d9dc645c-6gn9v" event={"ID":"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a","Type":"ContainerStarted","Data":"a53232af3c6e0a7ba292d773dd9e289981a09bf2837d98bdb6e1a0d1d7a25005"} Jul 22 01:38:16.246723 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:16.246681 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-thj9m" Jul 22 01:38:16.450214 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:16.450140 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-thj9m"] Jul 22 01:38:16.454274 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:38:16.454232 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c9c232c_a93d_47aa_88a2_90fa05b1bc47.slice/crio-b00b28c1d049746519c59282a13374869606aaa77e21e31b0dc2347a1c982f6c WatchSource:0}: Error finding container b00b28c1d049746519c59282a13374869606aaa77e21e31b0dc2347a1c982f6c: Status 404 returned error can't find the container with id b00b28c1d049746519c59282a13374869606aaa77e21e31b0dc2347a1c982f6c Jul 22 01:38:17.197641 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:17.197571 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j4s4b" event={"ID":"f18aa922-9c77-4466-aad7-6e2f5fe12c59","Type":"ContainerStarted","Data":"71e3f33336f4512ee04b0cd7d20e491c5a2709530b253f62048ae9a8573ed599"} Jul 22 01:38:17.197641 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:17.197643 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j4s4b" event={"ID":"f18aa922-9c77-4466-aad7-6e2f5fe12c59","Type":"ContainerStarted","Data":"0d05fb47b5be2358977205b821082dee2c8772837e7d63f11e0a61c451d296bb"} Jul 22 01:38:17.202629 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:17.202596 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-thj9m" event={"ID":"2c9c232c-a93d-47aa-88a2-90fa05b1bc47","Type":"ContainerStarted","Data":"b00b28c1d049746519c59282a13374869606aaa77e21e31b0dc2347a1c982f6c"} Jul 22 01:38:17.217413 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:17.217366 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-j4s4b" podStartSLOduration=66.010439165 podStartE2EDuration="1m7.21734668s" totalImagesPullingTime="1.206907515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:38:17.2164765 +0000 UTC m=+67.809280887" watchObservedRunningTime="2026-07-22 01:38:17.21734668 +0000 UTC m=+67.810151044" Jul 22 01:38:20.555940 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:20.555897 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a297949-cd40-49e4-8643-0f3610080c45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmk5j\" (UID: \"1a297949-cd40-49e4-8643-0f3610080c45\") " pod="openshift-insights/insights-runtime-extractor-vmk5j" Jul 22 01:38:20.558689 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:20.558664 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a297949-cd40-49e4-8643-0f3610080c45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmk5j\" (UID: \"1a297949-cd40-49e4-8643-0f3610080c45\") " pod="openshift-insights/insights-runtime-extractor-vmk5j" Jul 22 01:38:20.827745 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:20.827496 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l77sq\"" Jul 22 01:38:20.835791 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:20.835765 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vmk5j" Jul 22 01:38:23.796824 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:23.796789 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vmk5j"] Jul 22 01:38:24.235324 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:24.235288 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769697667b-9wz4t" event={"ID":"3ee3e659-6e9c-4474-97a4-82946834605a","Type":"ContainerStarted","Data":"83a0b93244bdbcf99dc04b1b271fd1dec9b93856af869c332667c9eba905144f"} Jul 22 01:38:24.237601 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:24.237547 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74d9dc645c-6gn9v" event={"ID":"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a","Type":"ContainerStarted","Data":"5842bc56f1e478721c49e076f975af79c16d611d28b5b78c4ef40d50dc8a40db"} Jul 22 01:38:24.239950 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:24.239913 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fxfmm" event={"ID":"271b4b39-8156-44e0-a9fb-c937501425fa","Type":"ContainerStarted","Data":"10cb4c62a815ca11d39033e6ceb5273d09ddc38210b54aeb60de442f5e755d68"} Jul 22 01:38:24.240074 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:24.239985 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:38:24.242369 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:24.242048 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-thj9m" event={"ID":"2c9c232c-a93d-47aa-88a2-90fa05b1bc47","Type":"ContainerStarted","Data":"961e1a5d375b953542c55c254232ebe44e12640864969ecc51dafeb273e1db37"} Jul 22 01:38:24.245829 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:24.244031 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-thj9m" Jul 22 01:38:24.245829 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:24.245407 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vmk5j" event={"ID":"1a297949-cd40-49e4-8643-0f3610080c45","Type":"ContainerStarted","Data":"aa10dcfe9407aad8648867c54fc9fbeb1cf37b0345551f79436cb6c898751dee"} Jul 22 01:38:24.251020 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:24.251000 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-thj9m" Jul 22 01:38:24.253214 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:24.253166 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ghl8b" event={"ID":"448e76a1-df97-4a26-8433-7a51cdeb4a5a","Type":"ContainerStarted","Data":"bea8b95bc64a82fa962f2e6600d1fd8b3eb121b38e0363c14b3420ad6bf22303"} Jul 22 01:38:24.253672 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:24.253626 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-769697667b-9wz4t" podStartSLOduration=2.081542405 podStartE2EDuration="11.253609467s" totalImagesPullingTime="9.172067062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:38:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:38:24.252457471 +0000 UTC m=+74.845261834" watchObservedRunningTime="2026-07-22 01:38:24.253609467 +0000 UTC m=+74.846413831" Jul 22 01:38:24.288446 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:24.286617 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-thj9m" podStartSLOduration=2.067952698 podStartE2EDuration="9.286598885s" totalImagesPullingTime="7.218646187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:38:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:38:24.27016208 +0000 UTC m=+74.862966443" watchObservedRunningTime="2026-07-22 01:38:24.286598885 +0000 UTC m=+74.879403248" Jul 22 01:38:24.309531 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:24.308816 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74d9dc645c-6gn9v" podStartSLOduration=1.878983693 podStartE2EDuration="10.308794099s" totalImagesPullingTime="8.429810406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:38:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:38:24.288318283 +0000 UTC m=+74.881122647" watchObservedRunningTime="2026-07-22 01:38:24.308794099 +0000 UTC m=+74.901598464" Jul 22 01:38:24.309531 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:24.309104 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fxfmm" podStartSLOduration=65.584861341 podStartE2EDuration="1m14.309090129s" totalImagesPullingTime="8.724228788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:38:24.307102564 +0000 UTC m=+74.899906927" watchObservedRunningTime="2026-07-22 01:38:24.309090129 +0000 UTC m=+74.901894490" Jul 22 01:38:24.323467 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:24.322672 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ghl8b" podStartSLOduration=65.628847798 podStartE2EDuration="1m14.322654838s" totalImagesPullingTime="8.69380704s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:38:24.321794422 +0000 UTC m=+74.914598806" watchObservedRunningTime="2026-07-22 01:38:24.322654838 +0000 UTC m=+74.915459205" Jul 22 01:38:25.064567 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:25.064457 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74d9dc645c-6gn9v" Jul 22 01:38:25.064567 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:25.064529 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74d9dc645c-6gn9v" Jul 22 01:38:25.070259 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:25.070206 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74d9dc645c-6gn9v" Jul 22 01:38:25.263225 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:25.263197 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74d9dc645c-6gn9v" Jul 22 01:38:34.094959 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.094832 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-ncwrs"] Jul 22 01:38:34.160737 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.160685 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-ncwrs"] Jul 22 01:38:34.160902 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.160758 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tslm7"] Jul 22 01:38:34.160902 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.160821 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ncwrs" Jul 22 01:38:34.165010 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.164987 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 22 01:38:34.165217 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.165196 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 22 01:38:34.165295 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.165267 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-zlwzl\"" Jul 22 01:38:34.165360 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.165200 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 22 01:38:34.165491 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.165474 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 22 01:38:34.166248 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.166231 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 22 01:38:34.169962 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.169942 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.173829 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.173600 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 22 01:38:34.173829 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.173620 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 22 01:38:34.173829 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.173678 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 22 01:38:34.173829 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.173744 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-npxl9\"" Jul 22 01:38:34.269007 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.268974 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e08c73c4-2356-4d3e-add0-16acc7193db7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-ncwrs\" (UID: \"e08c73c4-2356-4d3e-add0-16acc7193db7\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ncwrs" Jul 22 01:38:34.269172 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.269025 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6acfad03-6cc4-4291-a5d4-97256ea86c87-sys\") pod \"node-exporter-tslm7\" (UID: \"6acfad03-6cc4-4291-a5d4-97256ea86c87\") " pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.269172 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.269055 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbggc\" (UniqueName: \"kubernetes.io/projected/6acfad03-6cc4-4291-a5d4-97256ea86c87-kube-api-access-hbggc\") pod \"node-exporter-tslm7\" (UID: \"6acfad03-6cc4-4291-a5d4-97256ea86c87\") " pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.269172 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.269079 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6acfad03-6cc4-4291-a5d4-97256ea86c87-node-exporter-accelerators-collector-config\") pod \"node-exporter-tslm7\" (UID: \"6acfad03-6cc4-4291-a5d4-97256ea86c87\") " pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.269172 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.269097 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e08c73c4-2356-4d3e-add0-16acc7193db7-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-ncwrs\" (UID: \"e08c73c4-2356-4d3e-add0-16acc7193db7\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ncwrs" Jul 22 01:38:34.269172 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.269117 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6acfad03-6cc4-4291-a5d4-97256ea86c87-node-exporter-textfile\") pod \"node-exporter-tslm7\" (UID: \"6acfad03-6cc4-4291-a5d4-97256ea86c87\") " pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.269172 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.269143 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6acfad03-6cc4-4291-a5d4-97256ea86c87-node-exporter-wtmp\") pod \"node-exporter-tslm7\" (UID: \"6acfad03-6cc4-4291-a5d4-97256ea86c87\") " pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.269444 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.269232 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9v2n\" (UniqueName: \"kubernetes.io/projected/e08c73c4-2356-4d3e-add0-16acc7193db7-kube-api-access-s9v2n\") pod \"openshift-state-metrics-79f6958cdf-ncwrs\" (UID: \"e08c73c4-2356-4d3e-add0-16acc7193db7\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ncwrs" Jul 22 01:38:34.269444 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.269273 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e08c73c4-2356-4d3e-add0-16acc7193db7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-ncwrs\" (UID: \"e08c73c4-2356-4d3e-add0-16acc7193db7\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ncwrs" Jul 22 01:38:34.269444 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.269308 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6acfad03-6cc4-4291-a5d4-97256ea86c87-metrics-client-ca\") pod \"node-exporter-tslm7\" (UID: \"6acfad03-6cc4-4291-a5d4-97256ea86c87\") " pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.269444 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.269370 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6acfad03-6cc4-4291-a5d4-97256ea86c87-root\") pod \"node-exporter-tslm7\" (UID: \"6acfad03-6cc4-4291-a5d4-97256ea86c87\") " pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.269444 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.269401 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6acfad03-6cc4-4291-a5d4-97256ea86c87-node-exporter-tls\") pod \"node-exporter-tslm7\" (UID: \"6acfad03-6cc4-4291-a5d4-97256ea86c87\") " pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.269444 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.269427 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6acfad03-6cc4-4291-a5d4-97256ea86c87-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tslm7\" (UID: \"6acfad03-6cc4-4291-a5d4-97256ea86c87\") " pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.310747 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.310710 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74d9dc645c-6gn9v"] Jul 22 01:38:34.370097 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.370021 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6acfad03-6cc4-4291-a5d4-97256ea86c87-sys\") pod \"node-exporter-tslm7\" (UID: \"6acfad03-6cc4-4291-a5d4-97256ea86c87\") " pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.370097 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.370067 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbggc\" (UniqueName: \"kubernetes.io/projected/6acfad03-6cc4-4291-a5d4-97256ea86c87-kube-api-access-hbggc\") pod \"node-exporter-tslm7\" (UID: \"6acfad03-6cc4-4291-a5d4-97256ea86c87\") " pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.370299 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.370098 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6acfad03-6cc4-4291-a5d4-97256ea86c87-node-exporter-accelerators-collector-config\") pod \"node-exporter-tslm7\" (UID: \"6acfad03-6cc4-4291-a5d4-97256ea86c87\") " pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.370299 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.370131 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e08c73c4-2356-4d3e-add0-16acc7193db7-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-ncwrs\" (UID: \"e08c73c4-2356-4d3e-add0-16acc7193db7\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ncwrs" Jul 22 01:38:34.370299 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.370160 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6acfad03-6cc4-4291-a5d4-97256ea86c87-node-exporter-textfile\") pod \"node-exporter-tslm7\" (UID: \"6acfad03-6cc4-4291-a5d4-97256ea86c87\") " pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.370299 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.370185 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6acfad03-6cc4-4291-a5d4-97256ea86c87-node-exporter-wtmp\") pod \"node-exporter-tslm7\" (UID: \"6acfad03-6cc4-4291-a5d4-97256ea86c87\") " pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.370299 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.370218 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9v2n\" (UniqueName: \"kubernetes.io/projected/e08c73c4-2356-4d3e-add0-16acc7193db7-kube-api-access-s9v2n\") pod \"openshift-state-metrics-79f6958cdf-ncwrs\" (UID: \"e08c73c4-2356-4d3e-add0-16acc7193db7\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ncwrs" Jul 22 01:38:34.370299 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.370250 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e08c73c4-2356-4d3e-add0-16acc7193db7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-ncwrs\" (UID: \"e08c73c4-2356-4d3e-add0-16acc7193db7\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ncwrs" Jul 22 01:38:34.370299 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.370280 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6acfad03-6cc4-4291-a5d4-97256ea86c87-metrics-client-ca\") pod \"node-exporter-tslm7\" (UID: \"6acfad03-6cc4-4291-a5d4-97256ea86c87\") " pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.370648 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.370321 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6acfad03-6cc4-4291-a5d4-97256ea86c87-root\") pod \"node-exporter-tslm7\" (UID: \"6acfad03-6cc4-4291-a5d4-97256ea86c87\") " pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.370648 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.370360 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6acfad03-6cc4-4291-a5d4-97256ea86c87-node-exporter-tls\") pod \"node-exporter-tslm7\" (UID: \"6acfad03-6cc4-4291-a5d4-97256ea86c87\") " pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.370648 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.370394 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6acfad03-6cc4-4291-a5d4-97256ea86c87-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tslm7\" (UID: \"6acfad03-6cc4-4291-a5d4-97256ea86c87\") " pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.370648 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.370430 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e08c73c4-2356-4d3e-add0-16acc7193db7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-ncwrs\" (UID: \"e08c73c4-2356-4d3e-add0-16acc7193db7\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ncwrs" Jul 22 01:38:34.371055 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.371029 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6acfad03-6cc4-4291-a5d4-97256ea86c87-node-exporter-wtmp\") pod \"node-exporter-tslm7\" (UID: \"6acfad03-6cc4-4291-a5d4-97256ea86c87\") " pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.371227 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:38:34.371212 2561 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 22 01:38:34.371447 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:38:34.371424 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6acfad03-6cc4-4291-a5d4-97256ea86c87-node-exporter-tls podName:6acfad03-6cc4-4291-a5d4-97256ea86c87 nodeName:}" failed. No retries permitted until 2026-07-22 01:38:34.871400583 +0000 UTC m=+85.464204936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/6acfad03-6cc4-4291-a5d4-97256ea86c87-node-exporter-tls") pod "node-exporter-tslm7" (UID: "6acfad03-6cc4-4291-a5d4-97256ea86c87") : secret "node-exporter-tls" not found Jul 22 01:38:34.371625 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.371592 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6acfad03-6cc4-4291-a5d4-97256ea86c87-node-exporter-accelerators-collector-config\") pod \"node-exporter-tslm7\" (UID: \"6acfad03-6cc4-4291-a5d4-97256ea86c87\") " pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.371783 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.371664 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6acfad03-6cc4-4291-a5d4-97256ea86c87-root\") pod \"node-exporter-tslm7\" (UID: \"6acfad03-6cc4-4291-a5d4-97256ea86c87\") " pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.371959 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.371937 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6acfad03-6cc4-4291-a5d4-97256ea86c87-node-exporter-textfile\") pod \"node-exporter-tslm7\" (UID: \"6acfad03-6cc4-4291-a5d4-97256ea86c87\") " pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.372015 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.371954 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e08c73c4-2356-4d3e-add0-16acc7193db7-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-ncwrs\" (UID: \"e08c73c4-2356-4d3e-add0-16acc7193db7\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ncwrs" Jul 22 01:38:34.372072 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.372028 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6acfad03-6cc4-4291-a5d4-97256ea86c87-sys\") pod \"node-exporter-tslm7\" (UID: \"6acfad03-6cc4-4291-a5d4-97256ea86c87\") " pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.372360 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.372338 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6acfad03-6cc4-4291-a5d4-97256ea86c87-metrics-client-ca\") pod \"node-exporter-tslm7\" (UID: \"6acfad03-6cc4-4291-a5d4-97256ea86c87\") " pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.374053 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.374019 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e08c73c4-2356-4d3e-add0-16acc7193db7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-ncwrs\" (UID: \"e08c73c4-2356-4d3e-add0-16acc7193db7\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ncwrs" Jul 22 01:38:34.374164 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.374052 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e08c73c4-2356-4d3e-add0-16acc7193db7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-ncwrs\" (UID: \"e08c73c4-2356-4d3e-add0-16acc7193db7\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ncwrs" Jul 22 01:38:34.374343 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.374323 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6acfad03-6cc4-4291-a5d4-97256ea86c87-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tslm7\" (UID: \"6acfad03-6cc4-4291-a5d4-97256ea86c87\") " pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.382399 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.382374 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9v2n\" (UniqueName: \"kubernetes.io/projected/e08c73c4-2356-4d3e-add0-16acc7193db7-kube-api-access-s9v2n\") pod \"openshift-state-metrics-79f6958cdf-ncwrs\" (UID: \"e08c73c4-2356-4d3e-add0-16acc7193db7\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ncwrs" Jul 22 01:38:34.382842 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.382774 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbggc\" (UniqueName: \"kubernetes.io/projected/6acfad03-6cc4-4291-a5d4-97256ea86c87-kube-api-access-hbggc\") pod \"node-exporter-tslm7\" (UID: \"6acfad03-6cc4-4291-a5d4-97256ea86c87\") " pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.472635 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.472611 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ncwrs" Jul 22 01:38:34.876194 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.876160 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6acfad03-6cc4-4291-a5d4-97256ea86c87-node-exporter-tls\") pod \"node-exporter-tslm7\" (UID: \"6acfad03-6cc4-4291-a5d4-97256ea86c87\") " pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.879161 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.879134 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6acfad03-6cc4-4291-a5d4-97256ea86c87-node-exporter-tls\") pod \"node-exporter-tslm7\" (UID: \"6acfad03-6cc4-4291-a5d4-97256ea86c87\") " pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:34.976820 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:34.976790 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 01:38:35.000318 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.000248 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 01:38:35.000468 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.000374 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.005926 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.005898 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 22 01:38:35.006288 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.006059 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 22 01:38:35.006401 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.006337 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 22 01:38:35.006665 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.006642 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 22 01:38:35.006801 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.006648 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xcbzb\"" Jul 22 01:38:35.008235 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.007224 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 22 01:38:35.008235 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.007251 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 22 01:38:35.008235 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.007462 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 22 01:38:35.008235 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.007483 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 22 01:38:35.012205 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.012183 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 22 01:38:35.077294 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.077260 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6db543ab-b8f6-4f88-a53b-7f37db4799fb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.077461 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.077306 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gx9v\" (UniqueName: \"kubernetes.io/projected/6db543ab-b8f6-4f88-a53b-7f37db4799fb-kube-api-access-5gx9v\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.077461 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.077370 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6db543ab-b8f6-4f88-a53b-7f37db4799fb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.077461 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.077393 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6db543ab-b8f6-4f88-a53b-7f37db4799fb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.077461 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.077417 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6db543ab-b8f6-4f88-a53b-7f37db4799fb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.077461 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.077445 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6db543ab-b8f6-4f88-a53b-7f37db4799fb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.077715 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.077469 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6db543ab-b8f6-4f88-a53b-7f37db4799fb-config-out\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.077715 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.077490 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6db543ab-b8f6-4f88-a53b-7f37db4799fb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.077715 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.077568 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6db543ab-b8f6-4f88-a53b-7f37db4799fb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.077715 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.077606 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6db543ab-b8f6-4f88-a53b-7f37db4799fb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.077715 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.077637 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6db543ab-b8f6-4f88-a53b-7f37db4799fb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.077715 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.077668 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6db543ab-b8f6-4f88-a53b-7f37db4799fb-config-volume\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.077951 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.077718 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6db543ab-b8f6-4f88-a53b-7f37db4799fb-web-config\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.081258 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.081238 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tslm7" Jul 22 01:38:35.178251 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.178219 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6db543ab-b8f6-4f88-a53b-7f37db4799fb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.178724 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.178277 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gx9v\" (UniqueName: \"kubernetes.io/projected/6db543ab-b8f6-4f88-a53b-7f37db4799fb-kube-api-access-5gx9v\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.178724 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.178323 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6db543ab-b8f6-4f88-a53b-7f37db4799fb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.178724 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.178350 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6db543ab-b8f6-4f88-a53b-7f37db4799fb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.178724 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.178377 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6db543ab-b8f6-4f88-a53b-7f37db4799fb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.178724 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.178407 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6db543ab-b8f6-4f88-a53b-7f37db4799fb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.178724 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:38:35.178471 2561 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 22 01:38:35.178724 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:38:35.178555 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6db543ab-b8f6-4f88-a53b-7f37db4799fb-secret-alertmanager-main-tls podName:6db543ab-b8f6-4f88-a53b-7f37db4799fb nodeName:}" failed. No retries permitted until 2026-07-22 01:38:35.678530752 +0000 UTC m=+86.271335105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/6db543ab-b8f6-4f88-a53b-7f37db4799fb-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "6db543ab-b8f6-4f88-a53b-7f37db4799fb") : secret "alertmanager-main-tls" not found Jul 22 01:38:35.178724 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.178668 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6db543ab-b8f6-4f88-a53b-7f37db4799fb-config-out\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.178724 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.178722 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6db543ab-b8f6-4f88-a53b-7f37db4799fb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.179104 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.178761 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6db543ab-b8f6-4f88-a53b-7f37db4799fb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.179104 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.178788 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6db543ab-b8f6-4f88-a53b-7f37db4799fb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.179104 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.178864 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6db543ab-b8f6-4f88-a53b-7f37db4799fb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.179780 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.179360 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6db543ab-b8f6-4f88-a53b-7f37db4799fb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.179780 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.179414 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6db543ab-b8f6-4f88-a53b-7f37db4799fb-config-volume\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.179780 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.179446 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6db543ab-b8f6-4f88-a53b-7f37db4799fb-web-config\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.180217 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.180036 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6db543ab-b8f6-4f88-a53b-7f37db4799fb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.180217 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.180118 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6db543ab-b8f6-4f88-a53b-7f37db4799fb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.181949 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.181908 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6db543ab-b8f6-4f88-a53b-7f37db4799fb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.182667 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.182620 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6db543ab-b8f6-4f88-a53b-7f37db4799fb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.183068 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.183042 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6db543ab-b8f6-4f88-a53b-7f37db4799fb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.183241 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.183215 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6db543ab-b8f6-4f88-a53b-7f37db4799fb-config-out\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.183337 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.183259 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6db543ab-b8f6-4f88-a53b-7f37db4799fb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.183620 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.183603 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6db543ab-b8f6-4f88-a53b-7f37db4799fb-config-volume\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.183768 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.183746 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6db543ab-b8f6-4f88-a53b-7f37db4799fb-web-config\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.184249 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.184206 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6db543ab-b8f6-4f88-a53b-7f37db4799fb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.190190 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.190172 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gx9v\" (UniqueName: \"kubernetes.io/projected/6db543ab-b8f6-4f88-a53b-7f37db4799fb-kube-api-access-5gx9v\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.684153 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.684120 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6db543ab-b8f6-4f88-a53b-7f37db4799fb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:35.684777 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:38:35.684744 2561 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 22 01:38:35.684881 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:38:35.684839 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6db543ab-b8f6-4f88-a53b-7f37db4799fb-secret-alertmanager-main-tls podName:6db543ab-b8f6-4f88-a53b-7f37db4799fb nodeName:}" failed. No retries permitted until 2026-07-22 01:38:36.684815727 +0000 UTC m=+87.277620082 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/6db543ab-b8f6-4f88-a53b-7f37db4799fb-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "6db543ab-b8f6-4f88-a53b-7f37db4799fb") : secret "alertmanager-main-tls" not found Jul 22 01:38:35.851362 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.851323 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6bc54c4b65-8rdms"] Jul 22 01:38:35.868105 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.868074 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" Jul 22 01:38:35.869923 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.869896 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6bc54c4b65-8rdms"] Jul 22 01:38:35.870970 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.870951 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dr0u8a8534s2g\"" Jul 22 01:38:35.871246 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.871229 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 22 01:38:35.871246 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.871241 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 22 01:38:35.871944 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.871422 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 22 01:38:35.871944 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.871471 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 22 01:38:35.871944 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.871655 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-xh4wq\"" Jul 22 01:38:35.872182 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.872163 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 22 01:38:35.986687 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.986604 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n55s\" (UniqueName: \"kubernetes.io/projected/fdbe34c0-975c-4776-863b-5c06b0b5b850-kube-api-access-2n55s\") pod \"thanos-querier-6bc54c4b65-8rdms\" (UID: \"fdbe34c0-975c-4776-863b-5c06b0b5b850\") " pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" Jul 22 01:38:35.986687 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.986657 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fdbe34c0-975c-4776-863b-5c06b0b5b850-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bc54c4b65-8rdms\" (UID: \"fdbe34c0-975c-4776-863b-5c06b0b5b850\") " pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" Jul 22 01:38:35.986687 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.986687 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fdbe34c0-975c-4776-863b-5c06b0b5b850-secret-thanos-querier-tls\") pod \"thanos-querier-6bc54c4b65-8rdms\" (UID: \"fdbe34c0-975c-4776-863b-5c06b0b5b850\") " pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" Jul 22 01:38:35.986980 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.986738 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fdbe34c0-975c-4776-863b-5c06b0b5b850-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bc54c4b65-8rdms\" (UID: \"fdbe34c0-975c-4776-863b-5c06b0b5b850\") " pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" Jul 22 01:38:35.986980 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.986771 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fdbe34c0-975c-4776-863b-5c06b0b5b850-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bc54c4b65-8rdms\" (UID: \"fdbe34c0-975c-4776-863b-5c06b0b5b850\") " pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" Jul 22 01:38:35.986980 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.986842 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fdbe34c0-975c-4776-863b-5c06b0b5b850-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bc54c4b65-8rdms\" (UID: \"fdbe34c0-975c-4776-863b-5c06b0b5b850\") " pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" Jul 22 01:38:35.986980 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.986877 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdbe34c0-975c-4776-863b-5c06b0b5b850-metrics-client-ca\") pod \"thanos-querier-6bc54c4b65-8rdms\" (UID: \"fdbe34c0-975c-4776-863b-5c06b0b5b850\") " pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" Jul 22 01:38:35.986980 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:35.986907 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fdbe34c0-975c-4776-863b-5c06b0b5b850-secret-grpc-tls\") pod \"thanos-querier-6bc54c4b65-8rdms\" (UID: \"fdbe34c0-975c-4776-863b-5c06b0b5b850\") " pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" Jul 22 01:38:36.088204 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:36.088163 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fdbe34c0-975c-4776-863b-5c06b0b5b850-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bc54c4b65-8rdms\" (UID: \"fdbe34c0-975c-4776-863b-5c06b0b5b850\") " pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" Jul 22 01:38:36.088384 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:36.088214 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdbe34c0-975c-4776-863b-5c06b0b5b850-metrics-client-ca\") pod \"thanos-querier-6bc54c4b65-8rdms\" (UID: \"fdbe34c0-975c-4776-863b-5c06b0b5b850\") " pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" Jul 22 01:38:36.088384 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:36.088252 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fdbe34c0-975c-4776-863b-5c06b0b5b850-secret-grpc-tls\") pod \"thanos-querier-6bc54c4b65-8rdms\" (UID: \"fdbe34c0-975c-4776-863b-5c06b0b5b850\") " pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" Jul 22 01:38:36.088384 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:36.088292 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n55s\" (UniqueName: \"kubernetes.io/projected/fdbe34c0-975c-4776-863b-5c06b0b5b850-kube-api-access-2n55s\") pod \"thanos-querier-6bc54c4b65-8rdms\" (UID: \"fdbe34c0-975c-4776-863b-5c06b0b5b850\") " pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" Jul 22 01:38:36.088384 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:36.088331 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fdbe34c0-975c-4776-863b-5c06b0b5b850-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bc54c4b65-8rdms\" (UID: \"fdbe34c0-975c-4776-863b-5c06b0b5b850\") " pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" Jul 22 01:38:36.088384 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:36.088367 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fdbe34c0-975c-4776-863b-5c06b0b5b850-secret-thanos-querier-tls\") pod \"thanos-querier-6bc54c4b65-8rdms\" (UID: \"fdbe34c0-975c-4776-863b-5c06b0b5b850\") " pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" Jul 22 01:38:36.088630 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:36.088403 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fdbe34c0-975c-4776-863b-5c06b0b5b850-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bc54c4b65-8rdms\" (UID: \"fdbe34c0-975c-4776-863b-5c06b0b5b850\") " pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" Jul 22 01:38:36.088630 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:36.088427 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fdbe34c0-975c-4776-863b-5c06b0b5b850-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bc54c4b65-8rdms\" (UID: \"fdbe34c0-975c-4776-863b-5c06b0b5b850\") " pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" Jul 22 01:38:36.090277 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:36.090241 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdbe34c0-975c-4776-863b-5c06b0b5b850-metrics-client-ca\") pod \"thanos-querier-6bc54c4b65-8rdms\" (UID: \"fdbe34c0-975c-4776-863b-5c06b0b5b850\") " pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" Jul 22 01:38:36.091867 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:36.091822 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fdbe34c0-975c-4776-863b-5c06b0b5b850-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bc54c4b65-8rdms\" (UID: \"fdbe34c0-975c-4776-863b-5c06b0b5b850\") " pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" Jul 22 01:38:36.091867 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:36.091822 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fdbe34c0-975c-4776-863b-5c06b0b5b850-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bc54c4b65-8rdms\" (UID: \"fdbe34c0-975c-4776-863b-5c06b0b5b850\") " pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" Jul 22 01:38:36.092156 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:36.092108 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fdbe34c0-975c-4776-863b-5c06b0b5b850-secret-grpc-tls\") pod \"thanos-querier-6bc54c4b65-8rdms\" (UID: \"fdbe34c0-975c-4776-863b-5c06b0b5b850\") " pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" Jul 22 01:38:36.092440 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:36.092419 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fdbe34c0-975c-4776-863b-5c06b0b5b850-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bc54c4b65-8rdms\" (UID: \"fdbe34c0-975c-4776-863b-5c06b0b5b850\") " pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" Jul 22 01:38:36.092529 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:36.092448 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fdbe34c0-975c-4776-863b-5c06b0b5b850-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bc54c4b65-8rdms\" (UID: \"fdbe34c0-975c-4776-863b-5c06b0b5b850\") " pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" Jul 22 01:38:36.092529 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:36.092506 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fdbe34c0-975c-4776-863b-5c06b0b5b850-secret-thanos-querier-tls\") pod \"thanos-querier-6bc54c4b65-8rdms\" (UID: \"fdbe34c0-975c-4776-863b-5c06b0b5b850\") " pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" Jul 22 01:38:36.102747 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:36.102722 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n55s\" (UniqueName: \"kubernetes.io/projected/fdbe34c0-975c-4776-863b-5c06b0b5b850-kube-api-access-2n55s\") pod \"thanos-querier-6bc54c4b65-8rdms\" (UID: \"fdbe34c0-975c-4776-863b-5c06b0b5b850\") " pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" Jul 22 01:38:36.180029 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:36.180001 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" Jul 22 01:38:36.194164 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:36.194145 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67d86b484b-dtqd8" Jul 22 01:38:36.572056 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:38:36.572020 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6acfad03_6cc4_4291_a5d4_97256ea86c87.slice/crio-462fadcc2a42a87a10fb19caad4f9d2846c8468162d29128db823dfff2698284 WatchSource:0}: Error finding container 462fadcc2a42a87a10fb19caad4f9d2846c8468162d29128db823dfff2698284: Status 404 returned error can't find the container with id 462fadcc2a42a87a10fb19caad4f9d2846c8468162d29128db823dfff2698284 Jul 22 01:38:36.694445 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:36.694413 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6db543ab-b8f6-4f88-a53b-7f37db4799fb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:36.697253 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:36.697223 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6db543ab-b8f6-4f88-a53b-7f37db4799fb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6db543ab-b8f6-4f88-a53b-7f37db4799fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:36.699503 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:36.699477 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-ncwrs"] Jul 22 01:38:36.706218 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:38:36.706193 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode08c73c4_2356_4d3e_add0_16acc7193db7.slice/crio-7da61843c248ed2348fb7948d5142a2d7b5239332e6273a2f214f55295e4f0eb WatchSource:0}: Error finding container 7da61843c248ed2348fb7948d5142a2d7b5239332e6273a2f214f55295e4f0eb: Status 404 returned error can't find the container with id 7da61843c248ed2348fb7948d5142a2d7b5239332e6273a2f214f55295e4f0eb Jul 22 01:38:36.730817 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:36.730790 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6bc54c4b65-8rdms"] Jul 22 01:38:36.732430 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:38:36.732408 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdbe34c0_975c_4776_863b_5c06b0b5b850.slice/crio-67a230c65c8f2e0b2afd05ae66007cdded9601ca7757bd26c57c68d5bc426b43 WatchSource:0}: Error finding container 67a230c65c8f2e0b2afd05ae66007cdded9601ca7757bd26c57c68d5bc426b43: Status 404 returned error can't find the container with id 67a230c65c8f2e0b2afd05ae66007cdded9601ca7757bd26c57c68d5bc426b43 Jul 22 01:38:36.812169 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:36.812141 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 01:38:36.976502 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:36.976392 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 01:38:36.980268 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:38:36.980240 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db543ab_b8f6_4f88_a53b_7f37db4799fb.slice/crio-f5c455c7af2aeb463d750010ab3344503387601eea40d350d62f61e86907f927 WatchSource:0}: Error finding container f5c455c7af2aeb463d750010ab3344503387601eea40d350d62f61e86907f927: Status 404 returned error can't find the container with id f5c455c7af2aeb463d750010ab3344503387601eea40d350d62f61e86907f927 Jul 22 01:38:37.307528 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:37.306712 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-r6jqj" event={"ID":"cd37a5f0-67e8-4a2c-ade0-528e281a7a3e","Type":"ContainerStarted","Data":"b19b3dcbede2ef72b545b25b0ae1a38bf40b6e1eabb36a3c8e160cdb3b10e722"} Jul 22 01:38:37.307528 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:37.307481 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-r6jqj" Jul 22 01:38:37.310717 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:37.310632 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vmk5j" event={"ID":"1a297949-cd40-49e4-8643-0f3610080c45","Type":"ContainerStarted","Data":"d4082167b755dd91db906d16865705ce96703ccc5f998c4633bf8d0d35953f30"} Jul 22 01:38:37.317430 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:37.314510 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6db543ab-b8f6-4f88-a53b-7f37db4799fb","Type":"ContainerStarted","Data":"f5c455c7af2aeb463d750010ab3344503387601eea40d350d62f61e86907f927"} Jul 22 01:38:37.317430 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:37.316981 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" event={"ID":"fdbe34c0-975c-4776-863b-5c06b0b5b850","Type":"ContainerStarted","Data":"67a230c65c8f2e0b2afd05ae66007cdded9601ca7757bd26c57c68d5bc426b43"} Jul 22 01:38:37.319182 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:37.319038 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tslm7" event={"ID":"6acfad03-6cc4-4291-a5d4-97256ea86c87","Type":"ContainerStarted","Data":"462fadcc2a42a87a10fb19caad4f9d2846c8468162d29128db823dfff2698284"} Jul 22 01:38:37.319459 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:37.319407 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-r6jqj" Jul 22 01:38:37.322580 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:37.322388 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ncwrs" event={"ID":"e08c73c4-2356-4d3e-add0-16acc7193db7","Type":"ContainerStarted","Data":"94971323ecc08c5e360752cace49f8e0869b252a65e019be32ebec48dfc279f0"} Jul 22 01:38:37.322580 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:37.322441 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ncwrs" event={"ID":"e08c73c4-2356-4d3e-add0-16acc7193db7","Type":"ContainerStarted","Data":"b6d57dc69bae2ac70a28944e8686de2fb9ed609b24ce9db77c714fd0c909b1d3"} Jul 22 01:38:37.322580 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:37.322473 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ncwrs" event={"ID":"e08c73c4-2356-4d3e-add0-16acc7193db7","Type":"ContainerStarted","Data":"7da61843c248ed2348fb7948d5142a2d7b5239332e6273a2f214f55295e4f0eb"} Jul 22 01:38:37.331537 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:37.331482 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-r6jqj" podStartSLOduration=1.315274377 podStartE2EDuration="23.331464194s" totalImagesPullingTime="22.016189817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:38:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:38:37.330561343 +0000 UTC m=+87.923365745" watchObservedRunningTime="2026-07-22 01:38:37.331464194 +0000 UTC m=+87.924268558" Jul 22 01:38:38.474318 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.474252 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7ccd8878ff-lsgj6"] Jul 22 01:38:38.500078 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.500040 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7ccd8878ff-lsgj6"] Jul 22 01:38:38.500262 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.500179 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" Jul 22 01:38:38.505359 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.504968 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kljxf\"" Jul 22 01:38:38.505359 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.505206 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 22 01:38:38.507894 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.507041 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 22 01:38:38.507894 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.507405 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 22 01:38:38.507894 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.507677 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 22 01:38:38.508762 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.508615 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-forf1u0k4vqr9\"" Jul 22 01:38:38.576813 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.576252 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cz8pk"] Jul 22 01:38:38.604889 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.604524 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78fd98559d-c8ghg"] Jul 22 01:38:38.606554 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.606530 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cz8pk" Jul 22 01:38:38.610657 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.610632 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-fwpzw\"" Jul 22 01:38:38.610891 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.610874 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 22 01:38:38.617021 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.616343 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slzpz\" (UniqueName: \"kubernetes.io/projected/35a74ac3-5716-4629-8eb8-cf115f89234b-kube-api-access-slzpz\") pod \"metrics-server-7ccd8878ff-lsgj6\" (UID: \"35a74ac3-5716-4629-8eb8-cf115f89234b\") " pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" Jul 22 01:38:38.617021 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.616386 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35a74ac3-5716-4629-8eb8-cf115f89234b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7ccd8878ff-lsgj6\" (UID: \"35a74ac3-5716-4629-8eb8-cf115f89234b\") " pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" Jul 22 01:38:38.617021 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.616420 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/35a74ac3-5716-4629-8eb8-cf115f89234b-metrics-server-audit-profiles\") pod \"metrics-server-7ccd8878ff-lsgj6\" (UID: \"35a74ac3-5716-4629-8eb8-cf115f89234b\") " pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" Jul 22 01:38:38.617021 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.616456 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/35a74ac3-5716-4629-8eb8-cf115f89234b-secret-metrics-server-client-certs\") pod \"metrics-server-7ccd8878ff-lsgj6\" (UID: \"35a74ac3-5716-4629-8eb8-cf115f89234b\") " pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" Jul 22 01:38:38.617021 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.616500 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/35a74ac3-5716-4629-8eb8-cf115f89234b-audit-log\") pod \"metrics-server-7ccd8878ff-lsgj6\" (UID: \"35a74ac3-5716-4629-8eb8-cf115f89234b\") " pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" Jul 22 01:38:38.617021 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.616536 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/35a74ac3-5716-4629-8eb8-cf115f89234b-secret-metrics-server-tls\") pod \"metrics-server-7ccd8878ff-lsgj6\" (UID: \"35a74ac3-5716-4629-8eb8-cf115f89234b\") " pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" Jul 22 01:38:38.617021 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.616572 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/35a74ac3-5716-4629-8eb8-cf115f89234b-client-ca-bundle\") pod \"metrics-server-7ccd8878ff-lsgj6\" (UID: \"35a74ac3-5716-4629-8eb8-cf115f89234b\") " pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" Jul 22 01:38:38.618766 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.617754 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cz8pk"] Jul 22 01:38:38.618766 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.617793 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78fd98559d-c8ghg"] Jul 22 01:38:38.618766 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.617881 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:38:38.628482 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.628366 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 22 01:38:38.718550 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.717959 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-console-config\") pod \"console-78fd98559d-c8ghg\" (UID: \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\") " pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:38:38.718550 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.718022 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-oauth-serving-cert\") pod \"console-78fd98559d-c8ghg\" (UID: \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\") " pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:38:38.718550 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.718059 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5ac1bc03-0f09-4262-9160-1c19119eb8f4-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-cz8pk\" (UID: \"5ac1bc03-0f09-4262-9160-1c19119eb8f4\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cz8pk" Jul 22 01:38:38.718550 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.718092 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slzpz\" (UniqueName: \"kubernetes.io/projected/35a74ac3-5716-4629-8eb8-cf115f89234b-kube-api-access-slzpz\") pod \"metrics-server-7ccd8878ff-lsgj6\" (UID: \"35a74ac3-5716-4629-8eb8-cf115f89234b\") " pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" Jul 22 01:38:38.718550 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.718121 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35a74ac3-5716-4629-8eb8-cf115f89234b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7ccd8878ff-lsgj6\" (UID: \"35a74ac3-5716-4629-8eb8-cf115f89234b\") " pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" Jul 22 01:38:38.719884 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.719851 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35a74ac3-5716-4629-8eb8-cf115f89234b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7ccd8878ff-lsgj6\" (UID: \"35a74ac3-5716-4629-8eb8-cf115f89234b\") " pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" Jul 22 01:38:38.720008 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.719953 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/35a74ac3-5716-4629-8eb8-cf115f89234b-metrics-server-audit-profiles\") pod \"metrics-server-7ccd8878ff-lsgj6\" (UID: \"35a74ac3-5716-4629-8eb8-cf115f89234b\") " pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" Jul 22 01:38:38.720008 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.719989 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-service-ca\") pod \"console-78fd98559d-c8ghg\" (UID: \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\") " pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:38:38.720117 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.720028 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/35a74ac3-5716-4629-8eb8-cf115f89234b-secret-metrics-server-client-certs\") pod \"metrics-server-7ccd8878ff-lsgj6\" (UID: \"35a74ac3-5716-4629-8eb8-cf115f89234b\") " pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" Jul 22 01:38:38.720117 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.720072 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/35a74ac3-5716-4629-8eb8-cf115f89234b-audit-log\") pod \"metrics-server-7ccd8878ff-lsgj6\" (UID: \"35a74ac3-5716-4629-8eb8-cf115f89234b\") " pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" Jul 22 01:38:38.720117 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.720110 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/35a74ac3-5716-4629-8eb8-cf115f89234b-secret-metrics-server-tls\") pod \"metrics-server-7ccd8878ff-lsgj6\" (UID: \"35a74ac3-5716-4629-8eb8-cf115f89234b\") " pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" Jul 22 01:38:38.720286 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.720135 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-console-serving-cert\") pod \"console-78fd98559d-c8ghg\" (UID: \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\") " pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:38:38.720286 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.720164 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-console-oauth-config\") pod \"console-78fd98559d-c8ghg\" (UID: \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\") " pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:38:38.720286 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.720217 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-trusted-ca-bundle\") pod \"console-78fd98559d-c8ghg\" (UID: \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\") " pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:38:38.720286 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.720259 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/35a74ac3-5716-4629-8eb8-cf115f89234b-client-ca-bundle\") pod \"metrics-server-7ccd8878ff-lsgj6\" (UID: \"35a74ac3-5716-4629-8eb8-cf115f89234b\") " pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" Jul 22 01:38:38.720286 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.720284 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25xld\" (UniqueName: \"kubernetes.io/projected/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-kube-api-access-25xld\") pod \"console-78fd98559d-c8ghg\" (UID: \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\") " pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:38:38.720534 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.720331 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx2t7\" (UniqueName: \"kubernetes.io/projected/5ac1bc03-0f09-4262-9160-1c19119eb8f4-kube-api-access-fx2t7\") pod \"monitoring-plugin-7b7b7bc8bf-cz8pk\" (UID: \"5ac1bc03-0f09-4262-9160-1c19119eb8f4\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cz8pk" Jul 22 01:38:38.721366 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.721337 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/35a74ac3-5716-4629-8eb8-cf115f89234b-metrics-server-audit-profiles\") pod \"metrics-server-7ccd8878ff-lsgj6\" (UID: \"35a74ac3-5716-4629-8eb8-cf115f89234b\") " pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" Jul 22 01:38:38.722571 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.722071 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/35a74ac3-5716-4629-8eb8-cf115f89234b-audit-log\") pod \"metrics-server-7ccd8878ff-lsgj6\" (UID: \"35a74ac3-5716-4629-8eb8-cf115f89234b\") " pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" Jul 22 01:38:38.731070 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.730980 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slzpz\" (UniqueName: \"kubernetes.io/projected/35a74ac3-5716-4629-8eb8-cf115f89234b-kube-api-access-slzpz\") pod \"metrics-server-7ccd8878ff-lsgj6\" (UID: \"35a74ac3-5716-4629-8eb8-cf115f89234b\") " pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" Jul 22 01:38:38.731334 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.731310 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/35a74ac3-5716-4629-8eb8-cf115f89234b-secret-metrics-server-tls\") pod \"metrics-server-7ccd8878ff-lsgj6\" (UID: \"35a74ac3-5716-4629-8eb8-cf115f89234b\") " pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" Jul 22 01:38:38.744418 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.744340 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/35a74ac3-5716-4629-8eb8-cf115f89234b-client-ca-bundle\") pod \"metrics-server-7ccd8878ff-lsgj6\" (UID: \"35a74ac3-5716-4629-8eb8-cf115f89234b\") " pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" Jul 22 01:38:38.748068 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.748007 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/35a74ac3-5716-4629-8eb8-cf115f89234b-secret-metrics-server-client-certs\") pod \"metrics-server-7ccd8878ff-lsgj6\" (UID: \"35a74ac3-5716-4629-8eb8-cf115f89234b\") " pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" Jul 22 01:38:38.819117 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.819075 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" Jul 22 01:38:38.823095 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.820802 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-oauth-serving-cert\") pod \"console-78fd98559d-c8ghg\" (UID: \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\") " pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:38:38.823095 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.820850 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5ac1bc03-0f09-4262-9160-1c19119eb8f4-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-cz8pk\" (UID: \"5ac1bc03-0f09-4262-9160-1c19119eb8f4\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cz8pk" Jul 22 01:38:38.823095 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.820884 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-service-ca\") pod \"console-78fd98559d-c8ghg\" (UID: \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\") " pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:38:38.823095 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.820956 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-console-serving-cert\") pod \"console-78fd98559d-c8ghg\" (UID: \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\") " pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:38:38.823095 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.820982 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-console-oauth-config\") pod \"console-78fd98559d-c8ghg\" (UID: \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\") " pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:38:38.823095 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.821005 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-trusted-ca-bundle\") pod \"console-78fd98559d-c8ghg\" (UID: \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\") " pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:38:38.823095 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.821042 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25xld\" (UniqueName: \"kubernetes.io/projected/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-kube-api-access-25xld\") pod \"console-78fd98559d-c8ghg\" (UID: \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\") " pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:38:38.823095 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.821081 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx2t7\" (UniqueName: \"kubernetes.io/projected/5ac1bc03-0f09-4262-9160-1c19119eb8f4-kube-api-access-fx2t7\") pod \"monitoring-plugin-7b7b7bc8bf-cz8pk\" (UID: \"5ac1bc03-0f09-4262-9160-1c19119eb8f4\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cz8pk" Jul 22 01:38:38.823095 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.821117 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-console-config\") pod \"console-78fd98559d-c8ghg\" (UID: \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\") " pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:38:38.823095 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.821858 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-console-config\") pod \"console-78fd98559d-c8ghg\" (UID: \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\") " pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:38:38.823095 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.822408 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-oauth-serving-cert\") pod \"console-78fd98559d-c8ghg\" (UID: \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\") " pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:38:38.823729 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.823405 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-service-ca\") pod \"console-78fd98559d-c8ghg\" (UID: \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\") " pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:38:38.824966 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.824940 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-trusted-ca-bundle\") pod \"console-78fd98559d-c8ghg\" (UID: \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\") " pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:38:38.828189 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.828144 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-console-oauth-config\") pod \"console-78fd98559d-c8ghg\" (UID: \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\") " pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:38:38.828338 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.828233 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-console-serving-cert\") pod \"console-78fd98559d-c8ghg\" (UID: \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\") " pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:38:38.848133 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.847124 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5ac1bc03-0f09-4262-9160-1c19119eb8f4-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-cz8pk\" (UID: \"5ac1bc03-0f09-4262-9160-1c19119eb8f4\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cz8pk" Jul 22 01:38:38.854138 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.853900 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx2t7\" (UniqueName: \"kubernetes.io/projected/5ac1bc03-0f09-4262-9160-1c19119eb8f4-kube-api-access-fx2t7\") pod \"monitoring-plugin-7b7b7bc8bf-cz8pk\" (UID: \"5ac1bc03-0f09-4262-9160-1c19119eb8f4\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cz8pk" Jul 22 01:38:38.860019 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.859964 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25xld\" (UniqueName: \"kubernetes.io/projected/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-kube-api-access-25xld\") pod \"console-78fd98559d-c8ghg\" (UID: \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\") " pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:38:38.929188 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.928821 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cz8pk" Jul 22 01:38:38.940877 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:38.940708 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:38:40.126093 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:40.126009 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7ccd8878ff-lsgj6"] Jul 22 01:38:41.315082 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:41.315047 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cz8pk"] Jul 22 01:38:42.886941 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:38:42.886898 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a74ac3_5716_4629_8eb8_cf115f89234b.slice/crio-661de0629157f42ebec7da5e4a3d9e1d3f29cfeca283442673dda72d5b4e7840 WatchSource:0}: Error finding container 661de0629157f42ebec7da5e4a3d9e1d3f29cfeca283442673dda72d5b4e7840: Status 404 returned error can't find the container with id 661de0629157f42ebec7da5e4a3d9e1d3f29cfeca283442673dda72d5b4e7840 Jul 22 01:38:43.058004 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:38:43.057958 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac1bc03_0f09_4262_9160_1c19119eb8f4.slice/crio-fb4fcb16d5166c59f10d1af701fd90c120d88dfd7958699a65e0d3be412448b7 WatchSource:0}: Error finding container fb4fcb16d5166c59f10d1af701fd90c120d88dfd7958699a65e0d3be412448b7: Status 404 returned error can't find the container with id fb4fcb16d5166c59f10d1af701fd90c120d88dfd7958699a65e0d3be412448b7 Jul 22 01:38:43.355252 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:43.355205 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cz8pk" event={"ID":"5ac1bc03-0f09-4262-9160-1c19119eb8f4","Type":"ContainerStarted","Data":"fb4fcb16d5166c59f10d1af701fd90c120d88dfd7958699a65e0d3be412448b7"} Jul 22 01:38:43.356577 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:43.356549 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" event={"ID":"35a74ac3-5716-4629-8eb8-cf115f89234b","Type":"ContainerStarted","Data":"661de0629157f42ebec7da5e4a3d9e1d3f29cfeca283442673dda72d5b4e7840"} Jul 22 01:38:43.565343 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:43.565173 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78fd98559d-c8ghg"] Jul 22 01:38:44.280340 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:38:44.280308 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa9c170_5ce6_4d78_b1ba_fa8c9e1d67c8.slice/crio-4ad83d1fb6f76a4c4387fff5aae7fc761086126e0095818cfaabe7b69d5699f6 WatchSource:0}: Error finding container 4ad83d1fb6f76a4c4387fff5aae7fc761086126e0095818cfaabe7b69d5699f6: Status 404 returned error can't find the container with id 4ad83d1fb6f76a4c4387fff5aae7fc761086126e0095818cfaabe7b69d5699f6 Jul 22 01:38:44.363270 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:44.363230 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78fd98559d-c8ghg" event={"ID":"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8","Type":"ContainerStarted","Data":"4ad83d1fb6f76a4c4387fff5aae7fc761086126e0095818cfaabe7b69d5699f6"} Jul 22 01:38:45.373316 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:45.373280 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" event={"ID":"fdbe34c0-975c-4776-863b-5c06b0b5b850","Type":"ContainerStarted","Data":"7344744a5a4e5403dc40636fccd93082384520fb735d82c95e390502f971a30b"} Jul 22 01:38:45.373857 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:45.373351 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" event={"ID":"fdbe34c0-975c-4776-863b-5c06b0b5b850","Type":"ContainerStarted","Data":"c601b841eb58aaf630eb8e51b0e599235ffb77682b0edd6f601b66cefbc24dfc"} Jul 22 01:38:45.373857 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:45.373376 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" event={"ID":"fdbe34c0-975c-4776-863b-5c06b0b5b850","Type":"ContainerStarted","Data":"80c88f36e373b4798752fd185a3510c3e241b1e5caade4c741493c80347bbb82"} Jul 22 01:38:45.376605 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:45.376570 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ncwrs" event={"ID":"e08c73c4-2356-4d3e-add0-16acc7193db7","Type":"ContainerStarted","Data":"a9697f400a49ca832417884387b22c8a22032aedaf1b6417d036bcb178bf9cd9"} Jul 22 01:38:45.380156 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:45.380130 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78fd98559d-c8ghg" event={"ID":"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8","Type":"ContainerStarted","Data":"9cfdd3111dec58db963da3b89960336c0b81b2c3de72a8e6462771fedd2687bc"} Jul 22 01:38:45.382780 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:45.382751 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vmk5j" event={"ID":"1a297949-cd40-49e4-8643-0f3610080c45","Type":"ContainerStarted","Data":"df88f30911475ce9179a689d4bad24ebe3cbdef1f32b4475ade8d8c3f5b37cae"} Jul 22 01:38:45.384480 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:45.384451 2561 generic.go:356] "Generic (PLEG): container finished" podID="6db543ab-b8f6-4f88-a53b-7f37db4799fb" containerID="dd2aecc8a0ba69889a04cefff4bd473b173a31124389decfeb6741108ede3f41" exitCode=0 Jul 22 01:38:45.384579 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:45.384530 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6db543ab-b8f6-4f88-a53b-7f37db4799fb","Type":"ContainerDied","Data":"dd2aecc8a0ba69889a04cefff4bd473b173a31124389decfeb6741108ede3f41"} Jul 22 01:38:45.388002 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:45.387973 2561 generic.go:356] "Generic (PLEG): container finished" podID="6acfad03-6cc4-4291-a5d4-97256ea86c87" containerID="4d0f79a4bfbcd7a1af4b3f9ec8fe46412076f60850c5a0154e46657dc0f1d8f4" exitCode=0 Jul 22 01:38:45.388102 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:45.388034 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tslm7" event={"ID":"6acfad03-6cc4-4291-a5d4-97256ea86c87","Type":"ContainerDied","Data":"4d0f79a4bfbcd7a1af4b3f9ec8fe46412076f60850c5a0154e46657dc0f1d8f4"} Jul 22 01:38:45.429094 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:45.427345 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ncwrs" podStartSLOduration=3.983693305 podStartE2EDuration="11.427322082s" totalImagesPullingTime="7.443628777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:38:45.399286421 +0000 UTC m=+95.992090822" watchObservedRunningTime="2026-07-22 01:38:45.427322082 +0000 UTC m=+96.020126446" Jul 22 01:38:45.467774 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:45.466516 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78fd98559d-c8ghg" podStartSLOduration=7.466497519 podStartE2EDuration="7.466497519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:38:45.4660106 +0000 UTC m=+96.058814965" watchObservedRunningTime="2026-07-22 01:38:45.466497519 +0000 UTC m=+96.059301885" Jul 22 01:38:46.396099 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:46.396014 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tslm7" event={"ID":"6acfad03-6cc4-4291-a5d4-97256ea86c87","Type":"ContainerStarted","Data":"03d2478540ea8ba3c0eec6a2226b67dc564952dfdd9f491d54835bcf4b985ca1"} Jul 22 01:38:48.941134 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:48.941100 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:38:48.941134 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:48.941145 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:38:48.946157 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:48.946136 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:38:49.413921 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:49.413326 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tslm7" event={"ID":"6acfad03-6cc4-4291-a5d4-97256ea86c87","Type":"ContainerStarted","Data":"d6fa65e1577ab0fa4a15457c08891d4afeb2d0a5ecba15da4d00f98f096017a6"} Jul 22 01:38:49.415763 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:49.415649 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" event={"ID":"fdbe34c0-975c-4776-863b-5c06b0b5b850","Type":"ContainerStarted","Data":"c8f2cfd74ee0613918a66671557459f7e42a51a57a0db6e8adb2367ea5045296"} Jul 22 01:38:49.417784 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:49.417581 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cz8pk" event={"ID":"5ac1bc03-0f09-4262-9160-1c19119eb8f4","Type":"ContainerStarted","Data":"617578f18a6014fe4547d74a276f7e57d24cbcf140f10265ad38a0a1b169eeb1"} Jul 22 01:38:49.417784 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:49.417632 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cz8pk" Jul 22 01:38:49.419379 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:49.419326 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" event={"ID":"35a74ac3-5716-4629-8eb8-cf115f89234b","Type":"ContainerStarted","Data":"2410664e71bf373f9156a08fba88f5640a9b3561128c0f781656e11b9e309797"} Jul 22 01:38:49.423601 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:49.423580 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cz8pk" Jul 22 01:38:49.423714 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:49.423648 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:38:49.432622 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:49.432576 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tslm7" podStartSLOduration=7.624943016 podStartE2EDuration="15.432560996s" totalImagesPullingTime="7.703939778s" totalInitContainerRuntime="103.678202ms" isStatefulPod=true podCreationTimestamp="2026-07-22 01:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:38:49.431191658 +0000 UTC m=+100.023996042" watchObservedRunningTime="2026-07-22 01:38:49.432560996 +0000 UTC m=+100.025365358" Jul 22 01:38:50.427363 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:50.427335 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" event={"ID":"fdbe34c0-975c-4776-863b-5c06b0b5b850","Type":"ContainerStarted","Data":"fd2bbcfb67e538f6fccafa1a63ac41d3272069521ae7448901944ee4bd9b824f"} Jul 22 01:38:50.427749 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:50.427402 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" event={"ID":"fdbe34c0-975c-4776-863b-5c06b0b5b850","Type":"ContainerStarted","Data":"5cc7ae8767905f3ab948294938a41f592debf2cf7bd3bc8ca74e7595a8e4285f"} Jul 22 01:38:50.429232 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:50.429183 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vmk5j" event={"ID":"1a297949-cd40-49e4-8643-0f3610080c45","Type":"ContainerStarted","Data":"2de3c307a3190328b706f482bc5c64e0ed7927bd0c5a946da4f52c8f6b16b5c4"} Jul 22 01:38:50.457495 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:50.457448 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cz8pk" podStartSLOduration=6.478968839 podStartE2EDuration="12.45743497s" totalImagesPullingTime="5.978466131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:38:49.473506001 +0000 UTC m=+100.066310392" watchObservedRunningTime="2026-07-22 01:38:50.45743497 +0000 UTC m=+101.050239332" Jul 22 01:38:50.458150 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:50.458115 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" podStartSLOduration=6.284176962 podStartE2EDuration="12.45810244s" totalImagesPullingTime="6.173925478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:38:50.456722112 +0000 UTC m=+101.049526479" watchObservedRunningTime="2026-07-22 01:38:50.45810244 +0000 UTC m=+101.050906799" Jul 22 01:38:50.478872 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:50.478825 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" podStartSLOduration=3.620551834 podStartE2EDuration="15.478809313s" totalImagesPullingTime="11.858257479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:38:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:38:50.476775252 +0000 UTC m=+101.069579632" watchObservedRunningTime="2026-07-22 01:38:50.478809313 +0000 UTC m=+101.071613675" Jul 22 01:38:50.494597 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:50.494548 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vmk5j" podStartSLOduration=36.399297828 podStartE2EDuration="1m2.494536406s" totalImagesPullingTime="26.095238578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:37:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:38:50.494095843 +0000 UTC m=+101.086900206" watchObservedRunningTime="2026-07-22 01:38:50.494536406 +0000 UTC m=+101.087340765" Jul 22 01:38:51.436946 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:51.436728 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6db543ab-b8f6-4f88-a53b-7f37db4799fb","Type":"ContainerStarted","Data":"07cfc8314dedb6fe193fa85fde99ed79a4ef2b64393d5ceea04c65bf039e2d7b"} Jul 22 01:38:51.436946 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:51.436819 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6db543ab-b8f6-4f88-a53b-7f37db4799fb","Type":"ContainerStarted","Data":"e07f964a6def186be0ad097aeaeade991d9a113680d3ab46b3705bfc94b84f03"} Jul 22 01:38:51.436946 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:51.436865 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6db543ab-b8f6-4f88-a53b-7f37db4799fb","Type":"ContainerStarted","Data":"7ccc9318bb3e01f6a10cab96d9d89c6d6cb0de77dd200d91fc8e0725cea62c24"} Jul 22 01:38:51.436946 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:51.436902 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6db543ab-b8f6-4f88-a53b-7f37db4799fb","Type":"ContainerStarted","Data":"6f72dbf33289c5af5b7046c89b327e79aa70fd817f885c77c427af91ba62baca"} Jul 22 01:38:51.436946 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:51.436949 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6db543ab-b8f6-4f88-a53b-7f37db4799fb","Type":"ContainerStarted","Data":"2118d59de68329799006885991e515fe31c2c518a3ff0496c503019ba955d23c"} Jul 22 01:38:51.437626 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:51.436986 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6db543ab-b8f6-4f88-a53b-7f37db4799fb","Type":"ContainerStarted","Data":"fdbd3dc8ba5737f43be6c20c1cac87c148ccae457de6dbb9b96624476fdd5a7b"} Jul 22 01:38:51.437626 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:51.437353 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" Jul 22 01:38:51.443949 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:51.443922 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6bc54c4b65-8rdms" Jul 22 01:38:51.470327 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:51.470267 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.929130507 podStartE2EDuration="17.470250278s" totalImagesPullingTime="12.519670986s" totalInitContainerRuntime="21.448785ms" isStatefulPod=false podCreationTimestamp="2026-07-22 01:38:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:38:51.468888856 +0000 UTC m=+102.061693241" watchObservedRunningTime="2026-07-22 01:38:51.470250278 +0000 UTC m=+102.063054644" Jul 22 01:38:55.260487 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:55.260458 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fxfmm" Jul 22 01:38:58.819676 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:58.819645 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" Jul 22 01:38:58.820049 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:58.819730 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" Jul 22 01:38:59.334364 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:59.334323 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74d9dc645c-6gn9v" podUID="7a3bbe67-f37d-4d0e-bea4-9f0862d6231a" containerName="console" containerID="cri-o://5842bc56f1e478721c49e076f975af79c16d611d28b5b78c4ef40d50dc8a40db" gracePeriod=15 Jul 22 01:38:59.469965 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:59.469934 2561 generic.go:356] "Generic (PLEG): container finished" podID="7a3bbe67-f37d-4d0e-bea4-9f0862d6231a" containerID="5842bc56f1e478721c49e076f975af79c16d611d28b5b78c4ef40d50dc8a40db" exitCode=0 Jul 22 01:38:59.470124 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:59.470005 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74d9dc645c-6gn9v" event={"ID":"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a","Type":"ContainerDied","Data":"5842bc56f1e478721c49e076f975af79c16d611d28b5b78c4ef40d50dc8a40db"} Jul 22 01:38:59.633794 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:59.633769 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74d9dc645c-6gn9v" Jul 22 01:38:59.699114 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:59.699087 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-console-serving-cert\") pod \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\" (UID: \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\") " Jul 22 01:38:59.699114 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:59.699114 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-console-oauth-config\") pod \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\" (UID: \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\") " Jul 22 01:38:59.699284 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:59.699161 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9thw\" (UniqueName: \"kubernetes.io/projected/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-kube-api-access-k9thw\") pod \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\" (UID: \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\") " Jul 22 01:38:59.699284 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:59.699213 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-service-ca\") pod \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\" (UID: \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\") " Jul 22 01:38:59.699284 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:59.699241 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-console-config\") pod \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\" (UID: \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\") " Jul 22 01:38:59.699284 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:59.699273 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-oauth-serving-cert\") pod \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\" (UID: \"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a\") " Jul 22 01:38:59.699601 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:59.699569 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-service-ca" (OuterVolumeSpecName: "service-ca") pod "7a3bbe67-f37d-4d0e-bea4-9f0862d6231a" (UID: "7a3bbe67-f37d-4d0e-bea4-9f0862d6231a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:38:59.699601 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:59.699574 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-console-config" (OuterVolumeSpecName: "console-config") pod "7a3bbe67-f37d-4d0e-bea4-9f0862d6231a" (UID: "7a3bbe67-f37d-4d0e-bea4-9f0862d6231a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:38:59.699763 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:59.699640 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7a3bbe67-f37d-4d0e-bea4-9f0862d6231a" (UID: "7a3bbe67-f37d-4d0e-bea4-9f0862d6231a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:38:59.701676 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:59.701654 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7a3bbe67-f37d-4d0e-bea4-9f0862d6231a" (UID: "7a3bbe67-f37d-4d0e-bea4-9f0862d6231a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:38:59.701778 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:59.701758 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-kube-api-access-k9thw" (OuterVolumeSpecName: "kube-api-access-k9thw") pod "7a3bbe67-f37d-4d0e-bea4-9f0862d6231a" (UID: "7a3bbe67-f37d-4d0e-bea4-9f0862d6231a"). InnerVolumeSpecName "kube-api-access-k9thw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:38:59.701818 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:59.701777 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7a3bbe67-f37d-4d0e-bea4-9f0862d6231a" (UID: "7a3bbe67-f37d-4d0e-bea4-9f0862d6231a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:38:59.800127 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:59.800104 2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-console-oauth-config\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:38:59.800127 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:59.800125 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9thw\" (UniqueName: \"kubernetes.io/projected/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-kube-api-access-k9thw\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:38:59.800270 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:59.800135 2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-service-ca\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:38:59.800270 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:59.800144 2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-console-config\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:38:59.800270 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:59.800152 2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-oauth-serving-cert\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:38:59.800270 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:38:59.800159 2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a-console-serving-cert\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:39:00.474657 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:00.474627 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74d9dc645c-6gn9v" Jul 22 01:39:00.475114 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:00.474619 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74d9dc645c-6gn9v" event={"ID":"7a3bbe67-f37d-4d0e-bea4-9f0862d6231a","Type":"ContainerDied","Data":"a53232af3c6e0a7ba292d773dd9e289981a09bf2837d98bdb6e1a0d1d7a25005"} Jul 22 01:39:00.475114 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:00.474799 2561 scope.go:117] "RemoveContainer" containerID="5842bc56f1e478721c49e076f975af79c16d611d28b5b78c4ef40d50dc8a40db" Jul 22 01:39:00.492772 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:00.492752 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74d9dc645c-6gn9v"] Jul 22 01:39:00.497091 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:00.497069 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74d9dc645c-6gn9v"] Jul 22 01:39:01.924357 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:01.924328 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a3bbe67-f37d-4d0e-bea4-9f0862d6231a" path="/var/lib/kubelet/pods/7a3bbe67-f37d-4d0e-bea4-9f0862d6231a/volumes" Jul 22 01:39:08.983366 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:08.983340 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ddbnn_8f8e967e-e9b7-47c7-83b0-9fedc09ce1d5/serve-healthcheck-canary/0.log" Jul 22 01:39:18.824791 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:18.824755 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" Jul 22 01:39:18.828617 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:18.828585 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7ccd8878ff-lsgj6" Jul 22 01:39:58.262050 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.262014 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz"] Jul 22 01:39:58.262480 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.262325 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a3bbe67-f37d-4d0e-bea4-9f0862d6231a" containerName="console" Jul 22 01:39:58.262480 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.262335 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a3bbe67-f37d-4d0e-bea4-9f0862d6231a" containerName="console" Jul 22 01:39:58.262480 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.262390 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a3bbe67-f37d-4d0e-bea4-9f0862d6231a" containerName="console" Jul 22 01:39:58.272161 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.272139 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" Jul 22 01:39:58.274787 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.274759 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 22 01:39:58.274932 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.274805 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 22 01:39:58.274932 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.274766 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 22 01:39:58.275122 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.275079 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 22 01:39:58.275433 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.275410 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 22 01:39:58.275537 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.275498 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-2dn5c\"" Jul 22 01:39:58.275735 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.275710 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz"] Jul 22 01:39:58.280321 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.280229 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 22 01:39:58.346947 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.346917 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7262c89e-c2bf-4fc2-915d-b91ee295628e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f5dd86dd9-dqjhz\" (UID: \"7262c89e-c2bf-4fc2-915d-b91ee295628e\") " pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" Jul 22 01:39:58.347066 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.346982 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7262c89e-c2bf-4fc2-915d-b91ee295628e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f5dd86dd9-dqjhz\" (UID: \"7262c89e-c2bf-4fc2-915d-b91ee295628e\") " pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" Jul 22 01:39:58.347066 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.347015 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7262c89e-c2bf-4fc2-915d-b91ee295628e-secret-telemeter-client\") pod \"telemeter-client-6f5dd86dd9-dqjhz\" (UID: \"7262c89e-c2bf-4fc2-915d-b91ee295628e\") " pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" Jul 22 01:39:58.347066 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.347032 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7262c89e-c2bf-4fc2-915d-b91ee295628e-metrics-client-ca\") pod \"telemeter-client-6f5dd86dd9-dqjhz\" (UID: \"7262c89e-c2bf-4fc2-915d-b91ee295628e\") " pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" Jul 22 01:39:58.347167 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.347090 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7262c89e-c2bf-4fc2-915d-b91ee295628e-serving-certs-ca-bundle\") pod \"telemeter-client-6f5dd86dd9-dqjhz\" (UID: \"7262c89e-c2bf-4fc2-915d-b91ee295628e\") " pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" Jul 22 01:39:58.347167 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.347115 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7262c89e-c2bf-4fc2-915d-b91ee295628e-federate-client-tls\") pod \"telemeter-client-6f5dd86dd9-dqjhz\" (UID: \"7262c89e-c2bf-4fc2-915d-b91ee295628e\") " pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" Jul 22 01:39:58.347167 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.347143 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btcw9\" (UniqueName: \"kubernetes.io/projected/7262c89e-c2bf-4fc2-915d-b91ee295628e-kube-api-access-btcw9\") pod \"telemeter-client-6f5dd86dd9-dqjhz\" (UID: \"7262c89e-c2bf-4fc2-915d-b91ee295628e\") " pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" Jul 22 01:39:58.347254 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.347183 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7262c89e-c2bf-4fc2-915d-b91ee295628e-telemeter-client-tls\") pod \"telemeter-client-6f5dd86dd9-dqjhz\" (UID: \"7262c89e-c2bf-4fc2-915d-b91ee295628e\") " pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" Jul 22 01:39:58.447867 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.447831 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7262c89e-c2bf-4fc2-915d-b91ee295628e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f5dd86dd9-dqjhz\" (UID: \"7262c89e-c2bf-4fc2-915d-b91ee295628e\") " pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" Jul 22 01:39:58.448013 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.447886 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7262c89e-c2bf-4fc2-915d-b91ee295628e-secret-telemeter-client\") pod \"telemeter-client-6f5dd86dd9-dqjhz\" (UID: \"7262c89e-c2bf-4fc2-915d-b91ee295628e\") " pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" Jul 22 01:39:58.448013 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.447923 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7262c89e-c2bf-4fc2-915d-b91ee295628e-metrics-client-ca\") pod \"telemeter-client-6f5dd86dd9-dqjhz\" (UID: \"7262c89e-c2bf-4fc2-915d-b91ee295628e\") " pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" Jul 22 01:39:58.448013 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.447967 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7262c89e-c2bf-4fc2-915d-b91ee295628e-serving-certs-ca-bundle\") pod \"telemeter-client-6f5dd86dd9-dqjhz\" (UID: \"7262c89e-c2bf-4fc2-915d-b91ee295628e\") " pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" Jul 22 01:39:58.448013 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.447989 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7262c89e-c2bf-4fc2-915d-b91ee295628e-federate-client-tls\") pod \"telemeter-client-6f5dd86dd9-dqjhz\" (UID: \"7262c89e-c2bf-4fc2-915d-b91ee295628e\") " pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" Jul 22 01:39:58.448013 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.448012 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btcw9\" (UniqueName: \"kubernetes.io/projected/7262c89e-c2bf-4fc2-915d-b91ee295628e-kube-api-access-btcw9\") pod \"telemeter-client-6f5dd86dd9-dqjhz\" (UID: \"7262c89e-c2bf-4fc2-915d-b91ee295628e\") " pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" Jul 22 01:39:58.448263 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.448063 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7262c89e-c2bf-4fc2-915d-b91ee295628e-telemeter-client-tls\") pod \"telemeter-client-6f5dd86dd9-dqjhz\" (UID: \"7262c89e-c2bf-4fc2-915d-b91ee295628e\") " pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" Jul 22 01:39:58.448263 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.448099 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7262c89e-c2bf-4fc2-915d-b91ee295628e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f5dd86dd9-dqjhz\" (UID: \"7262c89e-c2bf-4fc2-915d-b91ee295628e\") " pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" Jul 22 01:39:58.448774 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.448752 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7262c89e-c2bf-4fc2-915d-b91ee295628e-metrics-client-ca\") pod \"telemeter-client-6f5dd86dd9-dqjhz\" (UID: \"7262c89e-c2bf-4fc2-915d-b91ee295628e\") " pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" Jul 22 01:39:58.449066 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.449035 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7262c89e-c2bf-4fc2-915d-b91ee295628e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f5dd86dd9-dqjhz\" (UID: \"7262c89e-c2bf-4fc2-915d-b91ee295628e\") " pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" Jul 22 01:39:58.449182 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.449035 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7262c89e-c2bf-4fc2-915d-b91ee295628e-serving-certs-ca-bundle\") pod \"telemeter-client-6f5dd86dd9-dqjhz\" (UID: \"7262c89e-c2bf-4fc2-915d-b91ee295628e\") " pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" Jul 22 01:39:58.450765 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.450738 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7262c89e-c2bf-4fc2-915d-b91ee295628e-federate-client-tls\") pod \"telemeter-client-6f5dd86dd9-dqjhz\" (UID: \"7262c89e-c2bf-4fc2-915d-b91ee295628e\") " pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" Jul 22 01:39:58.450855 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.450799 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7262c89e-c2bf-4fc2-915d-b91ee295628e-telemeter-client-tls\") pod \"telemeter-client-6f5dd86dd9-dqjhz\" (UID: \"7262c89e-c2bf-4fc2-915d-b91ee295628e\") " pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" Jul 22 01:39:58.451060 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.451044 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7262c89e-c2bf-4fc2-915d-b91ee295628e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f5dd86dd9-dqjhz\" (UID: \"7262c89e-c2bf-4fc2-915d-b91ee295628e\") " pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" Jul 22 01:39:58.451094 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.451060 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7262c89e-c2bf-4fc2-915d-b91ee295628e-secret-telemeter-client\") pod \"telemeter-client-6f5dd86dd9-dqjhz\" (UID: \"7262c89e-c2bf-4fc2-915d-b91ee295628e\") " pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" Jul 22 01:39:58.456822 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.456800 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btcw9\" (UniqueName: \"kubernetes.io/projected/7262c89e-c2bf-4fc2-915d-b91ee295628e-kube-api-access-btcw9\") pod \"telemeter-client-6f5dd86dd9-dqjhz\" (UID: \"7262c89e-c2bf-4fc2-915d-b91ee295628e\") " pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" Jul 22 01:39:58.583492 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.583425 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" Jul 22 01:39:58.720193 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:58.720159 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz"] Jul 22 01:39:58.722758 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:39:58.722717 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7262c89e_c2bf_4fc2_915d_b91ee295628e.slice/crio-76f396cb187be7cdd7e9cfeb1c310b844c79dde4e2350964434c684903ef9ab7 WatchSource:0}: Error finding container 76f396cb187be7cdd7e9cfeb1c310b844c79dde4e2350964434c684903ef9ab7: Status 404 returned error can't find the container with id 76f396cb187be7cdd7e9cfeb1c310b844c79dde4e2350964434c684903ef9ab7 Jul 22 01:39:59.708099 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:39:59.708052 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" event={"ID":"7262c89e-c2bf-4fc2-915d-b91ee295628e","Type":"ContainerStarted","Data":"76f396cb187be7cdd7e9cfeb1c310b844c79dde4e2350964434c684903ef9ab7"} Jul 22 01:40:01.718195 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:01.718161 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" event={"ID":"7262c89e-c2bf-4fc2-915d-b91ee295628e","Type":"ContainerStarted","Data":"73fcbbbb5a4d8110cb13767e52fc70750b6b4b4bb8400a6ab8d6faa647fba6b2"} Jul 22 01:40:01.718544 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:01.718216 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" event={"ID":"7262c89e-c2bf-4fc2-915d-b91ee295628e","Type":"ContainerStarted","Data":"408885d0b1a219f72b16b47182c2719bcce4c0362b713a6a727728fa91ab1f2b"} Jul 22 01:40:01.718544 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:01.718236 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" event={"ID":"7262c89e-c2bf-4fc2-915d-b91ee295628e","Type":"ContainerStarted","Data":"b3c62149519a8d4e57ecfe7826fcf94fdaf695e6de3f1f6ca3a1e8773c669d1d"} Jul 22 01:40:01.747440 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:01.747364 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6f5dd86dd9-dqjhz" podStartSLOduration=1.58038195 podStartE2EDuration="3.747350306s" totalImagesPullingTime="2.166968356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:39:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:40:01.746275371 +0000 UTC m=+172.339079765" watchObservedRunningTime="2026-07-22 01:40:01.747350306 +0000 UTC m=+172.340154668" Jul 22 01:40:02.518249 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:02.518216 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-858b5f7dd9-r2p69"] Jul 22 01:40:02.520456 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:02.520438 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:02.531013 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:02.530993 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-858b5f7dd9-r2p69"] Jul 22 01:40:02.588373 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:02.588348 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-console-config\") pod \"console-858b5f7dd9-r2p69\" (UID: \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\") " pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:02.588518 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:02.588384 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrscm\" (UniqueName: \"kubernetes.io/projected/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-kube-api-access-xrscm\") pod \"console-858b5f7dd9-r2p69\" (UID: \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\") " pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:02.588518 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:02.588403 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-service-ca\") pod \"console-858b5f7dd9-r2p69\" (UID: \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\") " pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:02.588518 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:02.588420 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-console-serving-cert\") pod \"console-858b5f7dd9-r2p69\" (UID: \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\") " pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:02.588518 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:02.588458 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-console-oauth-config\") pod \"console-858b5f7dd9-r2p69\" (UID: \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\") " pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:02.588518 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:02.588496 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-trusted-ca-bundle\") pod \"console-858b5f7dd9-r2p69\" (UID: \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\") " pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:02.588766 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:02.588528 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-oauth-serving-cert\") pod \"console-858b5f7dd9-r2p69\" (UID: \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\") " pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:02.689306 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:02.689279 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-oauth-serving-cert\") pod \"console-858b5f7dd9-r2p69\" (UID: \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\") " pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:02.689433 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:02.689338 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-console-config\") pod \"console-858b5f7dd9-r2p69\" (UID: \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\") " pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:02.689433 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:02.689357 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrscm\" (UniqueName: \"kubernetes.io/projected/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-kube-api-access-xrscm\") pod \"console-858b5f7dd9-r2p69\" (UID: \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\") " pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:02.689433 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:02.689382 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-service-ca\") pod \"console-858b5f7dd9-r2p69\" (UID: \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\") " pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:02.689433 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:02.689408 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-console-serving-cert\") pod \"console-858b5f7dd9-r2p69\" (UID: \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\") " pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:02.689687 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:02.689437 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-console-oauth-config\") pod \"console-858b5f7dd9-r2p69\" (UID: \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\") " pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:02.689687 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:02.689460 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-trusted-ca-bundle\") pod \"console-858b5f7dd9-r2p69\" (UID: \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\") " pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:02.690106 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:02.690082 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-oauth-serving-cert\") pod \"console-858b5f7dd9-r2p69\" (UID: \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\") " pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:02.690213 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:02.690102 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-console-config\") pod \"console-858b5f7dd9-r2p69\" (UID: \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\") " pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:02.690255 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:02.690227 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-service-ca\") pod \"console-858b5f7dd9-r2p69\" (UID: \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\") " pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:02.690544 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:02.690522 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-trusted-ca-bundle\") pod \"console-858b5f7dd9-r2p69\" (UID: \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\") " pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:02.692080 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:02.692060 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-console-serving-cert\") pod \"console-858b5f7dd9-r2p69\" (UID: \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\") " pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:02.692130 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:02.692104 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-console-oauth-config\") pod \"console-858b5f7dd9-r2p69\" (UID: \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\") " pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:02.698770 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:02.698749 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrscm\" (UniqueName: \"kubernetes.io/projected/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-kube-api-access-xrscm\") pod \"console-858b5f7dd9-r2p69\" (UID: \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\") " pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:02.829206 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:02.829137 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:02.955411 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:02.955385 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-858b5f7dd9-r2p69"] Jul 22 01:40:02.957467 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:40:02.957439 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d87c85_d8ff_4b0c_9f55_76a0d2a84d48.slice/crio-308b1ec440449d94be1d25c6e2af5590d3cd9d52ea7598bfb163da83ba9fd82c WatchSource:0}: Error finding container 308b1ec440449d94be1d25c6e2af5590d3cd9d52ea7598bfb163da83ba9fd82c: Status 404 returned error can't find the container with id 308b1ec440449d94be1d25c6e2af5590d3cd9d52ea7598bfb163da83ba9fd82c Jul 22 01:40:03.726654 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:03.726615 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858b5f7dd9-r2p69" event={"ID":"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48","Type":"ContainerStarted","Data":"16c9eb49316e17e9ea22d46ee6cbda47b6c92b53ce3e7f7e5920d892841a9041"} Jul 22 01:40:03.726823 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:03.726670 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858b5f7dd9-r2p69" event={"ID":"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48","Type":"ContainerStarted","Data":"308b1ec440449d94be1d25c6e2af5590d3cd9d52ea7598bfb163da83ba9fd82c"} Jul 22 01:40:03.746080 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:03.746034 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-858b5f7dd9-r2p69" podStartSLOduration=1.746019923 podStartE2EDuration="1.746019923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:40:03.745166341 +0000 UTC m=+174.337970706" watchObservedRunningTime="2026-07-22 01:40:03.746019923 +0000 UTC m=+174.338824282" Jul 22 01:40:12.829827 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:12.829784 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:12.830202 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:12.829865 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:12.834448 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:12.834425 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:13.768187 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:13.768158 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:13.832555 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:13.832523 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78fd98559d-c8ghg"] Jul 22 01:40:13.863981 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:13.863953 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f5fc4bdf6-xvr87"] Jul 22 01:40:13.866664 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:13.866644 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:40:13.875940 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:13.875920 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f5fc4bdf6-xvr87"] Jul 22 01:40:13.982450 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:13.982417 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-trusted-ca-bundle\") pod \"console-6f5fc4bdf6-xvr87\" (UID: \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\") " pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:40:13.982561 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:13.982461 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwgng\" (UniqueName: \"kubernetes.io/projected/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-kube-api-access-pwgng\") pod \"console-6f5fc4bdf6-xvr87\" (UID: \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\") " pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:40:13.982561 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:13.982483 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-oauth-serving-cert\") pod \"console-6f5fc4bdf6-xvr87\" (UID: \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\") " pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:40:13.982561 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:13.982542 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-service-ca\") pod \"console-6f5fc4bdf6-xvr87\" (UID: \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\") " pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:40:13.982685 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:13.982581 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-console-serving-cert\") pod \"console-6f5fc4bdf6-xvr87\" (UID: \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\") " pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:40:13.982685 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:13.982604 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-console-config\") pod \"console-6f5fc4bdf6-xvr87\" (UID: \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\") " pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:40:13.982685 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:13.982624 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-console-oauth-config\") pod \"console-6f5fc4bdf6-xvr87\" (UID: \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\") " pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:40:14.083573 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:14.083497 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-trusted-ca-bundle\") pod \"console-6f5fc4bdf6-xvr87\" (UID: \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\") " pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:40:14.083573 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:14.083554 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwgng\" (UniqueName: \"kubernetes.io/projected/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-kube-api-access-pwgng\") pod \"console-6f5fc4bdf6-xvr87\" (UID: \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\") " pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:40:14.083776 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:14.083586 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-oauth-serving-cert\") pod \"console-6f5fc4bdf6-xvr87\" (UID: \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\") " pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:40:14.083776 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:14.083616 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-service-ca\") pod \"console-6f5fc4bdf6-xvr87\" (UID: \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\") " pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:40:14.083776 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:14.083657 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-console-serving-cert\") pod \"console-6f5fc4bdf6-xvr87\" (UID: \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\") " pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:40:14.083776 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:14.083680 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-console-config\") pod \"console-6f5fc4bdf6-xvr87\" (UID: \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\") " pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:40:14.083776 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:14.083731 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-console-oauth-config\") pod \"console-6f5fc4bdf6-xvr87\" (UID: \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\") " pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:40:14.084432 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:14.084393 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-console-config\") pod \"console-6f5fc4bdf6-xvr87\" (UID: \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\") " pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:40:14.084432 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:14.084413 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-service-ca\") pod \"console-6f5fc4bdf6-xvr87\" (UID: \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\") " pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:40:14.084584 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:14.084393 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-oauth-serving-cert\") pod \"console-6f5fc4bdf6-xvr87\" (UID: \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\") " pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:40:14.084624 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:14.084605 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-trusted-ca-bundle\") pod \"console-6f5fc4bdf6-xvr87\" (UID: \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\") " pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:40:14.086109 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:14.086082 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-console-serving-cert\") pod \"console-6f5fc4bdf6-xvr87\" (UID: \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\") " pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:40:14.086206 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:14.086157 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-console-oauth-config\") pod \"console-6f5fc4bdf6-xvr87\" (UID: \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\") " pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:40:14.093675 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:14.093654 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwgng\" (UniqueName: \"kubernetes.io/projected/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-kube-api-access-pwgng\") pod \"console-6f5fc4bdf6-xvr87\" (UID: \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\") " pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:40:14.175565 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:14.175533 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:40:14.305411 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:14.305387 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f5fc4bdf6-xvr87"] Jul 22 01:40:14.307614 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:40:14.307583 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e37b8a7_9a0b_4be8_8224_09e98ba6a7c0.slice/crio-968fc4e3dca8153b4e006a4d309fdeedfecd3c3d93b4ac9f647bbdb642c356bb WatchSource:0}: Error finding container 968fc4e3dca8153b4e006a4d309fdeedfecd3c3d93b4ac9f647bbdb642c356bb: Status 404 returned error can't find the container with id 968fc4e3dca8153b4e006a4d309fdeedfecd3c3d93b4ac9f647bbdb642c356bb Jul 22 01:40:14.769863 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:14.769820 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5fc4bdf6-xvr87" event={"ID":"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0","Type":"ContainerStarted","Data":"c4628a6788e987b62bcd020cc69704b909e250a1200358021b65be73f92eba20"} Jul 22 01:40:14.770039 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:14.769880 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5fc4bdf6-xvr87" event={"ID":"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0","Type":"ContainerStarted","Data":"968fc4e3dca8153b4e006a4d309fdeedfecd3c3d93b4ac9f647bbdb642c356bb"} Jul 22 01:40:14.791081 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:14.791036 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f5fc4bdf6-xvr87" podStartSLOduration=1.7910207150000002 podStartE2EDuration="1.791020715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:40:14.789449019 +0000 UTC m=+185.382253425" watchObservedRunningTime="2026-07-22 01:40:14.791020715 +0000 UTC m=+185.383825077" Jul 22 01:40:24.176356 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:24.176321 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:40:24.176875 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:24.176610 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:40:24.180955 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:24.180934 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:40:24.814817 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:24.814794 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:40:24.863475 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:24.863449 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-858b5f7dd9-r2p69"] Jul 22 01:40:38.859251 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:38.859189 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-78fd98559d-c8ghg" podUID="aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8" containerName="console" containerID="cri-o://9cfdd3111dec58db963da3b89960336c0b81b2c3de72a8e6462771fedd2687bc" gracePeriod=15 Jul 22 01:40:39.104064 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.104039 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:40:39.186987 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.186964 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-console-config\") pod \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\" (UID: \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\") " Jul 22 01:40:39.187099 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.187013 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-service-ca\") pod \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\" (UID: \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\") " Jul 22 01:40:39.187099 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.187056 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-console-oauth-config\") pod \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\" (UID: \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\") " Jul 22 01:40:39.187099 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.187072 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-console-serving-cert\") pod \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\" (UID: \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\") " Jul 22 01:40:39.187241 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.187117 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25xld\" (UniqueName: \"kubernetes.io/projected/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-kube-api-access-25xld\") pod \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\" (UID: \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\") " Jul 22 01:40:39.187241 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.187147 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-trusted-ca-bundle\") pod \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\" (UID: \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\") " Jul 22 01:40:39.187241 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.187176 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-oauth-serving-cert\") pod \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\" (UID: \"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8\") " Jul 22 01:40:39.187437 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.187397 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-console-config" (OuterVolumeSpecName: "console-config") pod "aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8" (UID: "aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:40:39.187517 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.187498 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-service-ca" (OuterVolumeSpecName: "service-ca") pod "aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8" (UID: "aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:40:39.187797 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.187763 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8" (UID: "aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:40:39.187797 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.187777 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8" (UID: "aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:40:39.189453 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.189431 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-kube-api-access-25xld" (OuterVolumeSpecName: "kube-api-access-25xld") pod "aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8" (UID: "aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8"). InnerVolumeSpecName "kube-api-access-25xld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:40:39.189739 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.189713 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8" (UID: "aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:40:39.189829 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.189747 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8" (UID: "aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:40:39.288159 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.288134 2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-service-ca\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:40:39.288159 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.288157 2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-console-oauth-config\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:40:39.288289 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.288167 2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-console-serving-cert\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:40:39.288289 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.288176 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25xld\" (UniqueName: \"kubernetes.io/projected/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-kube-api-access-25xld\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:40:39.288289 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.288187 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-trusted-ca-bundle\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:40:39.288289 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.288197 2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-oauth-serving-cert\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:40:39.288289 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.288206 2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8-console-config\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:40:39.870864 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.870829 2561 generic.go:356] "Generic (PLEG): container finished" podID="aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8" containerID="9cfdd3111dec58db963da3b89960336c0b81b2c3de72a8e6462771fedd2687bc" exitCode=0 Jul 22 01:40:39.871299 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.870880 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78fd98559d-c8ghg" Jul 22 01:40:39.871299 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.870903 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78fd98559d-c8ghg" event={"ID":"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8","Type":"ContainerDied","Data":"9cfdd3111dec58db963da3b89960336c0b81b2c3de72a8e6462771fedd2687bc"} Jul 22 01:40:39.871299 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.870956 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78fd98559d-c8ghg" event={"ID":"aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8","Type":"ContainerDied","Data":"4ad83d1fb6f76a4c4387fff5aae7fc761086126e0095818cfaabe7b69d5699f6"} Jul 22 01:40:39.871299 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.870985 2561 scope.go:117] "RemoveContainer" containerID="9cfdd3111dec58db963da3b89960336c0b81b2c3de72a8e6462771fedd2687bc" Jul 22 01:40:39.879473 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.879447 2561 scope.go:117] "RemoveContainer" containerID="9cfdd3111dec58db963da3b89960336c0b81b2c3de72a8e6462771fedd2687bc" Jul 22 01:40:39.879770 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:40:39.879750 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cfdd3111dec58db963da3b89960336c0b81b2c3de72a8e6462771fedd2687bc\": container with ID starting with 9cfdd3111dec58db963da3b89960336c0b81b2c3de72a8e6462771fedd2687bc not found: ID does not exist" containerID="9cfdd3111dec58db963da3b89960336c0b81b2c3de72a8e6462771fedd2687bc" Jul 22 01:40:39.879839 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.879780 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cfdd3111dec58db963da3b89960336c0b81b2c3de72a8e6462771fedd2687bc"} err="failed to get container status \"9cfdd3111dec58db963da3b89960336c0b81b2c3de72a8e6462771fedd2687bc\": rpc error: code = NotFound desc = could not find container \"9cfdd3111dec58db963da3b89960336c0b81b2c3de72a8e6462771fedd2687bc\": container with ID starting with 9cfdd3111dec58db963da3b89960336c0b81b2c3de72a8e6462771fedd2687bc not found: ID does not exist" Jul 22 01:40:39.898282 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.898257 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78fd98559d-c8ghg"] Jul 22 01:40:39.903999 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.903981 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78fd98559d-c8ghg"] Jul 22 01:40:39.925301 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:39.925278 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8" path="/var/lib/kubelet/pods/aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8/volumes" Jul 22 01:40:49.883617 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:49.883564 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-858b5f7dd9-r2p69" podUID="05d87c85-d8ff-4b0c-9f55-76a0d2a84d48" containerName="console" containerID="cri-o://16c9eb49316e17e9ea22d46ee6cbda47b6c92b53ce3e7f7e5920d892841a9041" gracePeriod=15 Jul 22 01:40:50.120194 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.120169 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:50.164917 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.164859 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-service-ca\") pod \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\" (UID: \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\") " Jul 22 01:40:50.164917 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.164893 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-oauth-serving-cert\") pod \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\" (UID: \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\") " Jul 22 01:40:50.164917 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.164912 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-console-config\") pod \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\" (UID: \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\") " Jul 22 01:40:50.165125 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.165106 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-trusted-ca-bundle\") pod \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\" (UID: \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\") " Jul 22 01:40:50.165201 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.165185 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrscm\" (UniqueName: \"kubernetes.io/projected/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-kube-api-access-xrscm\") pod \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\" (UID: \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\") " Jul 22 01:40:50.165257 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.165218 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-console-serving-cert\") pod \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\" (UID: \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\") " Jul 22 01:40:50.165257 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.165245 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-console-oauth-config\") pod \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\" (UID: \"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48\") " Jul 22 01:40:50.165353 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.165258 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-service-ca" (OuterVolumeSpecName: "service-ca") pod "05d87c85-d8ff-4b0c-9f55-76a0d2a84d48" (UID: "05d87c85-d8ff-4b0c-9f55-76a0d2a84d48"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:40:50.165353 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.165266 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "05d87c85-d8ff-4b0c-9f55-76a0d2a84d48" (UID: "05d87c85-d8ff-4b0c-9f55-76a0d2a84d48"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:40:50.165353 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.165294 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-console-config" (OuterVolumeSpecName: "console-config") pod "05d87c85-d8ff-4b0c-9f55-76a0d2a84d48" (UID: "05d87c85-d8ff-4b0c-9f55-76a0d2a84d48"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:40:50.165522 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.165490 2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-service-ca\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:40:50.165522 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.165521 2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-oauth-serving-cert\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:40:50.165709 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.165532 2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-console-config\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:40:50.165709 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.165534 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "05d87c85-d8ff-4b0c-9f55-76a0d2a84d48" (UID: "05d87c85-d8ff-4b0c-9f55-76a0d2a84d48"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:40:50.167360 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.167334 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "05d87c85-d8ff-4b0c-9f55-76a0d2a84d48" (UID: "05d87c85-d8ff-4b0c-9f55-76a0d2a84d48"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:40:50.167445 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.167387 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "05d87c85-d8ff-4b0c-9f55-76a0d2a84d48" (UID: "05d87c85-d8ff-4b0c-9f55-76a0d2a84d48"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:40:50.167445 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.167405 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-kube-api-access-xrscm" (OuterVolumeSpecName: "kube-api-access-xrscm") pod "05d87c85-d8ff-4b0c-9f55-76a0d2a84d48" (UID: "05d87c85-d8ff-4b0c-9f55-76a0d2a84d48"). InnerVolumeSpecName "kube-api-access-xrscm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:40:50.266002 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.265980 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrscm\" (UniqueName: \"kubernetes.io/projected/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-kube-api-access-xrscm\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:40:50.266002 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.266000 2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-console-serving-cert\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:40:50.266125 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.266011 2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-console-oauth-config\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:40:50.266125 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.266020 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48-trusted-ca-bundle\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:40:50.914668 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.914633 2561 generic.go:356] "Generic (PLEG): container finished" podID="05d87c85-d8ff-4b0c-9f55-76a0d2a84d48" containerID="16c9eb49316e17e9ea22d46ee6cbda47b6c92b53ce3e7f7e5920d892841a9041" exitCode=0 Jul 22 01:40:50.915086 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.914679 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858b5f7dd9-r2p69" Jul 22 01:40:50.915086 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.914733 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858b5f7dd9-r2p69" event={"ID":"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48","Type":"ContainerDied","Data":"16c9eb49316e17e9ea22d46ee6cbda47b6c92b53ce3e7f7e5920d892841a9041"} Jul 22 01:40:50.915086 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.914788 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858b5f7dd9-r2p69" event={"ID":"05d87c85-d8ff-4b0c-9f55-76a0d2a84d48","Type":"ContainerDied","Data":"308b1ec440449d94be1d25c6e2af5590d3cd9d52ea7598bfb163da83ba9fd82c"} Jul 22 01:40:50.915086 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.914814 2561 scope.go:117] "RemoveContainer" containerID="16c9eb49316e17e9ea22d46ee6cbda47b6c92b53ce3e7f7e5920d892841a9041" Jul 22 01:40:50.923598 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.923583 2561 scope.go:117] "RemoveContainer" containerID="16c9eb49316e17e9ea22d46ee6cbda47b6c92b53ce3e7f7e5920d892841a9041" Jul 22 01:40:50.923853 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:40:50.923834 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16c9eb49316e17e9ea22d46ee6cbda47b6c92b53ce3e7f7e5920d892841a9041\": container with ID starting with 16c9eb49316e17e9ea22d46ee6cbda47b6c92b53ce3e7f7e5920d892841a9041 not found: ID does not exist" containerID="16c9eb49316e17e9ea22d46ee6cbda47b6c92b53ce3e7f7e5920d892841a9041" Jul 22 01:40:50.923912 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.923861 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16c9eb49316e17e9ea22d46ee6cbda47b6c92b53ce3e7f7e5920d892841a9041"} err="failed to get container status \"16c9eb49316e17e9ea22d46ee6cbda47b6c92b53ce3e7f7e5920d892841a9041\": rpc error: code = NotFound desc = could not find container \"16c9eb49316e17e9ea22d46ee6cbda47b6c92b53ce3e7f7e5920d892841a9041\": container with ID starting with 16c9eb49316e17e9ea22d46ee6cbda47b6c92b53ce3e7f7e5920d892841a9041 not found: ID does not exist" Jul 22 01:40:50.937317 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.937291 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-858b5f7dd9-r2p69"] Jul 22 01:40:50.945310 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:50.945279 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-858b5f7dd9-r2p69"] Jul 22 01:40:51.923282 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:40:51.923247 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05d87c85-d8ff-4b0c-9f55-76a0d2a84d48" path="/var/lib/kubelet/pods/05d87c85-d8ff-4b0c-9f55-76a0d2a84d48/volumes" Jul 22 01:41:15.885226 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:15.885193 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w"] Jul 22 01:41:15.885652 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:15.885489 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05d87c85-d8ff-4b0c-9f55-76a0d2a84d48" containerName="console" Jul 22 01:41:15.885652 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:15.885500 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="05d87c85-d8ff-4b0c-9f55-76a0d2a84d48" containerName="console" Jul 22 01:41:15.885652 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:15.885517 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8" containerName="console" Jul 22 01:41:15.885652 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:15.885523 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8" containerName="console" Jul 22 01:41:15.885652 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:15.885580 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="05d87c85-d8ff-4b0c-9f55-76a0d2a84d48" containerName="console" Jul 22 01:41:15.885652 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:15.885589 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaa9c170-5ce6-4d78-b1ba-fa8c9e1d67c8" containerName="console" Jul 22 01:41:15.888785 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:15.888768 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w" Jul 22 01:41:15.891289 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:15.891262 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 01:41:15.891410 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:15.891346 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 01:41:15.892135 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:15.892120 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-snmkh\"" Jul 22 01:41:15.897493 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:15.897469 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w"] Jul 22 01:41:15.959153 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:15.959125 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0aa00af-8c2f-440c-a415-3f56afab5cc7-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w\" (UID: \"f0aa00af-8c2f-440c-a415-3f56afab5cc7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w" Jul 22 01:41:15.959282 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:15.959197 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0aa00af-8c2f-440c-a415-3f56afab5cc7-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w\" (UID: \"f0aa00af-8c2f-440c-a415-3f56afab5cc7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w" Jul 22 01:41:15.959282 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:15.959225 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrqtl\" (UniqueName: \"kubernetes.io/projected/f0aa00af-8c2f-440c-a415-3f56afab5cc7-kube-api-access-jrqtl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w\" (UID: \"f0aa00af-8c2f-440c-a415-3f56afab5cc7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w" Jul 22 01:41:16.060417 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:16.060383 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0aa00af-8c2f-440c-a415-3f56afab5cc7-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w\" (UID: \"f0aa00af-8c2f-440c-a415-3f56afab5cc7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w" Jul 22 01:41:16.060566 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:16.060424 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrqtl\" (UniqueName: \"kubernetes.io/projected/f0aa00af-8c2f-440c-a415-3f56afab5cc7-kube-api-access-jrqtl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w\" (UID: \"f0aa00af-8c2f-440c-a415-3f56afab5cc7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w" Jul 22 01:41:16.060566 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:16.060489 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0aa00af-8c2f-440c-a415-3f56afab5cc7-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w\" (UID: \"f0aa00af-8c2f-440c-a415-3f56afab5cc7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w" Jul 22 01:41:16.060807 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:16.060786 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0aa00af-8c2f-440c-a415-3f56afab5cc7-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w\" (UID: \"f0aa00af-8c2f-440c-a415-3f56afab5cc7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w" Jul 22 01:41:16.060895 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:16.060832 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0aa00af-8c2f-440c-a415-3f56afab5cc7-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w\" (UID: \"f0aa00af-8c2f-440c-a415-3f56afab5cc7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w" Jul 22 01:41:16.069994 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:16.069976 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrqtl\" (UniqueName: \"kubernetes.io/projected/f0aa00af-8c2f-440c-a415-3f56afab5cc7-kube-api-access-jrqtl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w\" (UID: \"f0aa00af-8c2f-440c-a415-3f56afab5cc7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w" Jul 22 01:41:16.198471 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:16.198447 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w" Jul 22 01:41:16.327184 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:16.327156 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w"] Jul 22 01:41:16.330125 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:41:16.330097 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0aa00af_8c2f_440c_a415_3f56afab5cc7.slice/crio-e129c7f8eb5967a256953dc41b7820289b4fa2bfe2643d75568a1454e9eab568 WatchSource:0}: Error finding container e129c7f8eb5967a256953dc41b7820289b4fa2bfe2643d75568a1454e9eab568: Status 404 returned error can't find the container with id e129c7f8eb5967a256953dc41b7820289b4fa2bfe2643d75568a1454e9eab568 Jul 22 01:41:17.017002 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:17.016965 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w" event={"ID":"f0aa00af-8c2f-440c-a415-3f56afab5cc7","Type":"ContainerStarted","Data":"e129c7f8eb5967a256953dc41b7820289b4fa2bfe2643d75568a1454e9eab568"} Jul 22 01:41:24.044959 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:24.044926 2561 generic.go:356] "Generic (PLEG): container finished" podID="f0aa00af-8c2f-440c-a415-3f56afab5cc7" containerID="5297b0b16a0fc3515e440da0b7514d8463cef45a7dd74693ebab0da542a94f0f" exitCode=0 Jul 22 01:41:24.045324 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:24.044985 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w" event={"ID":"f0aa00af-8c2f-440c-a415-3f56afab5cc7","Type":"ContainerDied","Data":"5297b0b16a0fc3515e440da0b7514d8463cef45a7dd74693ebab0da542a94f0f"} Jul 22 01:41:27.059392 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:27.059307 2561 generic.go:356] "Generic (PLEG): container finished" podID="f0aa00af-8c2f-440c-a415-3f56afab5cc7" containerID="06e94ebb4d3de540ad5941d317333e5be2bfe5aa13da44a1e8232c61e7655133" exitCode=0 Jul 22 01:41:27.059762 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:27.059392 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w" event={"ID":"f0aa00af-8c2f-440c-a415-3f56afab5cc7","Type":"ContainerDied","Data":"06e94ebb4d3de540ad5941d317333e5be2bfe5aa13da44a1e8232c61e7655133"} Jul 22 01:41:35.096368 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:35.096332 2561 generic.go:356] "Generic (PLEG): container finished" podID="f0aa00af-8c2f-440c-a415-3f56afab5cc7" containerID="1c9feb0e29da76a5fef81fefbf02f70108914c18e76accd13d4881c96f134bc8" exitCode=0 Jul 22 01:41:35.096778 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:35.096421 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w" event={"ID":"f0aa00af-8c2f-440c-a415-3f56afab5cc7","Type":"ContainerDied","Data":"1c9feb0e29da76a5fef81fefbf02f70108914c18e76accd13d4881c96f134bc8"} Jul 22 01:41:36.219283 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:36.219256 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w" Jul 22 01:41:36.337254 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:36.337224 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0aa00af-8c2f-440c-a415-3f56afab5cc7-util\") pod \"f0aa00af-8c2f-440c-a415-3f56afab5cc7\" (UID: \"f0aa00af-8c2f-440c-a415-3f56afab5cc7\") " Jul 22 01:41:36.337404 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:36.337293 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0aa00af-8c2f-440c-a415-3f56afab5cc7-bundle\") pod \"f0aa00af-8c2f-440c-a415-3f56afab5cc7\" (UID: \"f0aa00af-8c2f-440c-a415-3f56afab5cc7\") " Jul 22 01:41:36.337404 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:36.337327 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrqtl\" (UniqueName: \"kubernetes.io/projected/f0aa00af-8c2f-440c-a415-3f56afab5cc7-kube-api-access-jrqtl\") pod \"f0aa00af-8c2f-440c-a415-3f56afab5cc7\" (UID: \"f0aa00af-8c2f-440c-a415-3f56afab5cc7\") " Jul 22 01:41:36.337997 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:36.337930 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0aa00af-8c2f-440c-a415-3f56afab5cc7-bundle" (OuterVolumeSpecName: "bundle") pod "f0aa00af-8c2f-440c-a415-3f56afab5cc7" (UID: "f0aa00af-8c2f-440c-a415-3f56afab5cc7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:41:36.339546 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:36.339523 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0aa00af-8c2f-440c-a415-3f56afab5cc7-kube-api-access-jrqtl" (OuterVolumeSpecName: "kube-api-access-jrqtl") pod "f0aa00af-8c2f-440c-a415-3f56afab5cc7" (UID: "f0aa00af-8c2f-440c-a415-3f56afab5cc7"). InnerVolumeSpecName "kube-api-access-jrqtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:41:36.341441 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:36.341417 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0aa00af-8c2f-440c-a415-3f56afab5cc7-util" (OuterVolumeSpecName: "util") pod "f0aa00af-8c2f-440c-a415-3f56afab5cc7" (UID: "f0aa00af-8c2f-440c-a415-3f56afab5cc7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:41:36.437937 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:36.437917 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0aa00af-8c2f-440c-a415-3f56afab5cc7-util\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:41:36.437937 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:36.437937 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0aa00af-8c2f-440c-a415-3f56afab5cc7-bundle\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:41:36.438042 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:36.437946 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrqtl\" (UniqueName: \"kubernetes.io/projected/f0aa00af-8c2f-440c-a415-3f56afab5cc7-kube-api-access-jrqtl\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:41:37.105852 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:37.105825 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w" Jul 22 01:41:37.106028 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:37.105853 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zqf5w" event={"ID":"f0aa00af-8c2f-440c-a415-3f56afab5cc7","Type":"ContainerDied","Data":"e129c7f8eb5967a256953dc41b7820289b4fa2bfe2643d75568a1454e9eab568"} Jul 22 01:41:37.106028 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:37.105900 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e129c7f8eb5967a256953dc41b7820289b4fa2bfe2643d75568a1454e9eab568" Jul 22 01:41:48.559644 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.559607 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs"] Jul 22 01:41:48.560181 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.560088 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0aa00af-8c2f-440c-a415-3f56afab5cc7" containerName="pull" Jul 22 01:41:48.560181 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.560107 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0aa00af-8c2f-440c-a415-3f56afab5cc7" containerName="pull" Jul 22 01:41:48.560181 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.560131 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0aa00af-8c2f-440c-a415-3f56afab5cc7" containerName="extract" Jul 22 01:41:48.560181 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.560140 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0aa00af-8c2f-440c-a415-3f56afab5cc7" containerName="extract" Jul 22 01:41:48.560181 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.560158 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0aa00af-8c2f-440c-a415-3f56afab5cc7" containerName="util" Jul 22 01:41:48.560181 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.560166 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0aa00af-8c2f-440c-a415-3f56afab5cc7" containerName="util" Jul 22 01:41:48.560476 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.560248 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0aa00af-8c2f-440c-a415-3f56afab5cc7" containerName="extract" Jul 22 01:41:48.564225 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.564204 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs" Jul 22 01:41:48.566663 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.566644 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 01:41:48.566782 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.566660 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 01:41:48.567548 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.567532 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-snmkh\"" Jul 22 01:41:48.571794 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.571517 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs"] Jul 22 01:41:48.628337 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.628314 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67b85af5-48a4-4a3b-b369-6fdea737dea0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs\" (UID: \"67b85af5-48a4-4a3b-b369-6fdea737dea0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs" Jul 22 01:41:48.628443 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.628355 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67b85af5-48a4-4a3b-b369-6fdea737dea0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs\" (UID: \"67b85af5-48a4-4a3b-b369-6fdea737dea0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs" Jul 22 01:41:48.628443 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.628390 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bndbh\" (UniqueName: \"kubernetes.io/projected/67b85af5-48a4-4a3b-b369-6fdea737dea0-kube-api-access-bndbh\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs\" (UID: \"67b85af5-48a4-4a3b-b369-6fdea737dea0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs" Jul 22 01:41:48.729260 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.729222 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67b85af5-48a4-4a3b-b369-6fdea737dea0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs\" (UID: \"67b85af5-48a4-4a3b-b369-6fdea737dea0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs" Jul 22 01:41:48.729466 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.729273 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67b85af5-48a4-4a3b-b369-6fdea737dea0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs\" (UID: \"67b85af5-48a4-4a3b-b369-6fdea737dea0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs" Jul 22 01:41:48.729466 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.729304 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bndbh\" (UniqueName: \"kubernetes.io/projected/67b85af5-48a4-4a3b-b369-6fdea737dea0-kube-api-access-bndbh\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs\" (UID: \"67b85af5-48a4-4a3b-b369-6fdea737dea0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs" Jul 22 01:41:48.729756 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.729732 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67b85af5-48a4-4a3b-b369-6fdea737dea0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs\" (UID: \"67b85af5-48a4-4a3b-b369-6fdea737dea0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs" Jul 22 01:41:48.729756 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.729748 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67b85af5-48a4-4a3b-b369-6fdea737dea0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs\" (UID: \"67b85af5-48a4-4a3b-b369-6fdea737dea0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs" Jul 22 01:41:48.738475 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.738451 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bndbh\" (UniqueName: \"kubernetes.io/projected/67b85af5-48a4-4a3b-b369-6fdea737dea0-kube-api-access-bndbh\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs\" (UID: \"67b85af5-48a4-4a3b-b369-6fdea737dea0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs" Jul 22 01:41:48.874224 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.874160 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs" Jul 22 01:41:48.948417 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.948389 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wlslm"] Jul 22 01:41:48.950725 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.950708 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wlslm" Jul 22 01:41:48.953494 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.953468 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-dvqh5\"" Jul 22 01:41:48.954201 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.954115 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 22 01:41:48.954201 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.954143 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 22 01:41:48.965058 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:48.965037 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wlslm"] Jul 22 01:41:49.008738 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:49.008710 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs"] Jul 22 01:41:49.010935 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:41:49.010910 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b85af5_48a4_4a3b_b369_6fdea737dea0.slice/crio-625caa33f6df6771d838839ea5bcc875fe50a1e46d996e264a99f9a27d564e31 WatchSource:0}: Error finding container 625caa33f6df6771d838839ea5bcc875fe50a1e46d996e264a99f9a27d564e31: Status 404 returned error can't find the container with id 625caa33f6df6771d838839ea5bcc875fe50a1e46d996e264a99f9a27d564e31 Jul 22 01:41:49.031802 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:49.031782 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2dpx\" (UniqueName: \"kubernetes.io/projected/df49a874-9777-4e02-a84a-c3fe3339c56d-kube-api-access-n2dpx\") pod \"cert-manager-webhook-6486ffd997-wlslm\" (UID: \"df49a874-9777-4e02-a84a-c3fe3339c56d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wlslm" Jul 22 01:41:49.031887 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:49.031839 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df49a874-9777-4e02-a84a-c3fe3339c56d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wlslm\" (UID: \"df49a874-9777-4e02-a84a-c3fe3339c56d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wlslm" Jul 22 01:41:49.132801 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:49.132738 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2dpx\" (UniqueName: \"kubernetes.io/projected/df49a874-9777-4e02-a84a-c3fe3339c56d-kube-api-access-n2dpx\") pod \"cert-manager-webhook-6486ffd997-wlslm\" (UID: \"df49a874-9777-4e02-a84a-c3fe3339c56d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wlslm" Jul 22 01:41:49.132910 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:49.132826 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df49a874-9777-4e02-a84a-c3fe3339c56d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wlslm\" (UID: \"df49a874-9777-4e02-a84a-c3fe3339c56d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wlslm" Jul 22 01:41:49.141221 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:49.141196 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df49a874-9777-4e02-a84a-c3fe3339c56d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wlslm\" (UID: \"df49a874-9777-4e02-a84a-c3fe3339c56d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wlslm" Jul 22 01:41:49.141386 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:49.141368 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2dpx\" (UniqueName: \"kubernetes.io/projected/df49a874-9777-4e02-a84a-c3fe3339c56d-kube-api-access-n2dpx\") pod \"cert-manager-webhook-6486ffd997-wlslm\" (UID: \"df49a874-9777-4e02-a84a-c3fe3339c56d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wlslm" Jul 22 01:41:49.154144 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:49.154120 2561 generic.go:356] "Generic (PLEG): container finished" podID="67b85af5-48a4-4a3b-b369-6fdea737dea0" containerID="15f0b4625f0d3e91493de0f44052314a690da0d179da47abc895eaa22dcb2232" exitCode=0 Jul 22 01:41:49.154228 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:49.154197 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs" event={"ID":"67b85af5-48a4-4a3b-b369-6fdea737dea0","Type":"ContainerDied","Data":"15f0b4625f0d3e91493de0f44052314a690da0d179da47abc895eaa22dcb2232"} Jul 22 01:41:49.154277 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:49.154245 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs" event={"ID":"67b85af5-48a4-4a3b-b369-6fdea737dea0","Type":"ContainerStarted","Data":"625caa33f6df6771d838839ea5bcc875fe50a1e46d996e264a99f9a27d564e31"} Jul 22 01:41:49.261647 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:49.261620 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wlslm" Jul 22 01:41:49.384705 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:49.384667 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wlslm"] Jul 22 01:41:49.386362 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:41:49.386331 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf49a874_9777_4e02_a84a_c3fe3339c56d.slice/crio-184a76a720c646d4dba744f3db69e75bf3d22aafd2af03fc9482835399900913 WatchSource:0}: Error finding container 184a76a720c646d4dba744f3db69e75bf3d22aafd2af03fc9482835399900913: Status 404 returned error can't find the container with id 184a76a720c646d4dba744f3db69e75bf3d22aafd2af03fc9482835399900913 Jul 22 01:41:50.161250 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:50.161155 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wlslm" event={"ID":"df49a874-9777-4e02-a84a-c3fe3339c56d","Type":"ContainerStarted","Data":"184a76a720c646d4dba744f3db69e75bf3d22aafd2af03fc9482835399900913"} Jul 22 01:41:53.182631 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:53.182584 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wlslm" event={"ID":"df49a874-9777-4e02-a84a-c3fe3339c56d","Type":"ContainerStarted","Data":"5bb46fc983a88eac6b6e29194e9421a2fe10a9208428b4d68e52dd3b78b9f9a0"} Jul 22 01:41:53.183066 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:53.182648 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wlslm" Jul 22 01:41:53.184454 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:53.184428 2561 generic.go:356] "Generic (PLEG): container finished" podID="67b85af5-48a4-4a3b-b369-6fdea737dea0" containerID="dd8c90dd6e781245828352972e1e6681ba8b5c01ce6323337e210d6d2d51a370" exitCode=0 Jul 22 01:41:53.184562 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:53.184473 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs" event={"ID":"67b85af5-48a4-4a3b-b369-6fdea737dea0","Type":"ContainerDied","Data":"dd8c90dd6e781245828352972e1e6681ba8b5c01ce6323337e210d6d2d51a370"} Jul 22 01:41:53.202849 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:53.202803 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-wlslm" podStartSLOduration=2.248016203 podStartE2EDuration="5.202784589s" totalImagesPullingTime="2.954768386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:41:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:41:53.199790776 +0000 UTC m=+283.792595140" watchObservedRunningTime="2026-07-22 01:41:53.202784589 +0000 UTC m=+283.795588955" Jul 22 01:41:54.190369 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:54.190335 2561 generic.go:356] "Generic (PLEG): container finished" podID="67b85af5-48a4-4a3b-b369-6fdea737dea0" containerID="2316e49c7f9b5e782af45d1f61f338f2319702297d790119db7cbeb10ff5debb" exitCode=0 Jul 22 01:41:54.190740 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:54.190407 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs" event={"ID":"67b85af5-48a4-4a3b-b369-6fdea737dea0","Type":"ContainerDied","Data":"2316e49c7f9b5e782af45d1f61f338f2319702297d790119db7cbeb10ff5debb"} Jul 22 01:41:55.317921 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:55.317893 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs" Jul 22 01:41:55.383978 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:55.383948 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bndbh\" (UniqueName: \"kubernetes.io/projected/67b85af5-48a4-4a3b-b369-6fdea737dea0-kube-api-access-bndbh\") pod \"67b85af5-48a4-4a3b-b369-6fdea737dea0\" (UID: \"67b85af5-48a4-4a3b-b369-6fdea737dea0\") " Jul 22 01:41:55.384098 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:55.384009 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67b85af5-48a4-4a3b-b369-6fdea737dea0-util\") pod \"67b85af5-48a4-4a3b-b369-6fdea737dea0\" (UID: \"67b85af5-48a4-4a3b-b369-6fdea737dea0\") " Jul 22 01:41:55.384098 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:55.384030 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67b85af5-48a4-4a3b-b369-6fdea737dea0-bundle\") pod \"67b85af5-48a4-4a3b-b369-6fdea737dea0\" (UID: \"67b85af5-48a4-4a3b-b369-6fdea737dea0\") " Jul 22 01:41:55.384442 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:55.384418 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b85af5-48a4-4a3b-b369-6fdea737dea0-bundle" (OuterVolumeSpecName: "bundle") pod "67b85af5-48a4-4a3b-b369-6fdea737dea0" (UID: "67b85af5-48a4-4a3b-b369-6fdea737dea0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:41:55.386184 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:55.386161 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b85af5-48a4-4a3b-b369-6fdea737dea0-kube-api-access-bndbh" (OuterVolumeSpecName: "kube-api-access-bndbh") pod "67b85af5-48a4-4a3b-b369-6fdea737dea0" (UID: "67b85af5-48a4-4a3b-b369-6fdea737dea0"). InnerVolumeSpecName "kube-api-access-bndbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:41:55.390957 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:55.390932 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b85af5-48a4-4a3b-b369-6fdea737dea0-util" (OuterVolumeSpecName: "util") pod "67b85af5-48a4-4a3b-b369-6fdea737dea0" (UID: "67b85af5-48a4-4a3b-b369-6fdea737dea0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:41:55.484770 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:55.484683 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bndbh\" (UniqueName: \"kubernetes.io/projected/67b85af5-48a4-4a3b-b369-6fdea737dea0-kube-api-access-bndbh\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:41:55.484770 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:55.484727 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67b85af5-48a4-4a3b-b369-6fdea737dea0-util\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:41:55.484770 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:55.484743 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67b85af5-48a4-4a3b-b369-6fdea737dea0-bundle\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:41:56.200098 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:56.200028 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs" Jul 22 01:41:56.200098 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:56.200048 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmfjzs" event={"ID":"67b85af5-48a4-4a3b-b369-6fdea737dea0","Type":"ContainerDied","Data":"625caa33f6df6771d838839ea5bcc875fe50a1e46d996e264a99f9a27d564e31"} Jul 22 01:41:56.200098 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:56.200090 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="625caa33f6df6771d838839ea5bcc875fe50a1e46d996e264a99f9a27d564e31" Jul 22 01:41:58.736144 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:58.736104 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4rbhc"] Jul 22 01:41:58.736584 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:58.736433 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67b85af5-48a4-4a3b-b369-6fdea737dea0" containerName="util" Jul 22 01:41:58.736584 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:58.736446 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="67b85af5-48a4-4a3b-b369-6fdea737dea0" containerName="util" Jul 22 01:41:58.736584 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:58.736457 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67b85af5-48a4-4a3b-b369-6fdea737dea0" containerName="pull" Jul 22 01:41:58.736584 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:58.736462 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="67b85af5-48a4-4a3b-b369-6fdea737dea0" containerName="pull" Jul 22 01:41:58.736584 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:58.736468 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67b85af5-48a4-4a3b-b369-6fdea737dea0" containerName="extract" Jul 22 01:41:58.736584 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:58.736474 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="67b85af5-48a4-4a3b-b369-6fdea737dea0" containerName="extract" Jul 22 01:41:58.736584 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:58.736532 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="67b85af5-48a4-4a3b-b369-6fdea737dea0" containerName="extract" Jul 22 01:41:58.738652 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:58.738637 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4rbhc" Jul 22 01:41:58.740686 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:58.740665 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ttx92\"" Jul 22 01:41:58.749768 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:58.749745 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4rbhc"] Jul 22 01:41:58.808898 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:58.808872 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhst6\" (UniqueName: \"kubernetes.io/projected/bc5c0052-9517-42b5-a48f-9716080c98fb-kube-api-access-nhst6\") pod \"cert-manager-55b4d954d5-4rbhc\" (UID: \"bc5c0052-9517-42b5-a48f-9716080c98fb\") " pod="cert-manager/cert-manager-55b4d954d5-4rbhc" Jul 22 01:41:58.809029 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:58.808940 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc5c0052-9517-42b5-a48f-9716080c98fb-bound-sa-token\") pod \"cert-manager-55b4d954d5-4rbhc\" (UID: \"bc5c0052-9517-42b5-a48f-9716080c98fb\") " pod="cert-manager/cert-manager-55b4d954d5-4rbhc" Jul 22 01:41:58.909381 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:58.909358 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc5c0052-9517-42b5-a48f-9716080c98fb-bound-sa-token\") pod \"cert-manager-55b4d954d5-4rbhc\" (UID: \"bc5c0052-9517-42b5-a48f-9716080c98fb\") " pod="cert-manager/cert-manager-55b4d954d5-4rbhc" Jul 22 01:41:58.909511 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:58.909394 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhst6\" (UniqueName: \"kubernetes.io/projected/bc5c0052-9517-42b5-a48f-9716080c98fb-kube-api-access-nhst6\") pod \"cert-manager-55b4d954d5-4rbhc\" (UID: \"bc5c0052-9517-42b5-a48f-9716080c98fb\") " pod="cert-manager/cert-manager-55b4d954d5-4rbhc" Jul 22 01:41:58.922059 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:58.922038 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhst6\" (UniqueName: \"kubernetes.io/projected/bc5c0052-9517-42b5-a48f-9716080c98fb-kube-api-access-nhst6\") pod \"cert-manager-55b4d954d5-4rbhc\" (UID: \"bc5c0052-9517-42b5-a48f-9716080c98fb\") " pod="cert-manager/cert-manager-55b4d954d5-4rbhc" Jul 22 01:41:58.922059 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:58.922043 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc5c0052-9517-42b5-a48f-9716080c98fb-bound-sa-token\") pod \"cert-manager-55b4d954d5-4rbhc\" (UID: \"bc5c0052-9517-42b5-a48f-9716080c98fb\") " pod="cert-manager/cert-manager-55b4d954d5-4rbhc" Jul 22 01:41:59.047785 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:59.047730 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4rbhc" Jul 22 01:41:59.170536 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:59.170511 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4rbhc"] Jul 22 01:41:59.172963 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:41:59.172934 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc5c0052_9517_42b5_a48f_9716080c98fb.slice/crio-749fd701bbaf129b7741fcdcf06d843867c1b430304adb33db325263ae26bcf4 WatchSource:0}: Error finding container 749fd701bbaf129b7741fcdcf06d843867c1b430304adb33db325263ae26bcf4: Status 404 returned error can't find the container with id 749fd701bbaf129b7741fcdcf06d843867c1b430304adb33db325263ae26bcf4 Jul 22 01:41:59.192807 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:59.192789 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wlslm" Jul 22 01:41:59.214775 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:41:59.214745 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4rbhc" event={"ID":"bc5c0052-9517-42b5-a48f-9716080c98fb","Type":"ContainerStarted","Data":"749fd701bbaf129b7741fcdcf06d843867c1b430304adb33db325263ae26bcf4"} Jul 22 01:42:00.221088 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:00.221054 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4rbhc" event={"ID":"bc5c0052-9517-42b5-a48f-9716080c98fb","Type":"ContainerStarted","Data":"93630ea7164d35db14024adc45bec7d709df674d521c6995447d1c497b948131"} Jul 22 01:42:00.239155 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:00.239111 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-4rbhc" podStartSLOduration=2.239095586 podStartE2EDuration="2.239095586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:42:00.238235518 +0000 UTC m=+290.831039881" watchObservedRunningTime="2026-07-22 01:42:00.239095586 +0000 UTC m=+290.831899948" Jul 22 01:42:09.554745 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:09.554710 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f"] Jul 22 01:42:09.559946 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:09.559929 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f" Jul 22 01:42:09.562450 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:09.562429 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 01:42:09.563306 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:09.563288 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 01:42:09.563404 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:09.563286 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-snmkh\"" Jul 22 01:42:09.566826 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:09.566804 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f"] Jul 22 01:42:09.692605 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:09.692575 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f43ceae-6bf0-4e87-bb12-e8bb4cde024b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f\" (UID: \"8f43ceae-6bf0-4e87-bb12-e8bb4cde024b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f" Jul 22 01:42:09.692605 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:09.692605 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4vcm\" (UniqueName: \"kubernetes.io/projected/8f43ceae-6bf0-4e87-bb12-e8bb4cde024b-kube-api-access-v4vcm\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f\" (UID: \"8f43ceae-6bf0-4e87-bb12-e8bb4cde024b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f" Jul 22 01:42:09.692825 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:09.692629 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f43ceae-6bf0-4e87-bb12-e8bb4cde024b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f\" (UID: \"8f43ceae-6bf0-4e87-bb12-e8bb4cde024b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f" Jul 22 01:42:09.793078 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:09.793049 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f43ceae-6bf0-4e87-bb12-e8bb4cde024b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f\" (UID: \"8f43ceae-6bf0-4e87-bb12-e8bb4cde024b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f" Jul 22 01:42:09.793078 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:09.793093 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4vcm\" (UniqueName: \"kubernetes.io/projected/8f43ceae-6bf0-4e87-bb12-e8bb4cde024b-kube-api-access-v4vcm\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f\" (UID: \"8f43ceae-6bf0-4e87-bb12-e8bb4cde024b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f" Jul 22 01:42:09.793279 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:09.793131 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f43ceae-6bf0-4e87-bb12-e8bb4cde024b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f\" (UID: \"8f43ceae-6bf0-4e87-bb12-e8bb4cde024b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f" Jul 22 01:42:09.793495 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:09.793467 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f43ceae-6bf0-4e87-bb12-e8bb4cde024b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f\" (UID: \"8f43ceae-6bf0-4e87-bb12-e8bb4cde024b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f" Jul 22 01:42:09.793598 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:09.793540 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f43ceae-6bf0-4e87-bb12-e8bb4cde024b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f\" (UID: \"8f43ceae-6bf0-4e87-bb12-e8bb4cde024b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f" Jul 22 01:42:09.801737 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:09.801689 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 01:42:09.811415 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:09.811358 2561 kubelet.go:1648] "Image garbage collection succeeded" Jul 22 01:42:09.811853 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:09.811834 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 01:42:09.822545 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:09.822522 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4vcm\" (UniqueName: \"kubernetes.io/projected/8f43ceae-6bf0-4e87-bb12-e8bb4cde024b-kube-api-access-v4vcm\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f\" (UID: \"8f43ceae-6bf0-4e87-bb12-e8bb4cde024b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f" Jul 22 01:42:09.872458 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:09.872432 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-snmkh\"" Jul 22 01:42:09.881235 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:09.881221 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f" Jul 22 01:42:10.014087 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:10.014063 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f"] Jul 22 01:42:10.015895 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:42:10.015870 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f43ceae_6bf0_4e87_bb12_e8bb4cde024b.slice/crio-367b3d8d31a8ac20e635debd036a432df93f04cee28848b92f8a58ffdeef229b WatchSource:0}: Error finding container 367b3d8d31a8ac20e635debd036a432df93f04cee28848b92f8a58ffdeef229b: Status 404 returned error can't find the container with id 367b3d8d31a8ac20e635debd036a432df93f04cee28848b92f8a58ffdeef229b Jul 22 01:42:10.267160 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:10.267130 2561 generic.go:356] "Generic (PLEG): container finished" podID="8f43ceae-6bf0-4e87-bb12-e8bb4cde024b" containerID="4a8ae4991a26f49038b9aae2fa272b12c290b6b8a1e61ce433382a83eac49033" exitCode=0 Jul 22 01:42:10.267268 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:10.267169 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f" event={"ID":"8f43ceae-6bf0-4e87-bb12-e8bb4cde024b","Type":"ContainerDied","Data":"4a8ae4991a26f49038b9aae2fa272b12c290b6b8a1e61ce433382a83eac49033"} Jul 22 01:42:10.267268 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:10.267205 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f" event={"ID":"8f43ceae-6bf0-4e87-bb12-e8bb4cde024b","Type":"ContainerStarted","Data":"367b3d8d31a8ac20e635debd036a432df93f04cee28848b92f8a58ffdeef229b"} Jul 22 01:42:12.277225 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:12.277181 2561 generic.go:356] "Generic (PLEG): container finished" podID="8f43ceae-6bf0-4e87-bb12-e8bb4cde024b" containerID="2e76b386815c5ce05d449a3c3cd3dd394a52566e9db2a0f880d53838ecf2a4e9" exitCode=0 Jul 22 01:42:12.277607 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:12.277261 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f" event={"ID":"8f43ceae-6bf0-4e87-bb12-e8bb4cde024b","Type":"ContainerDied","Data":"2e76b386815c5ce05d449a3c3cd3dd394a52566e9db2a0f880d53838ecf2a4e9"} Jul 22 01:42:13.284101 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:13.284065 2561 generic.go:356] "Generic (PLEG): container finished" podID="8f43ceae-6bf0-4e87-bb12-e8bb4cde024b" containerID="e488a323ebd02c631a1974b936b2129270fd11cee8122f1cd8ac78523df209b5" exitCode=0 Jul 22 01:42:13.284473 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:13.284147 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f" event={"ID":"8f43ceae-6bf0-4e87-bb12-e8bb4cde024b","Type":"ContainerDied","Data":"e488a323ebd02c631a1974b936b2129270fd11cee8122f1cd8ac78523df209b5"} Jul 22 01:42:14.412674 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:14.412651 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f" Jul 22 01:42:14.533724 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:14.533678 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f43ceae-6bf0-4e87-bb12-e8bb4cde024b-util\") pod \"8f43ceae-6bf0-4e87-bb12-e8bb4cde024b\" (UID: \"8f43ceae-6bf0-4e87-bb12-e8bb4cde024b\") " Jul 22 01:42:14.533896 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:14.533757 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4vcm\" (UniqueName: \"kubernetes.io/projected/8f43ceae-6bf0-4e87-bb12-e8bb4cde024b-kube-api-access-v4vcm\") pod \"8f43ceae-6bf0-4e87-bb12-e8bb4cde024b\" (UID: \"8f43ceae-6bf0-4e87-bb12-e8bb4cde024b\") " Jul 22 01:42:14.533896 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:14.533792 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f43ceae-6bf0-4e87-bb12-e8bb4cde024b-bundle\") pod \"8f43ceae-6bf0-4e87-bb12-e8bb4cde024b\" (UID: \"8f43ceae-6bf0-4e87-bb12-e8bb4cde024b\") " Jul 22 01:42:14.537067 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:14.535061 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f43ceae-6bf0-4e87-bb12-e8bb4cde024b-bundle" (OuterVolumeSpecName: "bundle") pod "8f43ceae-6bf0-4e87-bb12-e8bb4cde024b" (UID: "8f43ceae-6bf0-4e87-bb12-e8bb4cde024b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:42:14.541181 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:14.541096 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f43ceae-6bf0-4e87-bb12-e8bb4cde024b-kube-api-access-v4vcm" (OuterVolumeSpecName: "kube-api-access-v4vcm") pod "8f43ceae-6bf0-4e87-bb12-e8bb4cde024b" (UID: "8f43ceae-6bf0-4e87-bb12-e8bb4cde024b"). InnerVolumeSpecName "kube-api-access-v4vcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:42:14.542263 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:14.542225 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f43ceae-6bf0-4e87-bb12-e8bb4cde024b-util" (OuterVolumeSpecName: "util") pod "8f43ceae-6bf0-4e87-bb12-e8bb4cde024b" (UID: "8f43ceae-6bf0-4e87-bb12-e8bb4cde024b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:42:14.635618 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:14.635582 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4vcm\" (UniqueName: \"kubernetes.io/projected/8f43ceae-6bf0-4e87-bb12-e8bb4cde024b-kube-api-access-v4vcm\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:42:14.635618 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:14.635609 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f43ceae-6bf0-4e87-bb12-e8bb4cde024b-bundle\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:42:14.635618 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:14.635619 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f43ceae-6bf0-4e87-bb12-e8bb4cde024b-util\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:42:15.294768 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:15.294734 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f" event={"ID":"8f43ceae-6bf0-4e87-bb12-e8bb4cde024b","Type":"ContainerDied","Data":"367b3d8d31a8ac20e635debd036a432df93f04cee28848b92f8a58ffdeef229b"} Jul 22 01:42:15.294768 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:15.294765 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvs54f" Jul 22 01:42:15.294966 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:15.294779 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="367b3d8d31a8ac20e635debd036a432df93f04cee28848b92f8a58ffdeef229b" Jul 22 01:42:26.516920 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.516888 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c578697c6-lfq9r"] Jul 22 01:42:26.517393 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.517201 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f43ceae-6bf0-4e87-bb12-e8bb4cde024b" containerName="extract" Jul 22 01:42:26.517393 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.517212 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f43ceae-6bf0-4e87-bb12-e8bb4cde024b" containerName="extract" Jul 22 01:42:26.517393 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.517238 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f43ceae-6bf0-4e87-bb12-e8bb4cde024b" containerName="util" Jul 22 01:42:26.517393 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.517243 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f43ceae-6bf0-4e87-bb12-e8bb4cde024b" containerName="util" Jul 22 01:42:26.517393 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.517249 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f43ceae-6bf0-4e87-bb12-e8bb4cde024b" containerName="pull" Jul 22 01:42:26.517393 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.517254 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f43ceae-6bf0-4e87-bb12-e8bb4cde024b" containerName="pull" Jul 22 01:42:26.517393 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.517307 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f43ceae-6bf0-4e87-bb12-e8bb4cde024b" containerName="extract" Jul 22 01:42:26.520022 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.520004 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-lfq9r" Jul 22 01:42:26.522457 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.522431 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 22 01:42:26.522554 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.522444 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 22 01:42:26.522554 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.522520 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-6cfrp\"" Jul 22 01:42:26.524471 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.523233 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 22 01:42:26.524471 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.523278 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 22 01:42:26.535708 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.535673 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c578697c6-lfq9r"] Jul 22 01:42:26.626339 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.626304 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lggs\" (UniqueName: \"kubernetes.io/projected/7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135-kube-api-access-6lggs\") pod \"opendatahub-operator-controller-manager-5c578697c6-lfq9r\" (UID: \"7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-lfq9r" Jul 22 01:42:26.626473 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.626367 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c578697c6-lfq9r\" (UID: \"7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-lfq9r" Jul 22 01:42:26.626473 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.626447 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c578697c6-lfq9r\" (UID: \"7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-lfq9r" Jul 22 01:42:26.626569 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.626484 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135-manifests\") pod \"opendatahub-operator-controller-manager-5c578697c6-lfq9r\" (UID: \"7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-lfq9r" Jul 22 01:42:26.626569 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.626507 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135-tmp\") pod \"opendatahub-operator-controller-manager-5c578697c6-lfq9r\" (UID: \"7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-lfq9r" Jul 22 01:42:26.727741 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.727710 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c578697c6-lfq9r\" (UID: \"7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-lfq9r" Jul 22 01:42:26.727851 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.727777 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c578697c6-lfq9r\" (UID: \"7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-lfq9r" Jul 22 01:42:26.727851 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.727815 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135-manifests\") pod \"opendatahub-operator-controller-manager-5c578697c6-lfq9r\" (UID: \"7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-lfq9r" Jul 22 01:42:26.727851 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.727838 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135-tmp\") pod \"opendatahub-operator-controller-manager-5c578697c6-lfq9r\" (UID: \"7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-lfq9r" Jul 22 01:42:26.728005 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.727869 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lggs\" (UniqueName: \"kubernetes.io/projected/7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135-kube-api-access-6lggs\") pod \"opendatahub-operator-controller-manager-5c578697c6-lfq9r\" (UID: \"7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-lfq9r" Jul 22 01:42:26.728284 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.728257 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135-manifests\") pod \"opendatahub-operator-controller-manager-5c578697c6-lfq9r\" (UID: \"7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-lfq9r" Jul 22 01:42:26.730117 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.730097 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135-tmp\") pod \"opendatahub-operator-controller-manager-5c578697c6-lfq9r\" (UID: \"7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-lfq9r" Jul 22 01:42:26.730421 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.730404 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c578697c6-lfq9r\" (UID: \"7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-lfq9r" Jul 22 01:42:26.730465 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.730430 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c578697c6-lfq9r\" (UID: \"7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-lfq9r" Jul 22 01:42:26.736617 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.736593 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lggs\" (UniqueName: \"kubernetes.io/projected/7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135-kube-api-access-6lggs\") pod \"opendatahub-operator-controller-manager-5c578697c6-lfq9r\" (UID: \"7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135\") " pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-lfq9r" Jul 22 01:42:26.835600 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.835546 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-lfq9r" Jul 22 01:42:26.998817 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:26.998761 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c578697c6-lfq9r"] Jul 22 01:42:26.999579 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:42:26.999549 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2fecd4_5e9c_4d1a_a66e_65e2ffa3c135.slice/crio-90fed4acb4555a5853605d0e9062f55c00e9e10308bc35b2dff939975c331c8a WatchSource:0}: Error finding container 90fed4acb4555a5853605d0e9062f55c00e9e10308bc35b2dff939975c331c8a: Status 404 returned error can't find the container with id 90fed4acb4555a5853605d0e9062f55c00e9e10308bc35b2dff939975c331c8a Jul 22 01:42:27.001341 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:27.001325 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 01:42:27.349374 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:27.349335 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-lfq9r" event={"ID":"7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135","Type":"ContainerStarted","Data":"90fed4acb4555a5853605d0e9062f55c00e9e10308bc35b2dff939975c331c8a"} Jul 22 01:42:30.366003 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:30.365978 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-lfq9r" event={"ID":"7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135","Type":"ContainerStarted","Data":"d358f88431477ad39476c6061aedfdc7d6088792426c0268dec25078a144d429"} Jul 22 01:42:31.372604 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:31.372571 2561 generic.go:356] "Generic (PLEG): container finished" podID="7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135" containerID="d358f88431477ad39476c6061aedfdc7d6088792426c0268dec25078a144d429" exitCode=0 Jul 22 01:42:31.372933 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:31.372656 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-lfq9r" event={"ID":"7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135","Type":"ContainerDied","Data":"d358f88431477ad39476c6061aedfdc7d6088792426c0268dec25078a144d429"} Jul 22 01:42:32.381056 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:32.381018 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-lfq9r" event={"ID":"7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135","Type":"ContainerStarted","Data":"590325c7b935b20089f9ad1e3412ec67535dc7a4e6286469792c2b1964c9501d"} Jul 22 01:42:32.381456 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:32.381224 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-lfq9r" Jul 22 01:42:32.405286 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:32.405174 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-lfq9r" podStartSLOduration=1.363197291 podStartE2EDuration="6.405076086s" totalImagesPullingTime="4.03704944s" totalInitContainerRuntime="1.004829355s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:42:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:42:32.403538495 +0000 UTC m=+322.996342858" watchObservedRunningTime="2026-07-22 01:42:32.405076086 +0000 UTC m=+322.997880450" Jul 22 01:42:43.387904 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:43.387871 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5c578697c6-lfq9r" Jul 22 01:42:46.476870 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.476835 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5975cd7b94-4zh5g"] Jul 22 01:42:46.487179 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.487150 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5975cd7b94-4zh5g" Jul 22 01:42:46.491774 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.491747 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 22 01:42:46.491893 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.491847 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-h6x54\"" Jul 22 01:42:46.492120 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.492107 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 22 01:42:46.492290 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.492272 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 22 01:42:46.492351 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.492323 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 22 01:42:46.502716 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.501822 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5975cd7b94-4zh5g"] Jul 22 01:42:46.592224 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.592198 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mqdp\" (UniqueName: \"kubernetes.io/projected/32e6dba1-d57f-4e7c-bed0-2264b16668ea-kube-api-access-2mqdp\") pod \"kube-auth-proxy-5975cd7b94-4zh5g\" (UID: \"32e6dba1-d57f-4e7c-bed0-2264b16668ea\") " pod="openshift-ingress/kube-auth-proxy-5975cd7b94-4zh5g" Jul 22 01:42:46.592377 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.592263 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e6dba1-d57f-4e7c-bed0-2264b16668ea-tls-certs\") pod \"kube-auth-proxy-5975cd7b94-4zh5g\" (UID: \"32e6dba1-d57f-4e7c-bed0-2264b16668ea\") " pod="openshift-ingress/kube-auth-proxy-5975cd7b94-4zh5g" Jul 22 01:42:46.592377 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.592296 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32e6dba1-d57f-4e7c-bed0-2264b16668ea-tmp\") pod \"kube-auth-proxy-5975cd7b94-4zh5g\" (UID: \"32e6dba1-d57f-4e7c-bed0-2264b16668ea\") " pod="openshift-ingress/kube-auth-proxy-5975cd7b94-4zh5g" Jul 22 01:42:46.693564 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.693528 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e6dba1-d57f-4e7c-bed0-2264b16668ea-tls-certs\") pod \"kube-auth-proxy-5975cd7b94-4zh5g\" (UID: \"32e6dba1-d57f-4e7c-bed0-2264b16668ea\") " pod="openshift-ingress/kube-auth-proxy-5975cd7b94-4zh5g" Jul 22 01:42:46.693769 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.693585 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32e6dba1-d57f-4e7c-bed0-2264b16668ea-tmp\") pod \"kube-auth-proxy-5975cd7b94-4zh5g\" (UID: \"32e6dba1-d57f-4e7c-bed0-2264b16668ea\") " pod="openshift-ingress/kube-auth-proxy-5975cd7b94-4zh5g" Jul 22 01:42:46.693769 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.693647 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mqdp\" (UniqueName: \"kubernetes.io/projected/32e6dba1-d57f-4e7c-bed0-2264b16668ea-kube-api-access-2mqdp\") pod \"kube-auth-proxy-5975cd7b94-4zh5g\" (UID: \"32e6dba1-d57f-4e7c-bed0-2264b16668ea\") " pod="openshift-ingress/kube-auth-proxy-5975cd7b94-4zh5g" Jul 22 01:42:46.695712 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.695670 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-f724l"] Jul 22 01:42:46.696441 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.696413 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32e6dba1-d57f-4e7c-bed0-2264b16668ea-tmp\") pod \"kube-auth-proxy-5975cd7b94-4zh5g\" (UID: \"32e6dba1-d57f-4e7c-bed0-2264b16668ea\") " pod="openshift-ingress/kube-auth-proxy-5975cd7b94-4zh5g" Jul 22 01:42:46.696630 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.696607 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e6dba1-d57f-4e7c-bed0-2264b16668ea-tls-certs\") pod \"kube-auth-proxy-5975cd7b94-4zh5g\" (UID: \"32e6dba1-d57f-4e7c-bed0-2264b16668ea\") " pod="openshift-ingress/kube-auth-proxy-5975cd7b94-4zh5g" Jul 22 01:42:46.698282 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.698268 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-f724l" Jul 22 01:42:46.702165 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.702145 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 22 01:42:46.702441 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.702416 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-5m7xh\"" Jul 22 01:42:46.709737 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.709685 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mqdp\" (UniqueName: \"kubernetes.io/projected/32e6dba1-d57f-4e7c-bed0-2264b16668ea-kube-api-access-2mqdp\") pod \"kube-auth-proxy-5975cd7b94-4zh5g\" (UID: \"32e6dba1-d57f-4e7c-bed0-2264b16668ea\") " pod="openshift-ingress/kube-auth-proxy-5975cd7b94-4zh5g" Jul 22 01:42:46.712012 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.711990 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-f724l"] Jul 22 01:42:46.797571 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.797491 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5975cd7b94-4zh5g" Jul 22 01:42:46.895309 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.895280 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/233fa350-ced9-4ddb-8ddd-9aa88e1d32d1-cert\") pod \"odh-model-controller-7998c67dc4-f724l\" (UID: \"233fa350-ced9-4ddb-8ddd-9aa88e1d32d1\") " pod="opendatahub/odh-model-controller-7998c67dc4-f724l" Jul 22 01:42:46.895459 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.895331 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skgp2\" (UniqueName: \"kubernetes.io/projected/233fa350-ced9-4ddb-8ddd-9aa88e1d32d1-kube-api-access-skgp2\") pod \"odh-model-controller-7998c67dc4-f724l\" (UID: \"233fa350-ced9-4ddb-8ddd-9aa88e1d32d1\") " pod="opendatahub/odh-model-controller-7998c67dc4-f724l" Jul 22 01:42:46.952948 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.952917 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5975cd7b94-4zh5g"] Jul 22 01:42:46.955176 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:42:46.955148 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e6dba1_d57f_4e7c_bed0_2264b16668ea.slice/crio-0a16976ebd049010c9ad674445f46880b3e9ac6dcaa15d3ac55e0dac2dd0db46 WatchSource:0}: Error finding container 0a16976ebd049010c9ad674445f46880b3e9ac6dcaa15d3ac55e0dac2dd0db46: Status 404 returned error can't find the container with id 0a16976ebd049010c9ad674445f46880b3e9ac6dcaa15d3ac55e0dac2dd0db46 Jul 22 01:42:46.996084 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.996052 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/233fa350-ced9-4ddb-8ddd-9aa88e1d32d1-cert\") pod \"odh-model-controller-7998c67dc4-f724l\" (UID: \"233fa350-ced9-4ddb-8ddd-9aa88e1d32d1\") " pod="opendatahub/odh-model-controller-7998c67dc4-f724l" Jul 22 01:42:46.996239 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.996097 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skgp2\" (UniqueName: \"kubernetes.io/projected/233fa350-ced9-4ddb-8ddd-9aa88e1d32d1-kube-api-access-skgp2\") pod \"odh-model-controller-7998c67dc4-f724l\" (UID: \"233fa350-ced9-4ddb-8ddd-9aa88e1d32d1\") " pod="opendatahub/odh-model-controller-7998c67dc4-f724l" Jul 22 01:42:46.998496 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:46.998473 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/233fa350-ced9-4ddb-8ddd-9aa88e1d32d1-cert\") pod \"odh-model-controller-7998c67dc4-f724l\" (UID: \"233fa350-ced9-4ddb-8ddd-9aa88e1d32d1\") " pod="opendatahub/odh-model-controller-7998c67dc4-f724l" Jul 22 01:42:47.011585 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:47.011561 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skgp2\" (UniqueName: \"kubernetes.io/projected/233fa350-ced9-4ddb-8ddd-9aa88e1d32d1-kube-api-access-skgp2\") pod \"odh-model-controller-7998c67dc4-f724l\" (UID: \"233fa350-ced9-4ddb-8ddd-9aa88e1d32d1\") " pod="opendatahub/odh-model-controller-7998c67dc4-f724l" Jul 22 01:42:47.026555 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:47.026531 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-f724l" Jul 22 01:42:47.170984 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:47.170957 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-f724l"] Jul 22 01:42:47.173917 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:42:47.173883 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod233fa350_ced9_4ddb_8ddd_9aa88e1d32d1.slice/crio-c5380131eab2b0e9856f93904c3f1b6a532a3bba0bdc494c53650b8a41ad9f05 WatchSource:0}: Error finding container c5380131eab2b0e9856f93904c3f1b6a532a3bba0bdc494c53650b8a41ad9f05: Status 404 returned error can't find the container with id c5380131eab2b0e9856f93904c3f1b6a532a3bba0bdc494c53650b8a41ad9f05 Jul 22 01:42:47.450862 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:47.450820 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-f724l" event={"ID":"233fa350-ced9-4ddb-8ddd-9aa88e1d32d1","Type":"ContainerStarted","Data":"c5380131eab2b0e9856f93904c3f1b6a532a3bba0bdc494c53650b8a41ad9f05"} Jul 22 01:42:47.452037 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:47.452003 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5975cd7b94-4zh5g" event={"ID":"32e6dba1-d57f-4e7c-bed0-2264b16668ea","Type":"ContainerStarted","Data":"0a16976ebd049010c9ad674445f46880b3e9ac6dcaa15d3ac55e0dac2dd0db46"} Jul 22 01:42:47.547651 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:47.547616 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pnbqj"] Jul 22 01:42:47.550353 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:47.550331 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-pnbqj" Jul 22 01:42:47.553264 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:47.553237 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-5nhcr\"" Jul 22 01:42:47.560028 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:47.560007 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 22 01:42:47.578643 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:47.578616 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pnbqj"] Jul 22 01:42:47.600611 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:47.600585 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5785\" (UniqueName: \"kubernetes.io/projected/7fc9fa98-e5a6-4662-8a43-97c9373f5938-kube-api-access-n5785\") pod \"model-serving-api-54d89bd5fc-pnbqj\" (UID: \"7fc9fa98-e5a6-4662-8a43-97c9373f5938\") " pod="opendatahub/model-serving-api-54d89bd5fc-pnbqj" Jul 22 01:42:47.600762 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:47.600648 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fc9fa98-e5a6-4662-8a43-97c9373f5938-tls-certs\") pod \"model-serving-api-54d89bd5fc-pnbqj\" (UID: \"7fc9fa98-e5a6-4662-8a43-97c9373f5938\") " pod="opendatahub/model-serving-api-54d89bd5fc-pnbqj" Jul 22 01:42:47.701358 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:47.701277 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fc9fa98-e5a6-4662-8a43-97c9373f5938-tls-certs\") pod \"model-serving-api-54d89bd5fc-pnbqj\" (UID: \"7fc9fa98-e5a6-4662-8a43-97c9373f5938\") " pod="opendatahub/model-serving-api-54d89bd5fc-pnbqj" Jul 22 01:42:47.701502 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:47.701376 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5785\" (UniqueName: \"kubernetes.io/projected/7fc9fa98-e5a6-4662-8a43-97c9373f5938-kube-api-access-n5785\") pod \"model-serving-api-54d89bd5fc-pnbqj\" (UID: \"7fc9fa98-e5a6-4662-8a43-97c9373f5938\") " pod="opendatahub/model-serving-api-54d89bd5fc-pnbqj" Jul 22 01:42:47.701502 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:42:47.701450 2561 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 22 01:42:47.701600 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:42:47.701552 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fc9fa98-e5a6-4662-8a43-97c9373f5938-tls-certs podName:7fc9fa98-e5a6-4662-8a43-97c9373f5938 nodeName:}" failed. No retries permitted until 2026-07-22 01:42:48.201527561 +0000 UTC m=+338.794331903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7fc9fa98-e5a6-4662-8a43-97c9373f5938-tls-certs") pod "model-serving-api-54d89bd5fc-pnbqj" (UID: "7fc9fa98-e5a6-4662-8a43-97c9373f5938") : secret "model-serving-api-tls" not found Jul 22 01:42:47.717542 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:47.717514 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5785\" (UniqueName: \"kubernetes.io/projected/7fc9fa98-e5a6-4662-8a43-97c9373f5938-kube-api-access-n5785\") pod \"model-serving-api-54d89bd5fc-pnbqj\" (UID: \"7fc9fa98-e5a6-4662-8a43-97c9373f5938\") " pod="opendatahub/model-serving-api-54d89bd5fc-pnbqj" Jul 22 01:42:48.079315 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.079232 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz"] Jul 22 01:42:48.082720 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.082674 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" Jul 22 01:42:48.090514 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.089969 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 22 01:42:48.090514 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.090003 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-24kgn\"" Jul 22 01:42:48.090514 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.090176 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 22 01:42:48.090514 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.090232 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 22 01:42:48.090514 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.090343 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 22 01:42:48.095547 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.095326 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz"] Jul 22 01:42:48.105583 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.105554 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rtk4\" (UniqueName: \"kubernetes.io/projected/0c9100fd-2b91-4fc5-b652-f2b706586c9d-kube-api-access-9rtk4\") pod \"istiod-openshift-gateway-7574459656-9hgpz\" (UID: \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" Jul 22 01:42:48.105786 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.105665 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0c9100fd-2b91-4fc5-b652-f2b706586c9d-cacerts\") pod \"istiod-openshift-gateway-7574459656-9hgpz\" (UID: \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" Jul 22 01:42:48.105872 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.105793 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9100fd-2b91-4fc5-b652-f2b706586c9d-istio-token\") pod \"istiod-openshift-gateway-7574459656-9hgpz\" (UID: \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" Jul 22 01:42:48.105872 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.105830 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9100fd-2b91-4fc5-b652-f2b706586c9d-local-certs\") pod \"istiod-openshift-gateway-7574459656-9hgpz\" (UID: \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" Jul 22 01:42:48.105872 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.105855 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c9100fd-2b91-4fc5-b652-f2b706586c9d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-9hgpz\" (UID: \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" Jul 22 01:42:48.106022 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.105893 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c9100fd-2b91-4fc5-b652-f2b706586c9d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-9hgpz\" (UID: \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" Jul 22 01:42:48.106022 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.105939 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c9100fd-2b91-4fc5-b652-f2b706586c9d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-9hgpz\" (UID: \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" Jul 22 01:42:48.207341 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.207300 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9100fd-2b91-4fc5-b652-f2b706586c9d-istio-token\") pod \"istiod-openshift-gateway-7574459656-9hgpz\" (UID: \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" Jul 22 01:42:48.207540 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.207359 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9100fd-2b91-4fc5-b652-f2b706586c9d-local-certs\") pod \"istiod-openshift-gateway-7574459656-9hgpz\" (UID: \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" Jul 22 01:42:48.207540 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.207387 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c9100fd-2b91-4fc5-b652-f2b706586c9d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-9hgpz\" (UID: \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" Jul 22 01:42:48.207540 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.207440 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c9100fd-2b91-4fc5-b652-f2b706586c9d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-9hgpz\" (UID: \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" Jul 22 01:42:48.207540 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.207480 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c9100fd-2b91-4fc5-b652-f2b706586c9d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-9hgpz\" (UID: \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" Jul 22 01:42:48.207540 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.207533 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rtk4\" (UniqueName: \"kubernetes.io/projected/0c9100fd-2b91-4fc5-b652-f2b706586c9d-kube-api-access-9rtk4\") pod \"istiod-openshift-gateway-7574459656-9hgpz\" (UID: \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" Jul 22 01:42:48.207791 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.207580 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fc9fa98-e5a6-4662-8a43-97c9373f5938-tls-certs\") pod \"model-serving-api-54d89bd5fc-pnbqj\" (UID: \"7fc9fa98-e5a6-4662-8a43-97c9373f5938\") " pod="opendatahub/model-serving-api-54d89bd5fc-pnbqj" Jul 22 01:42:48.207791 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.207608 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0c9100fd-2b91-4fc5-b652-f2b706586c9d-cacerts\") pod \"istiod-openshift-gateway-7574459656-9hgpz\" (UID: \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" Jul 22 01:42:48.208960 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.208927 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c9100fd-2b91-4fc5-b652-f2b706586c9d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-9hgpz\" (UID: \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" Jul 22 01:42:48.211328 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.211273 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0c9100fd-2b91-4fc5-b652-f2b706586c9d-cacerts\") pod \"istiod-openshift-gateway-7574459656-9hgpz\" (UID: \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" Jul 22 01:42:48.211449 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.211356 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fc9fa98-e5a6-4662-8a43-97c9373f5938-tls-certs\") pod \"model-serving-api-54d89bd5fc-pnbqj\" (UID: \"7fc9fa98-e5a6-4662-8a43-97c9373f5938\") " pod="opendatahub/model-serving-api-54d89bd5fc-pnbqj" Jul 22 01:42:48.213559 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.213518 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c9100fd-2b91-4fc5-b652-f2b706586c9d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-9hgpz\" (UID: \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" Jul 22 01:42:48.214127 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.214065 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9100fd-2b91-4fc5-b652-f2b706586c9d-local-certs\") pod \"istiod-openshift-gateway-7574459656-9hgpz\" (UID: \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" Jul 22 01:42:48.215078 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.215030 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c9100fd-2b91-4fc5-b652-f2b706586c9d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-9hgpz\" (UID: \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" Jul 22 01:42:48.218023 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.217974 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rtk4\" (UniqueName: \"kubernetes.io/projected/0c9100fd-2b91-4fc5-b652-f2b706586c9d-kube-api-access-9rtk4\") pod \"istiod-openshift-gateway-7574459656-9hgpz\" (UID: \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" Jul 22 01:42:48.219662 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.219640 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9100fd-2b91-4fc5-b652-f2b706586c9d-istio-token\") pod \"istiod-openshift-gateway-7574459656-9hgpz\" (UID: \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" Jul 22 01:42:48.400459 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.400365 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" Jul 22 01:42:48.464286 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.464253 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-pnbqj" Jul 22 01:42:48.596066 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.596037 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz"] Jul 22 01:42:48.599072 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:42:48.599042 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9100fd_2b91_4fc5_b652_f2b706586c9d.slice/crio-27b9a880cec8c5c17e126f086f1305c137f533a73ac82537dce916b42ae3ed15 WatchSource:0}: Error finding container 27b9a880cec8c5c17e126f086f1305c137f533a73ac82537dce916b42ae3ed15: Status 404 returned error can't find the container with id 27b9a880cec8c5c17e126f086f1305c137f533a73ac82537dce916b42ae3ed15 Jul 22 01:42:48.671840 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:48.670594 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pnbqj"] Jul 22 01:42:48.677362 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:42:48.677228 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc9fa98_e5a6_4662_8a43_97c9373f5938.slice/crio-6194c00ef35611775ce0d4e6f69f6392ea4b5610633d8ead02802c76113363df WatchSource:0}: Error finding container 6194c00ef35611775ce0d4e6f69f6392ea4b5610633d8ead02802c76113363df: Status 404 returned error can't find the container with id 6194c00ef35611775ce0d4e6f69f6392ea4b5610633d8ead02802c76113363df Jul 22 01:42:49.471013 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:49.470972 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-pnbqj" event={"ID":"7fc9fa98-e5a6-4662-8a43-97c9373f5938","Type":"ContainerStarted","Data":"6194c00ef35611775ce0d4e6f69f6392ea4b5610633d8ead02802c76113363df"} Jul 22 01:42:49.472571 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:49.472529 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" event={"ID":"0c9100fd-2b91-4fc5-b652-f2b706586c9d","Type":"ContainerStarted","Data":"27b9a880cec8c5c17e126f086f1305c137f533a73ac82537dce916b42ae3ed15"} Jul 22 01:42:52.718854 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:52.718818 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 22 01:42:52.719202 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:52.718901 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 22 01:42:53.498168 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:53.498120 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" event={"ID":"0c9100fd-2b91-4fc5-b652-f2b706586c9d","Type":"ContainerStarted","Data":"66da2ec27f96ebcf7011ef10e8776bb393c55c8b86a539061157b91fa063ab50"} Jul 22 01:42:53.498377 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:53.498273 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" Jul 22 01:42:53.499888 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:53.499856 2561 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-9hgpz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 01:42:53.499888 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:53.499872 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-f724l" event={"ID":"233fa350-ced9-4ddb-8ddd-9aa88e1d32d1","Type":"ContainerStarted","Data":"702438118fbcc7893d42c06a78d2041cd07519d1ce3558444df2ae3ff5b53b49"} Jul 22 01:42:53.500070 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:53.499908 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" podUID="0c9100fd-2b91-4fc5-b652-f2b706586c9d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 01:42:53.500070 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:53.500001 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-f724l" Jul 22 01:42:53.501514 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:53.501490 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5975cd7b94-4zh5g" event={"ID":"32e6dba1-d57f-4e7c-bed0-2264b16668ea","Type":"ContainerStarted","Data":"948b3b9be33eef5e1b95b48daa99f992c66a4fc5eb10c4b62878b69214e017ab"} Jul 22 01:42:53.503146 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:53.503124 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-pnbqj" event={"ID":"7fc9fa98-e5a6-4662-8a43-97c9373f5938","Type":"ContainerStarted","Data":"3b45989e17c57f0e22a715bec2840bafaa1591791110961cc9888d19977669a9"} Jul 22 01:42:53.503268 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:53.503251 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-pnbqj" Jul 22 01:42:53.517407 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:53.517362 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" podStartSLOduration=1.402063598 podStartE2EDuration="5.517344315s" totalImagesPullingTime="4.115280717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:42:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:42:53.516416766 +0000 UTC m=+344.109221147" watchObservedRunningTime="2026-07-22 01:42:53.517344315 +0000 UTC m=+344.110148684" Jul 22 01:42:53.533027 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:53.532982 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5975cd7b94-4zh5g" podStartSLOduration=1.840049609 podStartE2EDuration="7.532970249s" totalImagesPullingTime="5.69292064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:42:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:42:53.531670068 +0000 UTC m=+344.124474467" watchObservedRunningTime="2026-07-22 01:42:53.532970249 +0000 UTC m=+344.125774655" Jul 22 01:42:53.552238 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:53.552188 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-pnbqj" podStartSLOduration=2.573398647 podStartE2EDuration="6.552171314s" totalImagesPullingTime="3.978772667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:42:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:42:53.551451562 +0000 UTC m=+344.144255929" watchObservedRunningTime="2026-07-22 01:42:53.552171314 +0000 UTC m=+344.144975678" Jul 22 01:42:53.569885 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:53.569847 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-f724l" podStartSLOduration=2.095272098 podStartE2EDuration="7.569832412s" totalImagesPullingTime="5.474560314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:42:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:42:53.567618614 +0000 UTC m=+344.160423028" watchObservedRunningTime="2026-07-22 01:42:53.569832412 +0000 UTC m=+344.162636775" Jul 22 01:42:54.513261 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:54.513234 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" Jul 22 01:42:57.781135 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:57.781103 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-t4pkb"] Jul 22 01:42:57.784680 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:57.784660 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t4pkb" Jul 22 01:42:57.786855 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:57.786830 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 22 01:42:57.787822 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:57.787800 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-c6jjp\"" Jul 22 01:42:57.789143 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:57.789124 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-j9twf"] Jul 22 01:42:57.792436 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:57.792382 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-j9twf" Jul 22 01:42:57.794464 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:57.794445 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 22 01:42:57.794535 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:57.794486 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-5j6kg\"" Jul 22 01:42:57.802595 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:57.802575 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-t4pkb"] Jul 22 01:42:57.819067 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:57.819044 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-j9twf"] Jul 22 01:42:57.905047 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:57.905023 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt666\" (UniqueName: \"kubernetes.io/projected/17416ded-8f9b-4822-8800-d65e3bf062b3-kube-api-access-dt666\") pod \"llmisvc-controller-manager-777c6bd7d5-j9twf\" (UID: \"17416ded-8f9b-4822-8800-d65e3bf062b3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-j9twf" Jul 22 01:42:57.905174 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:57.905141 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b563dcf-e028-4eaa-b33c-164306146411-cert\") pod \"kserve-controller-manager-d6f4b65b8-t4pkb\" (UID: \"8b563dcf-e028-4eaa-b33c-164306146411\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t4pkb" Jul 22 01:42:57.905229 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:57.905168 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bqcf\" (UniqueName: \"kubernetes.io/projected/8b563dcf-e028-4eaa-b33c-164306146411-kube-api-access-7bqcf\") pod \"kserve-controller-manager-d6f4b65b8-t4pkb\" (UID: \"8b563dcf-e028-4eaa-b33c-164306146411\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t4pkb" Jul 22 01:42:57.905229 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:57.905217 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17416ded-8f9b-4822-8800-d65e3bf062b3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-j9twf\" (UID: \"17416ded-8f9b-4822-8800-d65e3bf062b3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-j9twf" Jul 22 01:42:58.006047 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:58.006015 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt666\" (UniqueName: \"kubernetes.io/projected/17416ded-8f9b-4822-8800-d65e3bf062b3-kube-api-access-dt666\") pod \"llmisvc-controller-manager-777c6bd7d5-j9twf\" (UID: \"17416ded-8f9b-4822-8800-d65e3bf062b3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-j9twf" Jul 22 01:42:58.006182 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:58.006095 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b563dcf-e028-4eaa-b33c-164306146411-cert\") pod \"kserve-controller-manager-d6f4b65b8-t4pkb\" (UID: \"8b563dcf-e028-4eaa-b33c-164306146411\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t4pkb" Jul 22 01:42:58.006182 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:58.006117 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bqcf\" (UniqueName: \"kubernetes.io/projected/8b563dcf-e028-4eaa-b33c-164306146411-kube-api-access-7bqcf\") pod \"kserve-controller-manager-d6f4b65b8-t4pkb\" (UID: \"8b563dcf-e028-4eaa-b33c-164306146411\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t4pkb" Jul 22 01:42:58.006182 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:58.006153 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17416ded-8f9b-4822-8800-d65e3bf062b3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-j9twf\" (UID: \"17416ded-8f9b-4822-8800-d65e3bf062b3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-j9twf" Jul 22 01:42:58.006312 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:42:58.006179 2561 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 22 01:42:58.006312 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:42:58.006243 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b563dcf-e028-4eaa-b33c-164306146411-cert podName:8b563dcf-e028-4eaa-b33c-164306146411 nodeName:}" failed. No retries permitted until 2026-07-22 01:42:58.506223942 +0000 UTC m=+349.099028288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8b563dcf-e028-4eaa-b33c-164306146411-cert") pod "kserve-controller-manager-d6f4b65b8-t4pkb" (UID: "8b563dcf-e028-4eaa-b33c-164306146411") : secret "kserve-webhook-server-cert" not found Jul 22 01:42:58.006312 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:42:58.006257 2561 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 22 01:42:58.006456 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:42:58.006319 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17416ded-8f9b-4822-8800-d65e3bf062b3-cert podName:17416ded-8f9b-4822-8800-d65e3bf062b3 nodeName:}" failed. No retries permitted until 2026-07-22 01:42:58.506305907 +0000 UTC m=+349.099110247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/17416ded-8f9b-4822-8800-d65e3bf062b3-cert") pod "llmisvc-controller-manager-777c6bd7d5-j9twf" (UID: "17416ded-8f9b-4822-8800-d65e3bf062b3") : secret "llmisvc-webhook-server-cert" not found Jul 22 01:42:58.031807 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:58.031746 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt666\" (UniqueName: \"kubernetes.io/projected/17416ded-8f9b-4822-8800-d65e3bf062b3-kube-api-access-dt666\") pod \"llmisvc-controller-manager-777c6bd7d5-j9twf\" (UID: \"17416ded-8f9b-4822-8800-d65e3bf062b3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-j9twf" Jul 22 01:42:58.034134 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:58.034111 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bqcf\" (UniqueName: \"kubernetes.io/projected/8b563dcf-e028-4eaa-b33c-164306146411-kube-api-access-7bqcf\") pod \"kserve-controller-manager-d6f4b65b8-t4pkb\" (UID: \"8b563dcf-e028-4eaa-b33c-164306146411\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t4pkb" Jul 22 01:42:58.510997 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:58.510950 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17416ded-8f9b-4822-8800-d65e3bf062b3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-j9twf\" (UID: \"17416ded-8f9b-4822-8800-d65e3bf062b3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-j9twf" Jul 22 01:42:58.511173 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:58.511065 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b563dcf-e028-4eaa-b33c-164306146411-cert\") pod \"kserve-controller-manager-d6f4b65b8-t4pkb\" (UID: \"8b563dcf-e028-4eaa-b33c-164306146411\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t4pkb" Jul 22 01:42:58.513481 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:58.513437 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17416ded-8f9b-4822-8800-d65e3bf062b3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-j9twf\" (UID: \"17416ded-8f9b-4822-8800-d65e3bf062b3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-j9twf" Jul 22 01:42:58.513611 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:58.513521 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b563dcf-e028-4eaa-b33c-164306146411-cert\") pod \"kserve-controller-manager-d6f4b65b8-t4pkb\" (UID: \"8b563dcf-e028-4eaa-b33c-164306146411\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t4pkb" Jul 22 01:42:58.696082 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:58.696041 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t4pkb" Jul 22 01:42:58.704256 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:58.704235 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-j9twf" Jul 22 01:42:58.836105 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:58.836070 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-t4pkb"] Jul 22 01:42:58.838506 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:42:58.838474 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b563dcf_e028_4eaa_b33c_164306146411.slice/crio-5fd2ec39fe9fd2e76a2ce8f77ee5ff61aa093d8c75ce2e2e2276119a553ccbc7 WatchSource:0}: Error finding container 5fd2ec39fe9fd2e76a2ce8f77ee5ff61aa093d8c75ce2e2e2276119a553ccbc7: Status 404 returned error can't find the container with id 5fd2ec39fe9fd2e76a2ce8f77ee5ff61aa093d8c75ce2e2e2276119a553ccbc7 Jul 22 01:42:58.864607 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:58.864583 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-j9twf"] Jul 22 01:42:58.865843 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:42:58.865821 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod17416ded_8f9b_4822_8800_d65e3bf062b3.slice/crio-e6402a07a28f14bb0efe82625d6e6413764e6960f48d21600b4346ae97d4e7ca WatchSource:0}: Error finding container e6402a07a28f14bb0efe82625d6e6413764e6960f48d21600b4346ae97d4e7ca: Status 404 returned error can't find the container with id e6402a07a28f14bb0efe82625d6e6413764e6960f48d21600b4346ae97d4e7ca Jul 22 01:42:59.537896 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:59.537852 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-j9twf" event={"ID":"17416ded-8f9b-4822-8800-d65e3bf062b3","Type":"ContainerStarted","Data":"e6402a07a28f14bb0efe82625d6e6413764e6960f48d21600b4346ae97d4e7ca"} Jul 22 01:42:59.538903 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:42:59.538874 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t4pkb" event={"ID":"8b563dcf-e028-4eaa-b33c-164306146411","Type":"ContainerStarted","Data":"5fd2ec39fe9fd2e76a2ce8f77ee5ff61aa093d8c75ce2e2e2276119a553ccbc7"} Jul 22 01:43:01.531997 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:01.531891 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz"] Jul 22 01:43:01.532412 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:01.532134 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" podUID="0c9100fd-2b91-4fc5-b652-f2b706586c9d" containerName="discovery" containerID="cri-o://66da2ec27f96ebcf7011ef10e8776bb393c55c8b86a539061157b91fa063ab50" gracePeriod=30 Jul 22 01:43:02.203405 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.203384 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" Jul 22 01:43:02.249618 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.249594 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rtk4\" (UniqueName: \"kubernetes.io/projected/0c9100fd-2b91-4fc5-b652-f2b706586c9d-kube-api-access-9rtk4\") pod \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\" (UID: \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\") " Jul 22 01:43:02.249762 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.249633 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9100fd-2b91-4fc5-b652-f2b706586c9d-local-certs\") pod \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\" (UID: \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\") " Jul 22 01:43:02.249762 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.249655 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c9100fd-2b91-4fc5-b652-f2b706586c9d-istio-csr-dns-cert\") pod \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\" (UID: \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\") " Jul 22 01:43:02.249762 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.249683 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c9100fd-2b91-4fc5-b652-f2b706586c9d-istio-kubeconfig\") pod \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\" (UID: \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\") " Jul 22 01:43:02.249930 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.249770 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9100fd-2b91-4fc5-b652-f2b706586c9d-istio-token\") pod \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\" (UID: \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\") " Jul 22 01:43:02.249930 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.249807 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c9100fd-2b91-4fc5-b652-f2b706586c9d-istio-csr-ca-configmap\") pod \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\" (UID: \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\") " Jul 22 01:43:02.249930 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.249880 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0c9100fd-2b91-4fc5-b652-f2b706586c9d-cacerts\") pod \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\" (UID: \"0c9100fd-2b91-4fc5-b652-f2b706586c9d\") " Jul 22 01:43:02.250423 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.250376 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c9100fd-2b91-4fc5-b652-f2b706586c9d-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "0c9100fd-2b91-4fc5-b652-f2b706586c9d" (UID: "0c9100fd-2b91-4fc5-b652-f2b706586c9d"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:43:02.252309 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.252258 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9100fd-2b91-4fc5-b652-f2b706586c9d-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "0c9100fd-2b91-4fc5-b652-f2b706586c9d" (UID: "0c9100fd-2b91-4fc5-b652-f2b706586c9d"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:43:02.252419 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.252392 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9100fd-2b91-4fc5-b652-f2b706586c9d-kube-api-access-9rtk4" (OuterVolumeSpecName: "kube-api-access-9rtk4") pod "0c9100fd-2b91-4fc5-b652-f2b706586c9d" (UID: "0c9100fd-2b91-4fc5-b652-f2b706586c9d"). InnerVolumeSpecName "kube-api-access-9rtk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:43:02.252827 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.252790 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9100fd-2b91-4fc5-b652-f2b706586c9d-cacerts" (OuterVolumeSpecName: "cacerts") pod "0c9100fd-2b91-4fc5-b652-f2b706586c9d" (UID: "0c9100fd-2b91-4fc5-b652-f2b706586c9d"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:43:02.253097 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.253077 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9100fd-2b91-4fc5-b652-f2b706586c9d-istio-token" (OuterVolumeSpecName: "istio-token") pod "0c9100fd-2b91-4fc5-b652-f2b706586c9d" (UID: "0c9100fd-2b91-4fc5-b652-f2b706586c9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:43:02.253097 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.253082 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9100fd-2b91-4fc5-b652-f2b706586c9d-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "0c9100fd-2b91-4fc5-b652-f2b706586c9d" (UID: "0c9100fd-2b91-4fc5-b652-f2b706586c9d"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:43:02.253528 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.253501 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9100fd-2b91-4fc5-b652-f2b706586c9d-local-certs" (OuterVolumeSpecName: "local-certs") pod "0c9100fd-2b91-4fc5-b652-f2b706586c9d" (UID: "0c9100fd-2b91-4fc5-b652-f2b706586c9d"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:43:02.351442 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.351360 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9100fd-2b91-4fc5-b652-f2b706586c9d-istio-token\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:43:02.351442 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.351400 2561 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c9100fd-2b91-4fc5-b652-f2b706586c9d-istio-csr-ca-configmap\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:43:02.351442 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.351414 2561 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0c9100fd-2b91-4fc5-b652-f2b706586c9d-cacerts\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:43:02.351442 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.351427 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rtk4\" (UniqueName: \"kubernetes.io/projected/0c9100fd-2b91-4fc5-b652-f2b706586c9d-kube-api-access-9rtk4\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:43:02.351442 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.351444 2561 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9100fd-2b91-4fc5-b652-f2b706586c9d-local-certs\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:43:02.351713 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.351459 2561 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c9100fd-2b91-4fc5-b652-f2b706586c9d-istio-csr-dns-cert\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:43:02.351713 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.351472 2561 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c9100fd-2b91-4fc5-b652-f2b706586c9d-istio-kubeconfig\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:43:02.555295 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.555262 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-j9twf" event={"ID":"17416ded-8f9b-4822-8800-d65e3bf062b3","Type":"ContainerStarted","Data":"55a9d7a205b1859fb7cd7b2c7dd69886d52b8987eba420ffe26ba2c00ba23fb4"} Jul 22 01:43:02.555717 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.555341 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-j9twf" Jul 22 01:43:02.556531 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.556503 2561 generic.go:356] "Generic (PLEG): container finished" podID="0c9100fd-2b91-4fc5-b652-f2b706586c9d" containerID="66da2ec27f96ebcf7011ef10e8776bb393c55c8b86a539061157b91fa063ab50" exitCode=0 Jul 22 01:43:02.556649 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.556544 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" Jul 22 01:43:02.556649 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.556586 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" event={"ID":"0c9100fd-2b91-4fc5-b652-f2b706586c9d","Type":"ContainerDied","Data":"66da2ec27f96ebcf7011ef10e8776bb393c55c8b86a539061157b91fa063ab50"} Jul 22 01:43:02.556649 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.556631 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz" event={"ID":"0c9100fd-2b91-4fc5-b652-f2b706586c9d","Type":"ContainerDied","Data":"27b9a880cec8c5c17e126f086f1305c137f533a73ac82537dce916b42ae3ed15"} Jul 22 01:43:02.556856 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.556654 2561 scope.go:117] "RemoveContainer" containerID="66da2ec27f96ebcf7011ef10e8776bb393c55c8b86a539061157b91fa063ab50" Jul 22 01:43:02.558301 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.558279 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t4pkb" event={"ID":"8b563dcf-e028-4eaa-b33c-164306146411","Type":"ContainerStarted","Data":"e92483d2d632a2607d2bdfa23a02b3999da13b851f14db13936bbd6b89d23a41"} Jul 22 01:43:02.558409 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.558325 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t4pkb" Jul 22 01:43:02.567023 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.567006 2561 scope.go:117] "RemoveContainer" containerID="66da2ec27f96ebcf7011ef10e8776bb393c55c8b86a539061157b91fa063ab50" Jul 22 01:43:02.567279 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:43:02.567260 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66da2ec27f96ebcf7011ef10e8776bb393c55c8b86a539061157b91fa063ab50\": container with ID starting with 66da2ec27f96ebcf7011ef10e8776bb393c55c8b86a539061157b91fa063ab50 not found: ID does not exist" containerID="66da2ec27f96ebcf7011ef10e8776bb393c55c8b86a539061157b91fa063ab50" Jul 22 01:43:02.567330 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.567286 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66da2ec27f96ebcf7011ef10e8776bb393c55c8b86a539061157b91fa063ab50"} err="failed to get container status \"66da2ec27f96ebcf7011ef10e8776bb393c55c8b86a539061157b91fa063ab50\": rpc error: code = NotFound desc = could not find container \"66da2ec27f96ebcf7011ef10e8776bb393c55c8b86a539061157b91fa063ab50\": container with ID starting with 66da2ec27f96ebcf7011ef10e8776bb393c55c8b86a539061157b91fa063ab50 not found: ID does not exist" Jul 22 01:43:02.584607 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.584570 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-j9twf" podStartSLOduration=2.370665668 podStartE2EDuration="5.584559185s" totalImagesPullingTime="3.213893517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:43:02.583783414 +0000 UTC m=+353.176587793" watchObservedRunningTime="2026-07-22 01:43:02.584559185 +0000 UTC m=+353.177363548" Jul 22 01:43:02.603453 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.603402 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz"] Jul 22 01:43:02.609217 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.609192 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9hgpz"] Jul 22 01:43:02.626319 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:02.626282 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t4pkb" podStartSLOduration=2.389176722 podStartE2EDuration="5.626271499s" totalImagesPullingTime="3.237094777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:43:02.625314666 +0000 UTC m=+353.218119030" watchObservedRunningTime="2026-07-22 01:43:02.626271499 +0000 UTC m=+353.219075896" Jul 22 01:43:03.924833 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:03.924802 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c9100fd-2b91-4fc5-b652-f2b706586c9d" path="/var/lib/kubelet/pods/0c9100fd-2b91-4fc5-b652-f2b706586c9d/volumes" Jul 22 01:43:04.514652 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:04.514625 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-f724l" Jul 22 01:43:04.516454 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:04.516424 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-pnbqj" Jul 22 01:43:33.566842 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:33.566811 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-j9twf" Jul 22 01:43:33.569852 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:33.569832 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t4pkb" Jul 22 01:43:47.647850 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:47.647820 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dr5kk"] Jul 22 01:43:47.648222 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:47.648169 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c9100fd-2b91-4fc5-b652-f2b706586c9d" containerName="discovery" Jul 22 01:43:47.648222 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:47.648180 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c9100fd-2b91-4fc5-b652-f2b706586c9d" containerName="discovery" Jul 22 01:43:47.648295 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:47.648261 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c9100fd-2b91-4fc5-b652-f2b706586c9d" containerName="discovery" Jul 22 01:43:47.650338 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:47.650321 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dr5kk" Jul 22 01:43:47.652940 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:47.652911 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 01:43:47.653041 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:47.652971 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-s6b96\"" Jul 22 01:43:47.653041 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:47.653021 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 01:43:47.660459 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:47.660435 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dr5kk"] Jul 22 01:43:47.718364 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:47.718335 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zrcb\" (UniqueName: \"kubernetes.io/projected/f71f0163-3df8-468a-b38c-900f41b37429-kube-api-access-8zrcb\") pod \"kuadrant-operator-catalog-dr5kk\" (UID: \"f71f0163-3df8-468a-b38c-900f41b37429\") " pod="kuadrant-system/kuadrant-operator-catalog-dr5kk" Jul 22 01:43:47.818774 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:47.818751 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zrcb\" (UniqueName: \"kubernetes.io/projected/f71f0163-3df8-468a-b38c-900f41b37429-kube-api-access-8zrcb\") pod \"kuadrant-operator-catalog-dr5kk\" (UID: \"f71f0163-3df8-468a-b38c-900f41b37429\") " pod="kuadrant-system/kuadrant-operator-catalog-dr5kk" Jul 22 01:43:47.827842 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:47.827818 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zrcb\" (UniqueName: \"kubernetes.io/projected/f71f0163-3df8-468a-b38c-900f41b37429-kube-api-access-8zrcb\") pod \"kuadrant-operator-catalog-dr5kk\" (UID: \"f71f0163-3df8-468a-b38c-900f41b37429\") " pod="kuadrant-system/kuadrant-operator-catalog-dr5kk" Jul 22 01:43:47.961822 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:47.961795 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dr5kk" Jul 22 01:43:48.003771 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:48.003741 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dr5kk"] Jul 22 01:43:48.096146 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:48.096120 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dr5kk"] Jul 22 01:43:48.098147 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:43:48.098123 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71f0163_3df8_468a_b38c_900f41b37429.slice/crio-b65cb78e5651dd97db96b462e36119bdea70cae2ee358098f871932190f2b8e6 WatchSource:0}: Error finding container b65cb78e5651dd97db96b462e36119bdea70cae2ee358098f871932190f2b8e6: Status 404 returned error can't find the container with id b65cb78e5651dd97db96b462e36119bdea70cae2ee358098f871932190f2b8e6 Jul 22 01:43:48.211441 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:48.211407 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-z2n5w"] Jul 22 01:43:48.214416 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:48.214372 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-z2n5w" Jul 22 01:43:48.223215 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:48.223194 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-z2n5w"] Jul 22 01:43:48.321804 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:48.321764 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkdkf\" (UniqueName: \"kubernetes.io/projected/7b31df87-eb54-4fb0-9b11-a4abcb0287d1-kube-api-access-hkdkf\") pod \"kuadrant-operator-catalog-z2n5w\" (UID: \"7b31df87-eb54-4fb0-9b11-a4abcb0287d1\") " pod="kuadrant-system/kuadrant-operator-catalog-z2n5w" Jul 22 01:43:48.423043 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:48.422992 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkdkf\" (UniqueName: \"kubernetes.io/projected/7b31df87-eb54-4fb0-9b11-a4abcb0287d1-kube-api-access-hkdkf\") pod \"kuadrant-operator-catalog-z2n5w\" (UID: \"7b31df87-eb54-4fb0-9b11-a4abcb0287d1\") " pod="kuadrant-system/kuadrant-operator-catalog-z2n5w" Jul 22 01:43:48.431962 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:48.431936 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkdkf\" (UniqueName: \"kubernetes.io/projected/7b31df87-eb54-4fb0-9b11-a4abcb0287d1-kube-api-access-hkdkf\") pod \"kuadrant-operator-catalog-z2n5w\" (UID: \"7b31df87-eb54-4fb0-9b11-a4abcb0287d1\") " pod="kuadrant-system/kuadrant-operator-catalog-z2n5w" Jul 22 01:43:48.524967 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:48.524895 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-z2n5w" Jul 22 01:43:48.673443 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:48.673417 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-z2n5w"] Jul 22 01:43:48.684821 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:43:48.684778 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b31df87_eb54_4fb0_9b11_a4abcb0287d1.slice/crio-c3fb3ba6ae634a1cb3600c94d06e0d80cbfdb2f5dc4150c03e29777f08ae6c4f WatchSource:0}: Error finding container c3fb3ba6ae634a1cb3600c94d06e0d80cbfdb2f5dc4150c03e29777f08ae6c4f: Status 404 returned error can't find the container with id c3fb3ba6ae634a1cb3600c94d06e0d80cbfdb2f5dc4150c03e29777f08ae6c4f Jul 22 01:43:48.786290 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:48.786204 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-z2n5w" event={"ID":"7b31df87-eb54-4fb0-9b11-a4abcb0287d1","Type":"ContainerStarted","Data":"c3fb3ba6ae634a1cb3600c94d06e0d80cbfdb2f5dc4150c03e29777f08ae6c4f"} Jul 22 01:43:48.787386 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:48.787359 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dr5kk" event={"ID":"f71f0163-3df8-468a-b38c-900f41b37429","Type":"ContainerStarted","Data":"b65cb78e5651dd97db96b462e36119bdea70cae2ee358098f871932190f2b8e6"} Jul 22 01:43:50.799841 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:50.799796 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dr5kk" event={"ID":"f71f0163-3df8-468a-b38c-900f41b37429","Type":"ContainerStarted","Data":"c225caaba0c67dfd5be9cf02b878f8c6381963d151d901f55b34da111b0aea46"} Jul 22 01:43:50.800316 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:50.799825 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-dr5kk" podUID="f71f0163-3df8-468a-b38c-900f41b37429" containerName="registry-server" containerID="cri-o://c225caaba0c67dfd5be9cf02b878f8c6381963d151d901f55b34da111b0aea46" gracePeriod=2 Jul 22 01:43:50.801450 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:50.801417 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-z2n5w" event={"ID":"7b31df87-eb54-4fb0-9b11-a4abcb0287d1","Type":"ContainerStarted","Data":"f3cde073330e6d841bbaa35d18b62cc248d5a49a0c378bfd59823865eac74bdf"} Jul 22 01:43:50.817083 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:50.817041 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-dr5kk" podStartSLOduration=1.912135269 podStartE2EDuration="3.817026622s" totalImagesPullingTime="1.904891353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:43:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:43:50.815607863 +0000 UTC m=+401.408412226" watchObservedRunningTime="2026-07-22 01:43:50.817026622 +0000 UTC m=+401.409831019" Jul 22 01:43:50.835500 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:50.835450 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-z2n5w" podStartSLOduration=1.5216795159999998 podStartE2EDuration="2.835435523s" totalImagesPullingTime="1.313756007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:43:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:43:50.834135918 +0000 UTC m=+401.426940334" watchObservedRunningTime="2026-07-22 01:43:50.835435523 +0000 UTC m=+401.428239885" Jul 22 01:43:51.043543 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:51.043521 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dr5kk" Jul 22 01:43:51.146855 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:51.146777 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zrcb\" (UniqueName: \"kubernetes.io/projected/f71f0163-3df8-468a-b38c-900f41b37429-kube-api-access-8zrcb\") pod \"f71f0163-3df8-468a-b38c-900f41b37429\" (UID: \"f71f0163-3df8-468a-b38c-900f41b37429\") " Jul 22 01:43:51.149063 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:51.149039 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f71f0163-3df8-468a-b38c-900f41b37429-kube-api-access-8zrcb" (OuterVolumeSpecName: "kube-api-access-8zrcb") pod "f71f0163-3df8-468a-b38c-900f41b37429" (UID: "f71f0163-3df8-468a-b38c-900f41b37429"). InnerVolumeSpecName "kube-api-access-8zrcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:43:51.248244 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:51.248207 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zrcb\" (UniqueName: \"kubernetes.io/projected/f71f0163-3df8-468a-b38c-900f41b37429-kube-api-access-8zrcb\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:43:51.806997 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:51.806964 2561 generic.go:356] "Generic (PLEG): container finished" podID="f71f0163-3df8-468a-b38c-900f41b37429" containerID="c225caaba0c67dfd5be9cf02b878f8c6381963d151d901f55b34da111b0aea46" exitCode=0 Jul 22 01:43:51.807480 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:51.807014 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dr5kk" Jul 22 01:43:51.807480 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:51.807047 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dr5kk" event={"ID":"f71f0163-3df8-468a-b38c-900f41b37429","Type":"ContainerDied","Data":"c225caaba0c67dfd5be9cf02b878f8c6381963d151d901f55b34da111b0aea46"} Jul 22 01:43:51.807480 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:51.807094 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dr5kk" event={"ID":"f71f0163-3df8-468a-b38c-900f41b37429","Type":"ContainerDied","Data":"b65cb78e5651dd97db96b462e36119bdea70cae2ee358098f871932190f2b8e6"} Jul 22 01:43:51.807480 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:51.807115 2561 scope.go:117] "RemoveContainer" containerID="c225caaba0c67dfd5be9cf02b878f8c6381963d151d901f55b34da111b0aea46" Jul 22 01:43:51.817149 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:51.817132 2561 scope.go:117] "RemoveContainer" containerID="c225caaba0c67dfd5be9cf02b878f8c6381963d151d901f55b34da111b0aea46" Jul 22 01:43:51.817391 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:43:51.817374 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c225caaba0c67dfd5be9cf02b878f8c6381963d151d901f55b34da111b0aea46\": container with ID starting with c225caaba0c67dfd5be9cf02b878f8c6381963d151d901f55b34da111b0aea46 not found: ID does not exist" containerID="c225caaba0c67dfd5be9cf02b878f8c6381963d151d901f55b34da111b0aea46" Jul 22 01:43:51.817441 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:51.817401 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c225caaba0c67dfd5be9cf02b878f8c6381963d151d901f55b34da111b0aea46"} err="failed to get container status \"c225caaba0c67dfd5be9cf02b878f8c6381963d151d901f55b34da111b0aea46\": rpc error: code = NotFound desc = could not find container \"c225caaba0c67dfd5be9cf02b878f8c6381963d151d901f55b34da111b0aea46\": container with ID starting with c225caaba0c67dfd5be9cf02b878f8c6381963d151d901f55b34da111b0aea46 not found: ID does not exist" Jul 22 01:43:51.829602 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:51.829578 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dr5kk"] Jul 22 01:43:51.833774 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:51.833753 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dr5kk"] Jul 22 01:43:51.924448 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:51.924419 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f71f0163-3df8-468a-b38c-900f41b37429" path="/var/lib/kubelet/pods/f71f0163-3df8-468a-b38c-900f41b37429/volumes" Jul 22 01:43:58.525599 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:58.525563 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-z2n5w" Jul 22 01:43:58.526016 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:58.525623 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-z2n5w" Jul 22 01:43:58.548036 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:58.548002 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-z2n5w" Jul 22 01:43:58.861925 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:43:58.861850 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-z2n5w" Jul 22 01:44:03.023855 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.023818 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc"] Jul 22 01:44:03.024214 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.024196 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f71f0163-3df8-468a-b38c-900f41b37429" containerName="registry-server" Jul 22 01:44:03.024214 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.024207 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f71f0163-3df8-468a-b38c-900f41b37429" containerName="registry-server" Jul 22 01:44:03.024290 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.024279 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f71f0163-3df8-468a-b38c-900f41b37429" containerName="registry-server" Jul 22 01:44:03.026643 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.026627 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc" Jul 22 01:44:03.029078 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.029054 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2f55z\"" Jul 22 01:44:03.035781 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.035756 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc"] Jul 22 01:44:03.144240 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.144207 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e48954b4-33a3-4744-893d-cceb6c6c6ed0-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc\" (UID: \"e48954b4-33a3-4744-893d-cceb6c6c6ed0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc" Jul 22 01:44:03.144240 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.144238 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e48954b4-33a3-4744-893d-cceb6c6c6ed0-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc\" (UID: \"e48954b4-33a3-4744-893d-cceb6c6c6ed0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc" Jul 22 01:44:03.144422 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.144263 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99hn6\" (UniqueName: \"kubernetes.io/projected/e48954b4-33a3-4744-893d-cceb6c6c6ed0-kube-api-access-99hn6\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc\" (UID: \"e48954b4-33a3-4744-893d-cceb6c6c6ed0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc" Jul 22 01:44:03.245571 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.245542 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e48954b4-33a3-4744-893d-cceb6c6c6ed0-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc\" (UID: \"e48954b4-33a3-4744-893d-cceb6c6c6ed0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc" Jul 22 01:44:03.245571 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.245573 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e48954b4-33a3-4744-893d-cceb6c6c6ed0-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc\" (UID: \"e48954b4-33a3-4744-893d-cceb6c6c6ed0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc" Jul 22 01:44:03.245758 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.245597 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99hn6\" (UniqueName: \"kubernetes.io/projected/e48954b4-33a3-4744-893d-cceb6c6c6ed0-kube-api-access-99hn6\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc\" (UID: \"e48954b4-33a3-4744-893d-cceb6c6c6ed0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc" Jul 22 01:44:03.246019 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.245996 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e48954b4-33a3-4744-893d-cceb6c6c6ed0-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc\" (UID: \"e48954b4-33a3-4744-893d-cceb6c6c6ed0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc" Jul 22 01:44:03.246061 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.246005 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e48954b4-33a3-4744-893d-cceb6c6c6ed0-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc\" (UID: \"e48954b4-33a3-4744-893d-cceb6c6c6ed0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc" Jul 22 01:44:03.255436 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.255398 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99hn6\" (UniqueName: \"kubernetes.io/projected/e48954b4-33a3-4744-893d-cceb6c6c6ed0-kube-api-access-99hn6\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc\" (UID: \"e48954b4-33a3-4744-893d-cceb6c6c6ed0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc" Jul 22 01:44:03.335805 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.335744 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc" Jul 22 01:44:03.466779 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.466757 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc"] Jul 22 01:44:03.468589 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:44:03.468560 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode48954b4_33a3_4744_893d_cceb6c6c6ed0.slice/crio-e6ac0e3455fb8900814108bd476caf2443c335faea702b9c393634f4a0ac84b6 WatchSource:0}: Error finding container e6ac0e3455fb8900814108bd476caf2443c335faea702b9c393634f4a0ac84b6: Status 404 returned error can't find the container with id e6ac0e3455fb8900814108bd476caf2443c335faea702b9c393634f4a0ac84b6 Jul 22 01:44:03.624651 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.624586 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb"] Jul 22 01:44:03.627457 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.627438 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb" Jul 22 01:44:03.637736 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.637712 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb"] Jul 22 01:44:03.751147 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.751118 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97270ed7-a6bb-4bb3-898e-b08205c7ef5e-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb\" (UID: \"97270ed7-a6bb-4bb3-898e-b08205c7ef5e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb" Jul 22 01:44:03.751147 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.751150 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97270ed7-a6bb-4bb3-898e-b08205c7ef5e-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb\" (UID: \"97270ed7-a6bb-4bb3-898e-b08205c7ef5e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb" Jul 22 01:44:03.751332 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.751175 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnmdd\" (UniqueName: \"kubernetes.io/projected/97270ed7-a6bb-4bb3-898e-b08205c7ef5e-kube-api-access-xnmdd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb\" (UID: \"97270ed7-a6bb-4bb3-898e-b08205c7ef5e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb" Jul 22 01:44:03.851589 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.851565 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97270ed7-a6bb-4bb3-898e-b08205c7ef5e-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb\" (UID: \"97270ed7-a6bb-4bb3-898e-b08205c7ef5e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb" Jul 22 01:44:03.851690 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.851594 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97270ed7-a6bb-4bb3-898e-b08205c7ef5e-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb\" (UID: \"97270ed7-a6bb-4bb3-898e-b08205c7ef5e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb" Jul 22 01:44:03.851690 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.851623 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnmdd\" (UniqueName: \"kubernetes.io/projected/97270ed7-a6bb-4bb3-898e-b08205c7ef5e-kube-api-access-xnmdd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb\" (UID: \"97270ed7-a6bb-4bb3-898e-b08205c7ef5e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb" Jul 22 01:44:03.851968 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.851953 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97270ed7-a6bb-4bb3-898e-b08205c7ef5e-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb\" (UID: \"97270ed7-a6bb-4bb3-898e-b08205c7ef5e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb" Jul 22 01:44:03.852007 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.851989 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97270ed7-a6bb-4bb3-898e-b08205c7ef5e-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb\" (UID: \"97270ed7-a6bb-4bb3-898e-b08205c7ef5e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb" Jul 22 01:44:03.861028 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.860999 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnmdd\" (UniqueName: \"kubernetes.io/projected/97270ed7-a6bb-4bb3-898e-b08205c7ef5e-kube-api-access-xnmdd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb\" (UID: \"97270ed7-a6bb-4bb3-898e-b08205c7ef5e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb" Jul 22 01:44:03.867891 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.867869 2561 generic.go:356] "Generic (PLEG): container finished" podID="e48954b4-33a3-4744-893d-cceb6c6c6ed0" containerID="f6b14686f404d65bcac082d411ce2860e335fc29dc0484b83d539958311481d0" exitCode=0 Jul 22 01:44:03.867988 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.867925 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc" event={"ID":"e48954b4-33a3-4744-893d-cceb6c6c6ed0","Type":"ContainerDied","Data":"f6b14686f404d65bcac082d411ce2860e335fc29dc0484b83d539958311481d0"} Jul 22 01:44:03.867988 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.867980 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc" event={"ID":"e48954b4-33a3-4744-893d-cceb6c6c6ed0","Type":"ContainerStarted","Data":"e6ac0e3455fb8900814108bd476caf2443c335faea702b9c393634f4a0ac84b6"} Jul 22 01:44:03.936067 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:03.936042 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb" Jul 22 01:44:04.022924 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.022897 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf"] Jul 22 01:44:04.026750 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.026730 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf" Jul 22 01:44:04.035269 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.035195 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf"] Jul 22 01:44:04.068910 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.068885 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb"] Jul 22 01:44:04.070625 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:44:04.070601 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97270ed7_a6bb_4bb3_898e_b08205c7ef5e.slice/crio-cf3c55ab65bff10d302b283073a4963a69c09e82370b950e5e57ac48f742d706 WatchSource:0}: Error finding container cf3c55ab65bff10d302b283073a4963a69c09e82370b950e5e57ac48f742d706: Status 404 returned error can't find the container with id cf3c55ab65bff10d302b283073a4963a69c09e82370b950e5e57ac48f742d706 Jul 22 01:44:04.154129 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.154083 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ec8f212-80ab-497b-8f4f-da31514f063e-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf\" (UID: \"3ec8f212-80ab-497b-8f4f-da31514f063e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf" Jul 22 01:44:04.154329 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.154137 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ec8f212-80ab-497b-8f4f-da31514f063e-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf\" (UID: \"3ec8f212-80ab-497b-8f4f-da31514f063e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf" Jul 22 01:44:04.154329 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.154193 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl2v4\" (UniqueName: \"kubernetes.io/projected/3ec8f212-80ab-497b-8f4f-da31514f063e-kube-api-access-gl2v4\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf\" (UID: \"3ec8f212-80ab-497b-8f4f-da31514f063e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf" Jul 22 01:44:04.254772 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.254714 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ec8f212-80ab-497b-8f4f-da31514f063e-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf\" (UID: \"3ec8f212-80ab-497b-8f4f-da31514f063e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf" Jul 22 01:44:04.254772 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.254751 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ec8f212-80ab-497b-8f4f-da31514f063e-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf\" (UID: \"3ec8f212-80ab-497b-8f4f-da31514f063e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf" Jul 22 01:44:04.254926 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.254780 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl2v4\" (UniqueName: \"kubernetes.io/projected/3ec8f212-80ab-497b-8f4f-da31514f063e-kube-api-access-gl2v4\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf\" (UID: \"3ec8f212-80ab-497b-8f4f-da31514f063e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf" Jul 22 01:44:04.255125 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.255105 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ec8f212-80ab-497b-8f4f-da31514f063e-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf\" (UID: \"3ec8f212-80ab-497b-8f4f-da31514f063e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf" Jul 22 01:44:04.255164 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.255144 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ec8f212-80ab-497b-8f4f-da31514f063e-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf\" (UID: \"3ec8f212-80ab-497b-8f4f-da31514f063e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf" Jul 22 01:44:04.264338 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.264315 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl2v4\" (UniqueName: \"kubernetes.io/projected/3ec8f212-80ab-497b-8f4f-da31514f063e-kube-api-access-gl2v4\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf\" (UID: \"3ec8f212-80ab-497b-8f4f-da31514f063e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf" Jul 22 01:44:04.338432 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.338407 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf" Jul 22 01:44:04.428546 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.428518 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8"] Jul 22 01:44:04.434835 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.434800 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8" Jul 22 01:44:04.439805 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.439757 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8"] Jul 22 01:44:04.473582 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.473558 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf"] Jul 22 01:44:04.475670 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:44:04.475646 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec8f212_80ab_497b_8f4f_da31514f063e.slice/crio-b1ede701ed13584f822c763cdf88a285b707c334fd45ec530af6e70451b9e18e WatchSource:0}: Error finding container b1ede701ed13584f822c763cdf88a285b707c334fd45ec530af6e70451b9e18e: Status 404 returned error can't find the container with id b1ede701ed13584f822c763cdf88a285b707c334fd45ec530af6e70451b9e18e Jul 22 01:44:04.556809 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.556775 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57tz7\" (UniqueName: \"kubernetes.io/projected/2dba0d85-6bd2-4b3a-9811-eb5b58d6355b-kube-api-access-57tz7\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8\" (UID: \"2dba0d85-6bd2-4b3a-9811-eb5b58d6355b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8" Jul 22 01:44:04.556940 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.556840 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2dba0d85-6bd2-4b3a-9811-eb5b58d6355b-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8\" (UID: \"2dba0d85-6bd2-4b3a-9811-eb5b58d6355b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8" Jul 22 01:44:04.556940 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.556926 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2dba0d85-6bd2-4b3a-9811-eb5b58d6355b-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8\" (UID: \"2dba0d85-6bd2-4b3a-9811-eb5b58d6355b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8" Jul 22 01:44:04.658351 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.658324 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2dba0d85-6bd2-4b3a-9811-eb5b58d6355b-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8\" (UID: \"2dba0d85-6bd2-4b3a-9811-eb5b58d6355b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8" Jul 22 01:44:04.658481 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.658368 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57tz7\" (UniqueName: \"kubernetes.io/projected/2dba0d85-6bd2-4b3a-9811-eb5b58d6355b-kube-api-access-57tz7\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8\" (UID: \"2dba0d85-6bd2-4b3a-9811-eb5b58d6355b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8" Jul 22 01:44:04.658481 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.658409 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2dba0d85-6bd2-4b3a-9811-eb5b58d6355b-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8\" (UID: \"2dba0d85-6bd2-4b3a-9811-eb5b58d6355b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8" Jul 22 01:44:04.658683 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.658666 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2dba0d85-6bd2-4b3a-9811-eb5b58d6355b-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8\" (UID: \"2dba0d85-6bd2-4b3a-9811-eb5b58d6355b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8" Jul 22 01:44:04.658793 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.658772 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2dba0d85-6bd2-4b3a-9811-eb5b58d6355b-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8\" (UID: \"2dba0d85-6bd2-4b3a-9811-eb5b58d6355b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8" Jul 22 01:44:04.667923 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.667902 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57tz7\" (UniqueName: \"kubernetes.io/projected/2dba0d85-6bd2-4b3a-9811-eb5b58d6355b-kube-api-access-57tz7\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8\" (UID: \"2dba0d85-6bd2-4b3a-9811-eb5b58d6355b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8" Jul 22 01:44:04.748284 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.748261 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8" Jul 22 01:44:04.874518 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.874483 2561 generic.go:356] "Generic (PLEG): container finished" podID="e48954b4-33a3-4744-893d-cceb6c6c6ed0" containerID="913f16ed2359c0045d040113bd81fc603e4a0ebc3505a4adfc447586bd4abddb" exitCode=0 Jul 22 01:44:04.874665 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.874566 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc" event={"ID":"e48954b4-33a3-4744-893d-cceb6c6c6ed0","Type":"ContainerDied","Data":"913f16ed2359c0045d040113bd81fc603e4a0ebc3505a4adfc447586bd4abddb"} Jul 22 01:44:04.876123 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.876103 2561 generic.go:356] "Generic (PLEG): container finished" podID="3ec8f212-80ab-497b-8f4f-da31514f063e" containerID="ddb61b2abbb3a5fe2cf37bf63ad6e9aa3ea4da0c1f9d233b0edeed5b99304c53" exitCode=0 Jul 22 01:44:04.876204 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.876183 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf" event={"ID":"3ec8f212-80ab-497b-8f4f-da31514f063e","Type":"ContainerDied","Data":"ddb61b2abbb3a5fe2cf37bf63ad6e9aa3ea4da0c1f9d233b0edeed5b99304c53"} Jul 22 01:44:04.876278 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.876260 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf" event={"ID":"3ec8f212-80ab-497b-8f4f-da31514f063e","Type":"ContainerStarted","Data":"b1ede701ed13584f822c763cdf88a285b707c334fd45ec530af6e70451b9e18e"} Jul 22 01:44:04.878816 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.878791 2561 generic.go:356] "Generic (PLEG): container finished" podID="97270ed7-a6bb-4bb3-898e-b08205c7ef5e" containerID="2a5002756474217363a8b476cbe6e78038fe0a92b37fd3bc18a9efe0b877c297" exitCode=0 Jul 22 01:44:04.878887 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.878822 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb" event={"ID":"97270ed7-a6bb-4bb3-898e-b08205c7ef5e","Type":"ContainerDied","Data":"2a5002756474217363a8b476cbe6e78038fe0a92b37fd3bc18a9efe0b877c297"} Jul 22 01:44:04.878887 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.878876 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb" event={"ID":"97270ed7-a6bb-4bb3-898e-b08205c7ef5e","Type":"ContainerStarted","Data":"cf3c55ab65bff10d302b283073a4963a69c09e82370b950e5e57ac48f742d706"} Jul 22 01:44:04.881226 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:04.881207 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8"] Jul 22 01:44:04.882105 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:44:04.882086 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dba0d85_6bd2_4b3a_9811_eb5b58d6355b.slice/crio-cb03264b5d8a648c889480973d5586ad562cd94297b7f13047ab0824000d54cd WatchSource:0}: Error finding container cb03264b5d8a648c889480973d5586ad562cd94297b7f13047ab0824000d54cd: Status 404 returned error can't find the container with id cb03264b5d8a648c889480973d5586ad562cd94297b7f13047ab0824000d54cd Jul 22 01:44:05.886150 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:05.886120 2561 generic.go:356] "Generic (PLEG): container finished" podID="e48954b4-33a3-4744-893d-cceb6c6c6ed0" containerID="a605ce5e39cdea35747eda5e9c271670fa2f78a4f769919f2dbc426abed39445" exitCode=0 Jul 22 01:44:05.886487 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:05.886196 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc" event={"ID":"e48954b4-33a3-4744-893d-cceb6c6c6ed0","Type":"ContainerDied","Data":"a605ce5e39cdea35747eda5e9c271670fa2f78a4f769919f2dbc426abed39445"} Jul 22 01:44:05.887966 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:05.887949 2561 generic.go:356] "Generic (PLEG): container finished" podID="3ec8f212-80ab-497b-8f4f-da31514f063e" containerID="0d59a5bfedf98f03dfb0299e2b3792edfdf247b0bbd9bd96037c81566fa05b9b" exitCode=0 Jul 22 01:44:05.888065 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:05.888039 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf" event={"ID":"3ec8f212-80ab-497b-8f4f-da31514f063e","Type":"ContainerDied","Data":"0d59a5bfedf98f03dfb0299e2b3792edfdf247b0bbd9bd96037c81566fa05b9b"} Jul 22 01:44:05.890000 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:05.889981 2561 generic.go:356] "Generic (PLEG): container finished" podID="97270ed7-a6bb-4bb3-898e-b08205c7ef5e" containerID="e8be88ff43c593bd68ad52ad6cea6df5630ead1218502497502befeb2ba02f2b" exitCode=0 Jul 22 01:44:05.890144 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:05.890044 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb" event={"ID":"97270ed7-a6bb-4bb3-898e-b08205c7ef5e","Type":"ContainerDied","Data":"e8be88ff43c593bd68ad52ad6cea6df5630ead1218502497502befeb2ba02f2b"} Jul 22 01:44:05.891824 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:05.891804 2561 generic.go:356] "Generic (PLEG): container finished" podID="2dba0d85-6bd2-4b3a-9811-eb5b58d6355b" containerID="6e739c345a5586a64d5a908868e923dd2e7803536aeb9699354d3950ac27c2dd" exitCode=0 Jul 22 01:44:05.891919 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:05.891867 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8" event={"ID":"2dba0d85-6bd2-4b3a-9811-eb5b58d6355b","Type":"ContainerDied","Data":"6e739c345a5586a64d5a908868e923dd2e7803536aeb9699354d3950ac27c2dd"} Jul 22 01:44:05.891919 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:05.891894 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8" event={"ID":"2dba0d85-6bd2-4b3a-9811-eb5b58d6355b","Type":"ContainerStarted","Data":"cb03264b5d8a648c889480973d5586ad562cd94297b7f13047ab0824000d54cd"} Jul 22 01:44:06.899981 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:06.899898 2561 generic.go:356] "Generic (PLEG): container finished" podID="3ec8f212-80ab-497b-8f4f-da31514f063e" containerID="4daeb618329e65011c03bead726d16278efad3a4c868f983e7f537bddedac8a7" exitCode=0 Jul 22 01:44:06.900352 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:06.899990 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf" event={"ID":"3ec8f212-80ab-497b-8f4f-da31514f063e","Type":"ContainerDied","Data":"4daeb618329e65011c03bead726d16278efad3a4c868f983e7f537bddedac8a7"} Jul 22 01:44:06.902024 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:06.901992 2561 generic.go:356] "Generic (PLEG): container finished" podID="97270ed7-a6bb-4bb3-898e-b08205c7ef5e" containerID="e3ce105b9d5d81666408c455dd37e65b4c06d1c0df3ddbd4ae1f1e04b87583d2" exitCode=0 Jul 22 01:44:06.902138 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:06.902020 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb" event={"ID":"97270ed7-a6bb-4bb3-898e-b08205c7ef5e","Type":"ContainerDied","Data":"e3ce105b9d5d81666408c455dd37e65b4c06d1c0df3ddbd4ae1f1e04b87583d2"} Jul 22 01:44:06.903628 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:06.903608 2561 generic.go:356] "Generic (PLEG): container finished" podID="2dba0d85-6bd2-4b3a-9811-eb5b58d6355b" containerID="dabef632ec60cc790303cd8c552294bb59ed584b972e0871bb91524ea9c400d8" exitCode=0 Jul 22 01:44:06.903734 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:06.903632 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8" event={"ID":"2dba0d85-6bd2-4b3a-9811-eb5b58d6355b","Type":"ContainerDied","Data":"dabef632ec60cc790303cd8c552294bb59ed584b972e0871bb91524ea9c400d8"} Jul 22 01:44:07.039058 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:07.039035 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc" Jul 22 01:44:07.087941 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:07.087917 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e48954b4-33a3-4744-893d-cceb6c6c6ed0-bundle\") pod \"e48954b4-33a3-4744-893d-cceb6c6c6ed0\" (UID: \"e48954b4-33a3-4744-893d-cceb6c6c6ed0\") " Jul 22 01:44:07.088096 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:07.087978 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e48954b4-33a3-4744-893d-cceb6c6c6ed0-util\") pod \"e48954b4-33a3-4744-893d-cceb6c6c6ed0\" (UID: \"e48954b4-33a3-4744-893d-cceb6c6c6ed0\") " Jul 22 01:44:07.088096 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:07.088038 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99hn6\" (UniqueName: \"kubernetes.io/projected/e48954b4-33a3-4744-893d-cceb6c6c6ed0-kube-api-access-99hn6\") pod \"e48954b4-33a3-4744-893d-cceb6c6c6ed0\" (UID: \"e48954b4-33a3-4744-893d-cceb6c6c6ed0\") " Jul 22 01:44:07.088392 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:07.088367 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48954b4-33a3-4744-893d-cceb6c6c6ed0-bundle" (OuterVolumeSpecName: "bundle") pod "e48954b4-33a3-4744-893d-cceb6c6c6ed0" (UID: "e48954b4-33a3-4744-893d-cceb6c6c6ed0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:44:07.090063 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:07.090045 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e48954b4-33a3-4744-893d-cceb6c6c6ed0-kube-api-access-99hn6" (OuterVolumeSpecName: "kube-api-access-99hn6") pod "e48954b4-33a3-4744-893d-cceb6c6c6ed0" (UID: "e48954b4-33a3-4744-893d-cceb6c6c6ed0"). InnerVolumeSpecName "kube-api-access-99hn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:44:07.093883 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:07.093862 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48954b4-33a3-4744-893d-cceb6c6c6ed0-util" (OuterVolumeSpecName: "util") pod "e48954b4-33a3-4744-893d-cceb6c6c6ed0" (UID: "e48954b4-33a3-4744-893d-cceb6c6c6ed0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:44:07.189625 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:07.189599 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e48954b4-33a3-4744-893d-cceb6c6c6ed0-bundle\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:44:07.189625 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:07.189626 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e48954b4-33a3-4744-893d-cceb6c6c6ed0-util\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:44:07.189806 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:07.189637 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99hn6\" (UniqueName: \"kubernetes.io/projected/e48954b4-33a3-4744-893d-cceb6c6c6ed0-kube-api-access-99hn6\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:44:07.911536 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:07.911498 2561 generic.go:356] "Generic (PLEG): container finished" podID="2dba0d85-6bd2-4b3a-9811-eb5b58d6355b" containerID="dff7dc6cb49fea14d788ccf0489e7f10f8119dc65375a3795d43f442a055cd9a" exitCode=0 Jul 22 01:44:07.911964 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:07.911592 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8" event={"ID":"2dba0d85-6bd2-4b3a-9811-eb5b58d6355b","Type":"ContainerDied","Data":"dff7dc6cb49fea14d788ccf0489e7f10f8119dc65375a3795d43f442a055cd9a"} Jul 22 01:44:07.913508 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:07.913474 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc" event={"ID":"e48954b4-33a3-4744-893d-cceb6c6c6ed0","Type":"ContainerDied","Data":"e6ac0e3455fb8900814108bd476caf2443c335faea702b9c393634f4a0ac84b6"} Jul 22 01:44:07.913629 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:07.913524 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6ac0e3455fb8900814108bd476caf2443c335faea702b9c393634f4a0ac84b6" Jul 22 01:44:07.914319 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:07.913812 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc" Jul 22 01:44:08.052131 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:08.052111 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb" Jul 22 01:44:08.081323 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:08.081303 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf" Jul 22 01:44:08.097740 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:08.097716 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97270ed7-a6bb-4bb3-898e-b08205c7ef5e-bundle\") pod \"97270ed7-a6bb-4bb3-898e-b08205c7ef5e\" (UID: \"97270ed7-a6bb-4bb3-898e-b08205c7ef5e\") " Jul 22 01:44:08.097872 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:08.097767 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97270ed7-a6bb-4bb3-898e-b08205c7ef5e-util\") pod \"97270ed7-a6bb-4bb3-898e-b08205c7ef5e\" (UID: \"97270ed7-a6bb-4bb3-898e-b08205c7ef5e\") " Jul 22 01:44:08.097872 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:08.097796 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnmdd\" (UniqueName: \"kubernetes.io/projected/97270ed7-a6bb-4bb3-898e-b08205c7ef5e-kube-api-access-xnmdd\") pod \"97270ed7-a6bb-4bb3-898e-b08205c7ef5e\" (UID: \"97270ed7-a6bb-4bb3-898e-b08205c7ef5e\") " Jul 22 01:44:08.098290 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:08.098265 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97270ed7-a6bb-4bb3-898e-b08205c7ef5e-bundle" (OuterVolumeSpecName: "bundle") pod "97270ed7-a6bb-4bb3-898e-b08205c7ef5e" (UID: "97270ed7-a6bb-4bb3-898e-b08205c7ef5e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:44:08.099958 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:08.099934 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97270ed7-a6bb-4bb3-898e-b08205c7ef5e-kube-api-access-xnmdd" (OuterVolumeSpecName: "kube-api-access-xnmdd") pod "97270ed7-a6bb-4bb3-898e-b08205c7ef5e" (UID: "97270ed7-a6bb-4bb3-898e-b08205c7ef5e"). InnerVolumeSpecName "kube-api-access-xnmdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:44:08.102883 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:08.102852 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97270ed7-a6bb-4bb3-898e-b08205c7ef5e-util" (OuterVolumeSpecName: "util") pod "97270ed7-a6bb-4bb3-898e-b08205c7ef5e" (UID: "97270ed7-a6bb-4bb3-898e-b08205c7ef5e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:44:08.198577 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:08.198554 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gl2v4\" (UniqueName: \"kubernetes.io/projected/3ec8f212-80ab-497b-8f4f-da31514f063e-kube-api-access-gl2v4\") pod \"3ec8f212-80ab-497b-8f4f-da31514f063e\" (UID: \"3ec8f212-80ab-497b-8f4f-da31514f063e\") " Jul 22 01:44:08.198716 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:08.198586 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ec8f212-80ab-497b-8f4f-da31514f063e-util\") pod \"3ec8f212-80ab-497b-8f4f-da31514f063e\" (UID: \"3ec8f212-80ab-497b-8f4f-da31514f063e\") " Jul 22 01:44:08.198716 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:08.198677 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ec8f212-80ab-497b-8f4f-da31514f063e-bundle\") pod \"3ec8f212-80ab-497b-8f4f-da31514f063e\" (UID: \"3ec8f212-80ab-497b-8f4f-da31514f063e\") " Jul 22 01:44:08.198946 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:08.198927 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97270ed7-a6bb-4bb3-898e-b08205c7ef5e-util\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:44:08.199013 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:08.198952 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnmdd\" (UniqueName: \"kubernetes.io/projected/97270ed7-a6bb-4bb3-898e-b08205c7ef5e-kube-api-access-xnmdd\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:44:08.199013 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:08.198965 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97270ed7-a6bb-4bb3-898e-b08205c7ef5e-bundle\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:44:08.199227 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:08.199206 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec8f212-80ab-497b-8f4f-da31514f063e-bundle" (OuterVolumeSpecName: "bundle") pod "3ec8f212-80ab-497b-8f4f-da31514f063e" (UID: "3ec8f212-80ab-497b-8f4f-da31514f063e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:44:08.200770 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:08.200744 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec8f212-80ab-497b-8f4f-da31514f063e-kube-api-access-gl2v4" (OuterVolumeSpecName: "kube-api-access-gl2v4") pod "3ec8f212-80ab-497b-8f4f-da31514f063e" (UID: "3ec8f212-80ab-497b-8f4f-da31514f063e"). InnerVolumeSpecName "kube-api-access-gl2v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:44:08.204211 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:08.204191 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec8f212-80ab-497b-8f4f-da31514f063e-util" (OuterVolumeSpecName: "util") pod "3ec8f212-80ab-497b-8f4f-da31514f063e" (UID: "3ec8f212-80ab-497b-8f4f-da31514f063e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:44:08.300099 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:08.300059 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl2v4\" (UniqueName: \"kubernetes.io/projected/3ec8f212-80ab-497b-8f4f-da31514f063e-kube-api-access-gl2v4\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:44:08.300099 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:08.300090 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ec8f212-80ab-497b-8f4f-da31514f063e-util\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:44:08.300099 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:08.300102 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ec8f212-80ab-497b-8f4f-da31514f063e-bundle\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:44:08.920270 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:08.920245 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf" Jul 22 01:44:08.920714 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:08.920289 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf" event={"ID":"3ec8f212-80ab-497b-8f4f-da31514f063e","Type":"ContainerDied","Data":"b1ede701ed13584f822c763cdf88a285b707c334fd45ec530af6e70451b9e18e"} Jul 22 01:44:08.920714 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:08.920331 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1ede701ed13584f822c763cdf88a285b707c334fd45ec530af6e70451b9e18e" Jul 22 01:44:08.922232 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:08.922209 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb" Jul 22 01:44:08.922344 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:08.922239 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb" event={"ID":"97270ed7-a6bb-4bb3-898e-b08205c7ef5e","Type":"ContainerDied","Data":"cf3c55ab65bff10d302b283073a4963a69c09e82370b950e5e57ac48f742d706"} Jul 22 01:44:08.922344 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:08.922283 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf3c55ab65bff10d302b283073a4963a69c09e82370b950e5e57ac48f742d706" Jul 22 01:44:09.066043 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:09.066022 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8" Jul 22 01:44:09.208631 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:09.208589 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57tz7\" (UniqueName: \"kubernetes.io/projected/2dba0d85-6bd2-4b3a-9811-eb5b58d6355b-kube-api-access-57tz7\") pod \"2dba0d85-6bd2-4b3a-9811-eb5b58d6355b\" (UID: \"2dba0d85-6bd2-4b3a-9811-eb5b58d6355b\") " Jul 22 01:44:09.208631 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:09.208637 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2dba0d85-6bd2-4b3a-9811-eb5b58d6355b-bundle\") pod \"2dba0d85-6bd2-4b3a-9811-eb5b58d6355b\" (UID: \"2dba0d85-6bd2-4b3a-9811-eb5b58d6355b\") " Jul 22 01:44:09.208896 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:09.208672 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2dba0d85-6bd2-4b3a-9811-eb5b58d6355b-util\") pod \"2dba0d85-6bd2-4b3a-9811-eb5b58d6355b\" (UID: \"2dba0d85-6bd2-4b3a-9811-eb5b58d6355b\") " Jul 22 01:44:09.209109 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:09.209082 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dba0d85-6bd2-4b3a-9811-eb5b58d6355b-bundle" (OuterVolumeSpecName: "bundle") pod "2dba0d85-6bd2-4b3a-9811-eb5b58d6355b" (UID: "2dba0d85-6bd2-4b3a-9811-eb5b58d6355b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:44:09.210902 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:09.210878 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dba0d85-6bd2-4b3a-9811-eb5b58d6355b-kube-api-access-57tz7" (OuterVolumeSpecName: "kube-api-access-57tz7") pod "2dba0d85-6bd2-4b3a-9811-eb5b58d6355b" (UID: "2dba0d85-6bd2-4b3a-9811-eb5b58d6355b"). InnerVolumeSpecName "kube-api-access-57tz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:44:09.214088 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:09.214069 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dba0d85-6bd2-4b3a-9811-eb5b58d6355b-util" (OuterVolumeSpecName: "util") pod "2dba0d85-6bd2-4b3a-9811-eb5b58d6355b" (UID: "2dba0d85-6bd2-4b3a-9811-eb5b58d6355b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:44:09.309567 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:09.309547 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57tz7\" (UniqueName: \"kubernetes.io/projected/2dba0d85-6bd2-4b3a-9811-eb5b58d6355b-kube-api-access-57tz7\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:44:09.309567 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:09.309569 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2dba0d85-6bd2-4b3a-9811-eb5b58d6355b-bundle\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:44:09.309707 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:09.309578 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2dba0d85-6bd2-4b3a-9811-eb5b58d6355b-util\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:44:09.928459 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:09.928434 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8" Jul 22 01:44:09.928800 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:09.928431 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8" event={"ID":"2dba0d85-6bd2-4b3a-9811-eb5b58d6355b","Type":"ContainerDied","Data":"cb03264b5d8a648c889480973d5586ad562cd94297b7f13047ab0824000d54cd"} Jul 22 01:44:09.928800 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:09.928550 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb03264b5d8a648c889480973d5586ad562cd94297b7f13047ab0824000d54cd" Jul 22 01:44:19.094497 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.094462 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76kwj"] Jul 22 01:44:19.094866 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.094826 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e48954b4-33a3-4744-893d-cceb6c6c6ed0" containerName="extract" Jul 22 01:44:19.094866 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.094837 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e48954b4-33a3-4744-893d-cceb6c6c6ed0" containerName="extract" Jul 22 01:44:19.094866 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.094849 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97270ed7-a6bb-4bb3-898e-b08205c7ef5e" containerName="pull" Jul 22 01:44:19.094866 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.094855 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="97270ed7-a6bb-4bb3-898e-b08205c7ef5e" containerName="pull" Jul 22 01:44:19.094866 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.094861 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dba0d85-6bd2-4b3a-9811-eb5b58d6355b" containerName="util" Jul 22 01:44:19.094866 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.094867 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dba0d85-6bd2-4b3a-9811-eb5b58d6355b" containerName="util" Jul 22 01:44:19.095054 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.094874 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ec8f212-80ab-497b-8f4f-da31514f063e" containerName="extract" Jul 22 01:44:19.095054 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.094880 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ec8f212-80ab-497b-8f4f-da31514f063e" containerName="extract" Jul 22 01:44:19.095054 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.094893 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ec8f212-80ab-497b-8f4f-da31514f063e" containerName="util" Jul 22 01:44:19.095054 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.094898 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ec8f212-80ab-497b-8f4f-da31514f063e" containerName="util" Jul 22 01:44:19.095054 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.094907 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97270ed7-a6bb-4bb3-898e-b08205c7ef5e" containerName="util" Jul 22 01:44:19.095054 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.094913 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="97270ed7-a6bb-4bb3-898e-b08205c7ef5e" containerName="util" Jul 22 01:44:19.095054 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.094918 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e48954b4-33a3-4744-893d-cceb6c6c6ed0" containerName="pull" Jul 22 01:44:19.095054 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.094924 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e48954b4-33a3-4744-893d-cceb6c6c6ed0" containerName="pull" Jul 22 01:44:19.095054 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.094928 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ec8f212-80ab-497b-8f4f-da31514f063e" containerName="pull" Jul 22 01:44:19.095054 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.094933 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ec8f212-80ab-497b-8f4f-da31514f063e" containerName="pull" Jul 22 01:44:19.095054 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.094940 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dba0d85-6bd2-4b3a-9811-eb5b58d6355b" containerName="pull" Jul 22 01:44:19.095054 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.094945 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dba0d85-6bd2-4b3a-9811-eb5b58d6355b" containerName="pull" Jul 22 01:44:19.095054 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.094951 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dba0d85-6bd2-4b3a-9811-eb5b58d6355b" containerName="extract" Jul 22 01:44:19.095054 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.094955 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dba0d85-6bd2-4b3a-9811-eb5b58d6355b" containerName="extract" Jul 22 01:44:19.095054 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.094962 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e48954b4-33a3-4744-893d-cceb6c6c6ed0" containerName="util" Jul 22 01:44:19.095054 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.094966 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e48954b4-33a3-4744-893d-cceb6c6c6ed0" containerName="util" Jul 22 01:44:19.095054 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.094976 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97270ed7-a6bb-4bb3-898e-b08205c7ef5e" containerName="extract" Jul 22 01:44:19.095054 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.094980 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="97270ed7-a6bb-4bb3-898e-b08205c7ef5e" containerName="extract" Jul 22 01:44:19.095054 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.095037 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ec8f212-80ab-497b-8f4f-da31514f063e" containerName="extract" Jul 22 01:44:19.095054 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.095044 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e48954b4-33a3-4744-893d-cceb6c6c6ed0" containerName="extract" Jul 22 01:44:19.095054 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.095050 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dba0d85-6bd2-4b3a-9811-eb5b58d6355b" containerName="extract" Jul 22 01:44:19.095054 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.095056 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="97270ed7-a6bb-4bb3-898e-b08205c7ef5e" containerName="extract" Jul 22 01:44:19.099344 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.099326 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76kwj" Jul 22 01:44:19.101832 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.101811 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-lgckh\"" Jul 22 01:44:19.105669 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.105640 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76kwj"] Jul 22 01:44:19.197537 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.197506 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsr9f\" (UniqueName: \"kubernetes.io/projected/49e335d8-5ebe-4eec-a040-8cfe2ac628b7-kube-api-access-gsr9f\") pod \"limitador-operator-controller-manager-859d7b78bd-76kwj\" (UID: \"49e335d8-5ebe-4eec-a040-8cfe2ac628b7\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76kwj" Jul 22 01:44:19.298256 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.298231 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsr9f\" (UniqueName: \"kubernetes.io/projected/49e335d8-5ebe-4eec-a040-8cfe2ac628b7-kube-api-access-gsr9f\") pod \"limitador-operator-controller-manager-859d7b78bd-76kwj\" (UID: \"49e335d8-5ebe-4eec-a040-8cfe2ac628b7\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76kwj" Jul 22 01:44:19.315360 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.315330 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsr9f\" (UniqueName: \"kubernetes.io/projected/49e335d8-5ebe-4eec-a040-8cfe2ac628b7-kube-api-access-gsr9f\") pod \"limitador-operator-controller-manager-859d7b78bd-76kwj\" (UID: \"49e335d8-5ebe-4eec-a040-8cfe2ac628b7\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76kwj" Jul 22 01:44:19.411105 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.411018 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76kwj" Jul 22 01:44:19.548718 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.548681 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76kwj"] Jul 22 01:44:19.550494 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:44:19.550466 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e335d8_5ebe_4eec_a040_8cfe2ac628b7.slice/crio-4c2f2923f070996b5bd571e82c892a3d7ed20cbc00dab2884a9ee269c0c703a6 WatchSource:0}: Error finding container 4c2f2923f070996b5bd571e82c892a3d7ed20cbc00dab2884a9ee269c0c703a6: Status 404 returned error can't find the container with id 4c2f2923f070996b5bd571e82c892a3d7ed20cbc00dab2884a9ee269c0c703a6 Jul 22 01:44:19.977084 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:19.977050 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76kwj" event={"ID":"49e335d8-5ebe-4eec-a040-8cfe2ac628b7","Type":"ContainerStarted","Data":"4c2f2923f070996b5bd571e82c892a3d7ed20cbc00dab2884a9ee269c0c703a6"} Jul 22 01:44:21.989381 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:21.989338 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76kwj" event={"ID":"49e335d8-5ebe-4eec-a040-8cfe2ac628b7","Type":"ContainerStarted","Data":"d342dee016c3d572973f272ed032f8716eeafdac33163953828c405212afe14a"} Jul 22 01:44:21.989862 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:21.989418 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76kwj" Jul 22 01:44:22.011979 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:22.011918 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76kwj" podStartSLOduration=1.3496886080000001 podStartE2EDuration="3.01190512s" totalImagesPullingTime="1.662216512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:44:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:44:22.008930249 +0000 UTC m=+432.601734612" watchObservedRunningTime="2026-07-22 01:44:22.01190512 +0000 UTC m=+432.604709483" Jul 22 01:44:24.257809 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:24.257776 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk"] Jul 22 01:44:24.265780 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:24.265760 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk" Jul 22 01:44:24.269046 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:24.269023 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zxvtc\"" Jul 22 01:44:24.278977 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:24.278954 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk"] Jul 22 01:44:24.441913 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:24.441887 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e59c1cd1-70ec-4e59-95fb-baff32886c16-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-f8mxk\" (UID: \"e59c1cd1-70ec-4e59-95fb-baff32886c16\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk" Jul 22 01:44:24.442052 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:24.441981 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq2zt\" (UniqueName: \"kubernetes.io/projected/e59c1cd1-70ec-4e59-95fb-baff32886c16-kube-api-access-zq2zt\") pod \"kuadrant-operator-controller-manager-8684c9ff49-f8mxk\" (UID: \"e59c1cd1-70ec-4e59-95fb-baff32886c16\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk" Jul 22 01:44:24.542544 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:24.542466 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq2zt\" (UniqueName: \"kubernetes.io/projected/e59c1cd1-70ec-4e59-95fb-baff32886c16-kube-api-access-zq2zt\") pod \"kuadrant-operator-controller-manager-8684c9ff49-f8mxk\" (UID: \"e59c1cd1-70ec-4e59-95fb-baff32886c16\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk" Jul 22 01:44:24.542544 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:24.542507 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e59c1cd1-70ec-4e59-95fb-baff32886c16-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-f8mxk\" (UID: \"e59c1cd1-70ec-4e59-95fb-baff32886c16\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk" Jul 22 01:44:24.542863 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:24.542845 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e59c1cd1-70ec-4e59-95fb-baff32886c16-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-f8mxk\" (UID: \"e59c1cd1-70ec-4e59-95fb-baff32886c16\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk" Jul 22 01:44:24.560036 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:24.560005 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq2zt\" (UniqueName: \"kubernetes.io/projected/e59c1cd1-70ec-4e59-95fb-baff32886c16-kube-api-access-zq2zt\") pod \"kuadrant-operator-controller-manager-8684c9ff49-f8mxk\" (UID: \"e59c1cd1-70ec-4e59-95fb-baff32886c16\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk" Jul 22 01:44:24.575963 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:24.575940 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk" Jul 22 01:44:24.718334 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:24.718307 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk"] Jul 22 01:44:24.720394 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:44:24.720366 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode59c1cd1_70ec_4e59_95fb_baff32886c16.slice/crio-a82f28d2aaef184fc8ae07e643330508964d351427277cb12b49e5aee2488fc9 WatchSource:0}: Error finding container a82f28d2aaef184fc8ae07e643330508964d351427277cb12b49e5aee2488fc9: Status 404 returned error can't find the container with id a82f28d2aaef184fc8ae07e643330508964d351427277cb12b49e5aee2488fc9 Jul 22 01:44:25.004978 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:25.004940 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk" event={"ID":"e59c1cd1-70ec-4e59-95fb-baff32886c16","Type":"ContainerStarted","Data":"a82f28d2aaef184fc8ae07e643330508964d351427277cb12b49e5aee2488fc9"} Jul 22 01:44:26.887987 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:26.887951 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-hl4dc"] Jul 22 01:44:26.891820 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:26.891800 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hl4dc" Jul 22 01:44:26.898163 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:26.898139 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-v4qvl\"" Jul 22 01:44:26.904291 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:26.904264 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-hl4dc"] Jul 22 01:44:27.065313 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:27.065273 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrwlj\" (UniqueName: \"kubernetes.io/projected/ff897a45-88e9-48e8-a2e7-b9f33d926399-kube-api-access-vrwlj\") pod \"authorino-operator-75cc8d5cb7-hl4dc\" (UID: \"ff897a45-88e9-48e8-a2e7-b9f33d926399\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-hl4dc" Jul 22 01:44:27.166317 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:27.166221 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrwlj\" (UniqueName: \"kubernetes.io/projected/ff897a45-88e9-48e8-a2e7-b9f33d926399-kube-api-access-vrwlj\") pod \"authorino-operator-75cc8d5cb7-hl4dc\" (UID: \"ff897a45-88e9-48e8-a2e7-b9f33d926399\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-hl4dc" Jul 22 01:44:27.180166 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:27.180135 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrwlj\" (UniqueName: \"kubernetes.io/projected/ff897a45-88e9-48e8-a2e7-b9f33d926399-kube-api-access-vrwlj\") pod \"authorino-operator-75cc8d5cb7-hl4dc\" (UID: \"ff897a45-88e9-48e8-a2e7-b9f33d926399\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-hl4dc" Jul 22 01:44:27.205203 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:27.205178 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hl4dc" Jul 22 01:44:27.370601 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:27.370555 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-hl4dc"] Jul 22 01:44:27.377166 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:44:27.377138 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff897a45_88e9_48e8_a2e7_b9f33d926399.slice/crio-b9cf2e742b2b243c8f241054dc8b2799d0ac586277bcd3b051cfc2d172799287 WatchSource:0}: Error finding container b9cf2e742b2b243c8f241054dc8b2799d0ac586277bcd3b051cfc2d172799287: Status 404 returned error can't find the container with id b9cf2e742b2b243c8f241054dc8b2799d0ac586277bcd3b051cfc2d172799287 Jul 22 01:44:28.024812 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:28.024746 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hl4dc" event={"ID":"ff897a45-88e9-48e8-a2e7-b9f33d926399","Type":"ContainerStarted","Data":"b9cf2e742b2b243c8f241054dc8b2799d0ac586277bcd3b051cfc2d172799287"} Jul 22 01:44:30.344395 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:30.344290 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-xn275"] Jul 22 01:44:30.355221 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:30.355199 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xn275" Jul 22 01:44:30.357781 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:30.357758 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-wrscx\"" Jul 22 01:44:30.357936 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:30.357759 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 22 01:44:30.363649 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:30.363578 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-xn275"] Jul 22 01:44:30.499275 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:30.499234 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq68t\" (UniqueName: \"kubernetes.io/projected/8e9c947f-fe51-4870-b616-8fe740c067e5-kube-api-access-xq68t\") pod \"dns-operator-controller-manager-864dd78c4c-xn275\" (UID: \"8e9c947f-fe51-4870-b616-8fe740c067e5\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xn275" Jul 22 01:44:30.599834 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:30.599757 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq68t\" (UniqueName: \"kubernetes.io/projected/8e9c947f-fe51-4870-b616-8fe740c067e5-kube-api-access-xq68t\") pod \"dns-operator-controller-manager-864dd78c4c-xn275\" (UID: \"8e9c947f-fe51-4870-b616-8fe740c067e5\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xn275" Jul 22 01:44:30.611744 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:30.611721 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq68t\" (UniqueName: \"kubernetes.io/projected/8e9c947f-fe51-4870-b616-8fe740c067e5-kube-api-access-xq68t\") pod \"dns-operator-controller-manager-864dd78c4c-xn275\" (UID: \"8e9c947f-fe51-4870-b616-8fe740c067e5\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xn275" Jul 22 01:44:30.669464 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:30.669432 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xn275" Jul 22 01:44:30.957531 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:30.957507 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-xn275"] Jul 22 01:44:30.958973 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:44:30.958947 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e9c947f_fe51_4870_b616_8fe740c067e5.slice/crio-2f7c7e4f856a4cca1ece871bdef73fccbd7e66a7dbcca0a2bcf22c7fa7dc3a98 WatchSource:0}: Error finding container 2f7c7e4f856a4cca1ece871bdef73fccbd7e66a7dbcca0a2bcf22c7fa7dc3a98: Status 404 returned error can't find the container with id 2f7c7e4f856a4cca1ece871bdef73fccbd7e66a7dbcca0a2bcf22c7fa7dc3a98 Jul 22 01:44:31.042986 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:31.042952 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xn275" event={"ID":"8e9c947f-fe51-4870-b616-8fe740c067e5","Type":"ContainerStarted","Data":"2f7c7e4f856a4cca1ece871bdef73fccbd7e66a7dbcca0a2bcf22c7fa7dc3a98"} Jul 22 01:44:31.044661 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:31.044636 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hl4dc" event={"ID":"ff897a45-88e9-48e8-a2e7-b9f33d926399","Type":"ContainerStarted","Data":"556dcd6fb9acef718f5f4b6135b1adcbc099163122025f42f13942052d64beaf"} Jul 22 01:44:31.044795 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:31.044738 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hl4dc" Jul 22 01:44:31.046452 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:31.046428 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk" event={"ID":"e59c1cd1-70ec-4e59-95fb-baff32886c16","Type":"ContainerStarted","Data":"2bc72bdb9d2488710a782602044599ebc47877f8146dbe6e51c351cebe04f627"} Jul 22 01:44:31.046542 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:31.046511 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk" Jul 22 01:44:31.070850 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:31.070804 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hl4dc" podStartSLOduration=1.601739333 podStartE2EDuration="5.070790604s" totalImagesPullingTime="3.469051271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:44:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:44:31.067356506 +0000 UTC m=+441.660160910" watchObservedRunningTime="2026-07-22 01:44:31.070790604 +0000 UTC m=+441.663594966" Jul 22 01:44:31.086297 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:31.086233 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk" podStartSLOduration=0.977608559 podStartE2EDuration="7.086219962s" totalImagesPullingTime="6.108611403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:44:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:44:31.085587734 +0000 UTC m=+441.678392118" watchObservedRunningTime="2026-07-22 01:44:31.086219962 +0000 UTC m=+441.679024324" Jul 22 01:44:32.996207 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:32.996175 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76kwj" Jul 22 01:44:34.069146 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:34.069111 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xn275" event={"ID":"8e9c947f-fe51-4870-b616-8fe740c067e5","Type":"ContainerStarted","Data":"4eb67e92d754c45be260e04c1b93b27679c92c3ca57d966c355253cd981ca140"} Jul 22 01:44:34.069526 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:34.069172 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xn275" Jul 22 01:44:34.090146 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:34.090094 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xn275" podStartSLOduration=1.9552678289999998 podStartE2EDuration="4.090078774s" totalImagesPullingTime="2.134810945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:44:34.087208727 +0000 UTC m=+444.680013110" watchObservedRunningTime="2026-07-22 01:44:34.090078774 +0000 UTC m=+444.682883193" Jul 22 01:44:42.057283 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:42.057251 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hl4dc" Jul 22 01:44:42.057678 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:42.057408 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk" Jul 22 01:44:43.689567 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:43.689534 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-xm5f9"] Jul 22 01:44:43.693005 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:43.692972 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-xm5f9" Jul 22 01:44:43.703534 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:43.703514 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc7rk\" (UniqueName: \"kubernetes.io/projected/26dc87f1-384a-45df-8776-4471e83fbe74-kube-api-access-hc7rk\") pod \"kuadrant-operator-controller-manager-55467d9f7-xm5f9\" (UID: \"26dc87f1-384a-45df-8776-4471e83fbe74\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-xm5f9" Jul 22 01:44:43.703630 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:43.703560 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/26dc87f1-384a-45df-8776-4471e83fbe74-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-xm5f9\" (UID: \"26dc87f1-384a-45df-8776-4471e83fbe74\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-xm5f9" Jul 22 01:44:43.707331 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:43.707247 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-xm5f9"] Jul 22 01:44:43.804497 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:43.804464 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc7rk\" (UniqueName: \"kubernetes.io/projected/26dc87f1-384a-45df-8776-4471e83fbe74-kube-api-access-hc7rk\") pod \"kuadrant-operator-controller-manager-55467d9f7-xm5f9\" (UID: \"26dc87f1-384a-45df-8776-4471e83fbe74\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-xm5f9" Jul 22 01:44:43.804637 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:43.804516 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/26dc87f1-384a-45df-8776-4471e83fbe74-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-xm5f9\" (UID: \"26dc87f1-384a-45df-8776-4471e83fbe74\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-xm5f9" Jul 22 01:44:43.804871 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:43.804854 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/26dc87f1-384a-45df-8776-4471e83fbe74-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-xm5f9\" (UID: \"26dc87f1-384a-45df-8776-4471e83fbe74\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-xm5f9" Jul 22 01:44:43.814461 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:43.814439 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc7rk\" (UniqueName: \"kubernetes.io/projected/26dc87f1-384a-45df-8776-4471e83fbe74-kube-api-access-hc7rk\") pod \"kuadrant-operator-controller-manager-55467d9f7-xm5f9\" (UID: \"26dc87f1-384a-45df-8776-4471e83fbe74\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-xm5f9" Jul 22 01:44:43.945208 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:43.945147 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-xm5f9"] Jul 22 01:44:43.945378 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:43.945366 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-xm5f9" Jul 22 01:44:43.956966 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:43.956940 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-xm5f9"] Jul 22 01:44:43.971709 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:43.970970 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk"] Jul 22 01:44:43.971709 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:43.971259 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk" podUID="e59c1cd1-70ec-4e59-95fb-baff32886c16" containerName="manager" containerID="cri-o://2bc72bdb9d2488710a782602044599ebc47877f8146dbe6e51c351cebe04f627" gracePeriod=2 Jul 22 01:44:43.971884 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:43.971823 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-njhqn"] Jul 22 01:44:43.977188 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:43.977164 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-njhqn" Jul 22 01:44:43.987687 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:44:43.986933 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-f8mxk\" is forbidden: User \"system:node:ip-10-0-131-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-56.ec2.internal' and this object" podUID="e59c1cd1-70ec-4e59-95fb-baff32886c16" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk" Jul 22 01:44:43.987813 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:43.987736 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk"] Jul 22 01:44:43.991822 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:43.991796 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-njhqn"] Jul 22 01:44:43.996066 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:43.996033 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nmlh4"] Jul 22 01:44:43.996805 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:43.996784 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e59c1cd1-70ec-4e59-95fb-baff32886c16" containerName="manager" Jul 22 01:44:43.996879 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:43.996807 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e59c1cd1-70ec-4e59-95fb-baff32886c16" containerName="manager" Jul 22 01:44:43.996917 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:43.996891 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e59c1cd1-70ec-4e59-95fb-baff32886c16" containerName="manager" Jul 22 01:44:43.999236 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:43.999218 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76kwj"] Jul 22 01:44:43.999367 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:43.999349 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nmlh4" Jul 22 01:44:43.999428 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:43.999362 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76kwj" podUID="49e335d8-5ebe-4eec-a040-8cfe2ac628b7" containerName="manager" containerID="cri-o://d342dee016c3d572973f272ed032f8716eeafdac33163953828c405212afe14a" gracePeriod=2 Jul 22 01:44:44.007105 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.007064 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b55ebee-56f7-4c1c-b95e-85ad69d1ca68-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-njhqn\" (UID: \"9b55ebee-56f7-4c1c-b95e-85ad69d1ca68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-njhqn" Jul 22 01:44:44.007225 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.007117 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwfxf\" (UniqueName: \"kubernetes.io/projected/9b55ebee-56f7-4c1c-b95e-85ad69d1ca68-kube-api-access-wwfxf\") pod \"kuadrant-operator-controller-manager-55467d9f7-njhqn\" (UID: \"9b55ebee-56f7-4c1c-b95e-85ad69d1ca68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-njhqn" Jul 22 01:44:44.014972 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.014951 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76kwj"] Jul 22 01:44:44.022382 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.022359 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nmlh4"] Jul 22 01:44:44.035244 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.035220 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nzlv"] Jul 22 01:44:44.035829 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.035808 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49e335d8-5ebe-4eec-a040-8cfe2ac628b7" containerName="manager" Jul 22 01:44:44.035922 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.035830 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="49e335d8-5ebe-4eec-a040-8cfe2ac628b7" containerName="manager" Jul 22 01:44:44.035974 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.035956 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="49e335d8-5ebe-4eec-a040-8cfe2ac628b7" containerName="manager" Jul 22 01:44:44.038444 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.038424 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nzlv" Jul 22 01:44:44.051916 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:44:44.051882 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-76kwj\" is forbidden: User \"system:node:ip-10-0-131-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-56.ec2.internal' and this object" podUID="49e335d8-5ebe-4eec-a040-8cfe2ac628b7" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76kwj" Jul 22 01:44:44.053713 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:44:44.053644 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-f8mxk\" is forbidden: User \"system:node:ip-10-0-131-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-56.ec2.internal' and this object" podUID="e59c1cd1-70ec-4e59-95fb-baff32886c16" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk" Jul 22 01:44:44.055473 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:44:44.055451 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-76kwj\" is forbidden: User \"system:node:ip-10-0-131-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-56.ec2.internal' and this object" podUID="49e335d8-5ebe-4eec-a040-8cfe2ac628b7" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76kwj" Jul 22 01:44:44.057054 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.057030 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nzlv"] Jul 22 01:44:44.057313 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:44:44.057291 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-f8mxk\" is forbidden: User \"system:node:ip-10-0-131-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-56.ec2.internal' and this object" podUID="e59c1cd1-70ec-4e59-95fb-baff32886c16" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk" Jul 22 01:44:44.107800 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.107772 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b55ebee-56f7-4c1c-b95e-85ad69d1ca68-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-njhqn\" (UID: \"9b55ebee-56f7-4c1c-b95e-85ad69d1ca68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-njhqn" Jul 22 01:44:44.107922 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.107817 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwfxf\" (UniqueName: \"kubernetes.io/projected/9b55ebee-56f7-4c1c-b95e-85ad69d1ca68-kube-api-access-wwfxf\") pod \"kuadrant-operator-controller-manager-55467d9f7-njhqn\" (UID: \"9b55ebee-56f7-4c1c-b95e-85ad69d1ca68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-njhqn" Jul 22 01:44:44.107922 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.107893 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhsqm\" (UniqueName: \"kubernetes.io/projected/09fcdea7-4413-45df-a08b-6e05a3bc4243-kube-api-access-xhsqm\") pod \"limitador-operator-controller-manager-859d7b78bd-5nzlv\" (UID: \"09fcdea7-4413-45df-a08b-6e05a3bc4243\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nzlv" Jul 22 01:44:44.108039 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.107964 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a53a039f-7cc9-4132-a639-168f9498322b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nmlh4\" (UID: \"a53a039f-7cc9-4132-a639-168f9498322b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nmlh4" Jul 22 01:44:44.108101 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.108032 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdcnz\" (UniqueName: \"kubernetes.io/projected/a53a039f-7cc9-4132-a639-168f9498322b-kube-api-access-jdcnz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nmlh4\" (UID: \"a53a039f-7cc9-4132-a639-168f9498322b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nmlh4" Jul 22 01:44:44.108221 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.108203 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b55ebee-56f7-4c1c-b95e-85ad69d1ca68-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-njhqn\" (UID: \"9b55ebee-56f7-4c1c-b95e-85ad69d1ca68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-njhqn" Jul 22 01:44:44.123020 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.122997 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwfxf\" (UniqueName: \"kubernetes.io/projected/9b55ebee-56f7-4c1c-b95e-85ad69d1ca68-kube-api-access-wwfxf\") pod \"kuadrant-operator-controller-manager-55467d9f7-njhqn\" (UID: \"9b55ebee-56f7-4c1c-b95e-85ad69d1ca68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-njhqn" Jul 22 01:44:44.127460 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.127436 2561 generic.go:356] "Generic (PLEG): container finished" podID="e59c1cd1-70ec-4e59-95fb-baff32886c16" containerID="2bc72bdb9d2488710a782602044599ebc47877f8146dbe6e51c351cebe04f627" exitCode=0 Jul 22 01:44:44.129773 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.129718 2561 generic.go:356] "Generic (PLEG): container finished" podID="49e335d8-5ebe-4eec-a040-8cfe2ac628b7" containerID="d342dee016c3d572973f272ed032f8716eeafdac33163953828c405212afe14a" exitCode=0 Jul 22 01:44:44.208602 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.208535 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdcnz\" (UniqueName: \"kubernetes.io/projected/a53a039f-7cc9-4132-a639-168f9498322b-kube-api-access-jdcnz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nmlh4\" (UID: \"a53a039f-7cc9-4132-a639-168f9498322b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nmlh4" Jul 22 01:44:44.208760 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.208715 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhsqm\" (UniqueName: \"kubernetes.io/projected/09fcdea7-4413-45df-a08b-6e05a3bc4243-kube-api-access-xhsqm\") pod \"limitador-operator-controller-manager-859d7b78bd-5nzlv\" (UID: \"09fcdea7-4413-45df-a08b-6e05a3bc4243\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nzlv" Jul 22 01:44:44.208829 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.208793 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a53a039f-7cc9-4132-a639-168f9498322b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nmlh4\" (UID: \"a53a039f-7cc9-4132-a639-168f9498322b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nmlh4" Jul 22 01:44:44.209153 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.209137 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a53a039f-7cc9-4132-a639-168f9498322b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nmlh4\" (UID: \"a53a039f-7cc9-4132-a639-168f9498322b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nmlh4" Jul 22 01:44:44.224297 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.224267 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhsqm\" (UniqueName: \"kubernetes.io/projected/09fcdea7-4413-45df-a08b-6e05a3bc4243-kube-api-access-xhsqm\") pod \"limitador-operator-controller-manager-859d7b78bd-5nzlv\" (UID: \"09fcdea7-4413-45df-a08b-6e05a3bc4243\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nzlv" Jul 22 01:44:44.224398 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.224324 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdcnz\" (UniqueName: \"kubernetes.io/projected/a53a039f-7cc9-4132-a639-168f9498322b-kube-api-access-jdcnz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nmlh4\" (UID: \"a53a039f-7cc9-4132-a639-168f9498322b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nmlh4" Jul 22 01:44:44.226490 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.226472 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk" Jul 22 01:44:44.228410 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:44:44.228385 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-76kwj\" is forbidden: User \"system:node:ip-10-0-131-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-56.ec2.internal' and this object" podUID="49e335d8-5ebe-4eec-a040-8cfe2ac628b7" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76kwj" Jul 22 01:44:44.230229 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:44:44.230208 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-f8mxk\" is forbidden: User \"system:node:ip-10-0-131-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-56.ec2.internal' and this object" podUID="e59c1cd1-70ec-4e59-95fb-baff32886c16" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk" Jul 22 01:44:44.256161 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.256142 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76kwj" Jul 22 01:44:44.258066 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:44:44.258039 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-76kwj\" is forbidden: User \"system:node:ip-10-0-131-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-56.ec2.internal' and this object" podUID="49e335d8-5ebe-4eec-a040-8cfe2ac628b7" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76kwj" Jul 22 01:44:44.259881 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:44:44.259863 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-f8mxk\" is forbidden: User \"system:node:ip-10-0-131-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-56.ec2.internal' and this object" podUID="e59c1cd1-70ec-4e59-95fb-baff32886c16" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk" Jul 22 01:44:44.309135 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.309115 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e59c1cd1-70ec-4e59-95fb-baff32886c16-extensions-socket-volume\") pod \"e59c1cd1-70ec-4e59-95fb-baff32886c16\" (UID: \"e59c1cd1-70ec-4e59-95fb-baff32886c16\") " Jul 22 01:44:44.309255 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.309156 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zq2zt\" (UniqueName: \"kubernetes.io/projected/e59c1cd1-70ec-4e59-95fb-baff32886c16-kube-api-access-zq2zt\") pod \"e59c1cd1-70ec-4e59-95fb-baff32886c16\" (UID: \"e59c1cd1-70ec-4e59-95fb-baff32886c16\") " Jul 22 01:44:44.309255 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.309175 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsr9f\" (UniqueName: \"kubernetes.io/projected/49e335d8-5ebe-4eec-a040-8cfe2ac628b7-kube-api-access-gsr9f\") pod \"49e335d8-5ebe-4eec-a040-8cfe2ac628b7\" (UID: \"49e335d8-5ebe-4eec-a040-8cfe2ac628b7\") " Jul 22 01:44:44.309610 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.309586 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e59c1cd1-70ec-4e59-95fb-baff32886c16-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e59c1cd1-70ec-4e59-95fb-baff32886c16" (UID: "e59c1cd1-70ec-4e59-95fb-baff32886c16"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:44:44.311194 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.311177 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e59c1cd1-70ec-4e59-95fb-baff32886c16-kube-api-access-zq2zt" (OuterVolumeSpecName: "kube-api-access-zq2zt") pod "e59c1cd1-70ec-4e59-95fb-baff32886c16" (UID: "e59c1cd1-70ec-4e59-95fb-baff32886c16"). InnerVolumeSpecName "kube-api-access-zq2zt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:44:44.311308 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.311289 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e335d8-5ebe-4eec-a040-8cfe2ac628b7-kube-api-access-gsr9f" (OuterVolumeSpecName: "kube-api-access-gsr9f") pod "49e335d8-5ebe-4eec-a040-8cfe2ac628b7" (UID: "49e335d8-5ebe-4eec-a040-8cfe2ac628b7"). InnerVolumeSpecName "kube-api-access-gsr9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:44:44.410011 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.409991 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsr9f\" (UniqueName: \"kubernetes.io/projected/49e335d8-5ebe-4eec-a040-8cfe2ac628b7-kube-api-access-gsr9f\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:44:44.410011 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.410013 2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e59c1cd1-70ec-4e59-95fb-baff32886c16-extensions-socket-volume\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:44:44.410135 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.410022 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq2zt\" (UniqueName: \"kubernetes.io/projected/e59c1cd1-70ec-4e59-95fb-baff32886c16-kube-api-access-zq2zt\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:44:44.415877 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.415848 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-njhqn" Jul 22 01:44:44.425364 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.424570 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nmlh4" Jul 22 01:44:44.432504 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.432481 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nzlv" Jul 22 01:44:44.582960 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.582934 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-njhqn"] Jul 22 01:44:44.584906 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:44:44.584878 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b55ebee_56f7_4c1c_b95e_85ad69d1ca68.slice/crio-f1d60134b7aa177431cd5a89ed643a70eb2bfbde43c793db8799a779431f381b WatchSource:0}: Error finding container f1d60134b7aa177431cd5a89ed643a70eb2bfbde43c793db8799a779431f381b: Status 404 returned error can't find the container with id f1d60134b7aa177431cd5a89ed643a70eb2bfbde43c793db8799a779431f381b Jul 22 01:44:44.820540 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.820512 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nmlh4"] Jul 22 01:44:44.821918 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:44:44.821891 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53a039f_7cc9_4132_a639_168f9498322b.slice/crio-b891414ba2cf802fc79a04cb6bd5918285d22896a5988938a00ae5396d8bb1ab WatchSource:0}: Error finding container b891414ba2cf802fc79a04cb6bd5918285d22896a5988938a00ae5396d8bb1ab: Status 404 returned error can't find the container with id b891414ba2cf802fc79a04cb6bd5918285d22896a5988938a00ae5396d8bb1ab Jul 22 01:44:44.822675 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:44:44.822649 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09fcdea7_4413_45df_a08b_6e05a3bc4243.slice/crio-e248d6fe9a254bf7b52fc579faf4ffbda199a04d88db15961014e8d628ce37b4 WatchSource:0}: Error finding container e248d6fe9a254bf7b52fc579faf4ffbda199a04d88db15961014e8d628ce37b4: Status 404 returned error can't find the container with id e248d6fe9a254bf7b52fc579faf4ffbda199a04d88db15961014e8d628ce37b4 Jul 22 01:44:44.823043 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:44.823017 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nzlv"] Jul 22 01:44:45.076770 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:45.076665 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xn275" Jul 22 01:44:45.095006 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:44:45.094977 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-76kwj\" is forbidden: User \"system:node:ip-10-0-131-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-56.ec2.internal' and this object" podUID="49e335d8-5ebe-4eec-a040-8cfe2ac628b7" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76kwj" Jul 22 01:44:45.096606 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:44:45.096569 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-f8mxk\" is forbidden: User \"system:node:ip-10-0-131-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-56.ec2.internal' and this object" podUID="e59c1cd1-70ec-4e59-95fb-baff32886c16" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk" Jul 22 01:44:45.140360 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:45.140160 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk" Jul 22 01:44:45.140360 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:45.140224 2561 scope.go:117] "RemoveContainer" containerID="2bc72bdb9d2488710a782602044599ebc47877f8146dbe6e51c351cebe04f627" Jul 22 01:44:45.141848 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:45.141743 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76kwj" Jul 22 01:44:45.142626 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:44:45.142516 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-76kwj\" is forbidden: User \"system:node:ip-10-0-131-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-56.ec2.internal' and this object" podUID="49e335d8-5ebe-4eec-a040-8cfe2ac628b7" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76kwj" Jul 22 01:44:45.143600 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:45.143577 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nzlv" event={"ID":"09fcdea7-4413-45df-a08b-6e05a3bc4243","Type":"ContainerStarted","Data":"27115cf7af0ec62e9b8997c35b78eed5d002a6721bcde605dfca5e8861d2d1ab"} Jul 22 01:44:45.143736 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:45.143626 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nzlv" event={"ID":"09fcdea7-4413-45df-a08b-6e05a3bc4243","Type":"ContainerStarted","Data":"e248d6fe9a254bf7b52fc579faf4ffbda199a04d88db15961014e8d628ce37b4"} Jul 22 01:44:45.144066 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:45.144000 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nzlv" Jul 22 01:44:45.144416 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:44:45.144314 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-f8mxk\" is forbidden: User \"system:node:ip-10-0-131-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-56.ec2.internal' and this object" podUID="e59c1cd1-70ec-4e59-95fb-baff32886c16" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk" Jul 22 01:44:45.146162 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:45.145868 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nmlh4" event={"ID":"a53a039f-7cc9-4132-a639-168f9498322b","Type":"ContainerStarted","Data":"7d09e678f7ef9b26efd8a0527818434d52c0658aeb9b5fdb84b6d2481ec1a147"} Jul 22 01:44:45.146162 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:45.145924 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nmlh4" Jul 22 01:44:45.146162 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:45.145948 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nmlh4" event={"ID":"a53a039f-7cc9-4132-a639-168f9498322b","Type":"ContainerStarted","Data":"b891414ba2cf802fc79a04cb6bd5918285d22896a5988938a00ae5396d8bb1ab"} Jul 22 01:44:45.146162 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:44:45.146044 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-76kwj\" is forbidden: User \"system:node:ip-10-0-131-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-56.ec2.internal' and this object" podUID="49e335d8-5ebe-4eec-a040-8cfe2ac628b7" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76kwj" Jul 22 01:44:45.147820 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:44:45.147686 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-f8mxk\" is forbidden: User \"system:node:ip-10-0-131-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-56.ec2.internal' and this object" podUID="e59c1cd1-70ec-4e59-95fb-baff32886c16" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk" Jul 22 01:44:45.148486 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:45.148372 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-njhqn" event={"ID":"9b55ebee-56f7-4c1c-b95e-85ad69d1ca68","Type":"ContainerStarted","Data":"a44a98dc298b183dc3c917779485d0682f4a73b8f022abe0981c1f01caf84a6b"} Jul 22 01:44:45.148486 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:45.148420 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-njhqn" event={"ID":"9b55ebee-56f7-4c1c-b95e-85ad69d1ca68","Type":"ContainerStarted","Data":"f1d60134b7aa177431cd5a89ed643a70eb2bfbde43c793db8799a779431f381b"} Jul 22 01:44:45.148486 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:45.148448 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-njhqn" Jul 22 01:44:45.152686 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:45.152670 2561 scope.go:117] "RemoveContainer" containerID="d342dee016c3d572973f272ed032f8716eeafdac33163953828c405212afe14a" Jul 22 01:44:45.163347 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:45.163185 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nzlv" podStartSLOduration=1.163170466 podStartE2EDuration="1.163170466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:44:45.16300402 +0000 UTC m=+455.755808403" watchObservedRunningTime="2026-07-22 01:44:45.163170466 +0000 UTC m=+455.755974829" Jul 22 01:44:45.198994 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:45.198952 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nmlh4" podStartSLOduration=2.198939194 podStartE2EDuration="2.198939194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:44:45.198033573 +0000 UTC m=+455.790837969" watchObservedRunningTime="2026-07-22 01:44:45.198939194 +0000 UTC m=+455.791743557" Jul 22 01:44:45.200196 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:45.200165 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-njhqn" podStartSLOduration=2.200155387 podStartE2EDuration="2.200155387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:44:45.180033325 +0000 UTC m=+455.772837688" watchObservedRunningTime="2026-07-22 01:44:45.200155387 +0000 UTC m=+455.792959750" Jul 22 01:44:45.200320 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:44:45.200297 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-76kwj\" is forbidden: User \"system:node:ip-10-0-131-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-56.ec2.internal' and this object" podUID="49e335d8-5ebe-4eec-a040-8cfe2ac628b7" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-76kwj" Jul 22 01:44:45.202148 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:44:45.202127 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-f8mxk\" is forbidden: User \"system:node:ip-10-0-131-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-56.ec2.internal' and this object" podUID="e59c1cd1-70ec-4e59-95fb-baff32886c16" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f8mxk" Jul 22 01:44:45.926550 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:45.926511 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49e335d8-5ebe-4eec-a040-8cfe2ac628b7" path="/var/lib/kubelet/pods/49e335d8-5ebe-4eec-a040-8cfe2ac628b7/volumes" Jul 22 01:44:45.926917 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:45.926854 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e59c1cd1-70ec-4e59-95fb-baff32886c16" path="/var/lib/kubelet/pods/e59c1cd1-70ec-4e59-95fb-baff32886c16/volumes" Jul 22 01:44:46.497387 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:44:46.497351 2561 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-xm5f9_kuadrant-system_26dc87f1-384a-45df-8776-4471e83fbe74_0(84a5090aabca702a165d5641ef6314539c2cac5185b16e8df5b31ad6c3ae6f6f): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-xm5f9 to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 22 01:44:46.497461 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:44:46.497415 2561 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-xm5f9_kuadrant-system_26dc87f1-384a-45df-8776-4471e83fbe74_0(84a5090aabca702a165d5641ef6314539c2cac5185b16e8df5b31ad6c3ae6f6f): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-xm5f9 to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-xm5f9" Jul 22 01:44:47.162056 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:47.162024 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-xm5f9" Jul 22 01:44:47.168635 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:47.168612 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-xm5f9" Jul 22 01:44:47.170874 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:44:47.170850 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-xm5f9\" is forbidden: User \"system:node:ip-10-0-131-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-56.ec2.internal' and this object" podUID="26dc87f1-384a-45df-8776-4471e83fbe74" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-xm5f9" Jul 22 01:44:47.233760 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:47.233729 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/26dc87f1-384a-45df-8776-4471e83fbe74-extensions-socket-volume\") pod \"26dc87f1-384a-45df-8776-4471e83fbe74\" (UID: \"26dc87f1-384a-45df-8776-4471e83fbe74\") " Jul 22 01:44:47.233895 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:47.233797 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hc7rk\" (UniqueName: \"kubernetes.io/projected/26dc87f1-384a-45df-8776-4471e83fbe74-kube-api-access-hc7rk\") pod \"26dc87f1-384a-45df-8776-4471e83fbe74\" (UID: \"26dc87f1-384a-45df-8776-4471e83fbe74\") " Jul 22 01:44:47.233998 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:47.233978 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26dc87f1-384a-45df-8776-4471e83fbe74-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "26dc87f1-384a-45df-8776-4471e83fbe74" (UID: "26dc87f1-384a-45df-8776-4471e83fbe74"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:44:47.234054 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:47.234043 2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/26dc87f1-384a-45df-8776-4471e83fbe74-extensions-socket-volume\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:44:47.235975 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:47.235952 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26dc87f1-384a-45df-8776-4471e83fbe74-kube-api-access-hc7rk" (OuterVolumeSpecName: "kube-api-access-hc7rk") pod "26dc87f1-384a-45df-8776-4471e83fbe74" (UID: "26dc87f1-384a-45df-8776-4471e83fbe74"). InnerVolumeSpecName "kube-api-access-hc7rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:44:47.335405 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:47.335376 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc7rk\" (UniqueName: \"kubernetes.io/projected/26dc87f1-384a-45df-8776-4471e83fbe74-kube-api-access-hc7rk\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:44:47.925352 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:47.925316 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26dc87f1-384a-45df-8776-4471e83fbe74" path="/var/lib/kubelet/pods/26dc87f1-384a-45df-8776-4471e83fbe74/volumes" Jul 22 01:44:48.167980 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:48.167903 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-xm5f9" Jul 22 01:44:48.172210 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:44:48.172180 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-xm5f9\" is forbidden: User \"system:node:ip-10-0-131-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-56.ec2.internal' and this object" podUID="26dc87f1-384a-45df-8776-4471e83fbe74" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-xm5f9" Jul 22 01:44:49.923503 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:44:49.923458 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-xm5f9\" is forbidden: User \"system:node:ip-10-0-131-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-56.ec2.internal' and this object" podUID="26dc87f1-384a-45df-8776-4471e83fbe74" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-xm5f9" Jul 22 01:44:56.159633 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:56.159602 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5nzlv" Jul 22 01:44:56.160079 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:56.159670 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-njhqn" Jul 22 01:44:56.160079 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:56.159820 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nmlh4" Jul 22 01:44:56.216875 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:56.216847 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nmlh4"] Jul 22 01:44:56.217231 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:56.217194 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nmlh4" podUID="a53a039f-7cc9-4132-a639-168f9498322b" containerName="manager" containerID="cri-o://7d09e678f7ef9b26efd8a0527818434d52c0658aeb9b5fdb84b6d2481ec1a147" gracePeriod=10 Jul 22 01:44:56.475324 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:56.475303 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nmlh4" Jul 22 01:44:56.512634 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:56.512606 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdcnz\" (UniqueName: \"kubernetes.io/projected/a53a039f-7cc9-4132-a639-168f9498322b-kube-api-access-jdcnz\") pod \"a53a039f-7cc9-4132-a639-168f9498322b\" (UID: \"a53a039f-7cc9-4132-a639-168f9498322b\") " Jul 22 01:44:56.512804 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:56.512674 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a53a039f-7cc9-4132-a639-168f9498322b-extensions-socket-volume\") pod \"a53a039f-7cc9-4132-a639-168f9498322b\" (UID: \"a53a039f-7cc9-4132-a639-168f9498322b\") " Jul 22 01:44:56.513041 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:56.513003 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53a039f-7cc9-4132-a639-168f9498322b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a53a039f-7cc9-4132-a639-168f9498322b" (UID: "a53a039f-7cc9-4132-a639-168f9498322b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:44:56.515387 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:56.515355 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53a039f-7cc9-4132-a639-168f9498322b-kube-api-access-jdcnz" (OuterVolumeSpecName: "kube-api-access-jdcnz") pod "a53a039f-7cc9-4132-a639-168f9498322b" (UID: "a53a039f-7cc9-4132-a639-168f9498322b"). InnerVolumeSpecName "kube-api-access-jdcnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:44:56.567521 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:56.567488 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55c7m"] Jul 22 01:44:56.567912 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:56.567900 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a53a039f-7cc9-4132-a639-168f9498322b" containerName="manager" Jul 22 01:44:56.567973 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:56.567916 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a53a039f-7cc9-4132-a639-168f9498322b" containerName="manager" Jul 22 01:44:56.568008 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:56.567983 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a53a039f-7cc9-4132-a639-168f9498322b" containerName="manager" Jul 22 01:44:56.571160 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:56.571145 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55c7m" Jul 22 01:44:56.583160 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:56.583135 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55c7m"] Jul 22 01:44:56.614040 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:56.614008 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq68g\" (UniqueName: \"kubernetes.io/projected/378be3c0-a6b3-4335-836d-c7d1a461750d-kube-api-access-mq68g\") pod \"kuadrant-operator-controller-manager-86878c769f-55c7m\" (UID: \"378be3c0-a6b3-4335-836d-c7d1a461750d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55c7m" Jul 22 01:44:56.614186 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:56.614050 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/378be3c0-a6b3-4335-836d-c7d1a461750d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-55c7m\" (UID: \"378be3c0-a6b3-4335-836d-c7d1a461750d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55c7m" Jul 22 01:44:56.614186 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:56.614120 2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a53a039f-7cc9-4132-a639-168f9498322b-extensions-socket-volume\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:44:56.614186 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:56.614130 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdcnz\" (UniqueName: \"kubernetes.io/projected/a53a039f-7cc9-4132-a639-168f9498322b-kube-api-access-jdcnz\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:44:56.715431 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:56.715339 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq68g\" (UniqueName: \"kubernetes.io/projected/378be3c0-a6b3-4335-836d-c7d1a461750d-kube-api-access-mq68g\") pod \"kuadrant-operator-controller-manager-86878c769f-55c7m\" (UID: \"378be3c0-a6b3-4335-836d-c7d1a461750d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55c7m" Jul 22 01:44:56.715431 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:56.715394 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/378be3c0-a6b3-4335-836d-c7d1a461750d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-55c7m\" (UID: \"378be3c0-a6b3-4335-836d-c7d1a461750d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55c7m" Jul 22 01:44:56.715854 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:56.715834 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/378be3c0-a6b3-4335-836d-c7d1a461750d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-55c7m\" (UID: \"378be3c0-a6b3-4335-836d-c7d1a461750d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55c7m" Jul 22 01:44:56.727943 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:56.727921 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq68g\" (UniqueName: \"kubernetes.io/projected/378be3c0-a6b3-4335-836d-c7d1a461750d-kube-api-access-mq68g\") pod \"kuadrant-operator-controller-manager-86878c769f-55c7m\" (UID: \"378be3c0-a6b3-4335-836d-c7d1a461750d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55c7m" Jul 22 01:44:56.882816 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:56.882778 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55c7m" Jul 22 01:44:57.026673 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:57.026650 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55c7m"] Jul 22 01:44:57.029609 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:44:57.029582 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod378be3c0_a6b3_4335_836d_c7d1a461750d.slice/crio-b728a720e21cb21d221ddc6e199e6ef375d0bd6044b89874649e2e6f1f59e131 WatchSource:0}: Error finding container b728a720e21cb21d221ddc6e199e6ef375d0bd6044b89874649e2e6f1f59e131: Status 404 returned error can't find the container with id b728a720e21cb21d221ddc6e199e6ef375d0bd6044b89874649e2e6f1f59e131 Jul 22 01:44:57.226944 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:57.226854 2561 generic.go:356] "Generic (PLEG): container finished" podID="a53a039f-7cc9-4132-a639-168f9498322b" containerID="7d09e678f7ef9b26efd8a0527818434d52c0658aeb9b5fdb84b6d2481ec1a147" exitCode=0 Jul 22 01:44:57.226944 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:57.226895 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nmlh4" Jul 22 01:44:57.226944 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:57.226933 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nmlh4" event={"ID":"a53a039f-7cc9-4132-a639-168f9498322b","Type":"ContainerDied","Data":"7d09e678f7ef9b26efd8a0527818434d52c0658aeb9b5fdb84b6d2481ec1a147"} Jul 22 01:44:57.227465 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:57.226983 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nmlh4" event={"ID":"a53a039f-7cc9-4132-a639-168f9498322b","Type":"ContainerDied","Data":"b891414ba2cf802fc79a04cb6bd5918285d22896a5988938a00ae5396d8bb1ab"} Jul 22 01:44:57.227465 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:57.227016 2561 scope.go:117] "RemoveContainer" containerID="7d09e678f7ef9b26efd8a0527818434d52c0658aeb9b5fdb84b6d2481ec1a147" Jul 22 01:44:57.229599 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:57.229565 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55c7m" event={"ID":"378be3c0-a6b3-4335-836d-c7d1a461750d","Type":"ContainerStarted","Data":"59a076a0377c6d09f0bd92fb7b2cfc540a486e86d2fa9e81c993eee2db54258e"} Jul 22 01:44:57.229769 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:57.229632 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55c7m" Jul 22 01:44:57.229769 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:57.229656 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55c7m" event={"ID":"378be3c0-a6b3-4335-836d-c7d1a461750d","Type":"ContainerStarted","Data":"b728a720e21cb21d221ddc6e199e6ef375d0bd6044b89874649e2e6f1f59e131"} Jul 22 01:44:57.244538 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:57.244512 2561 scope.go:117] "RemoveContainer" containerID="7d09e678f7ef9b26efd8a0527818434d52c0658aeb9b5fdb84b6d2481ec1a147" Jul 22 01:44:57.244847 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:44:57.244827 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d09e678f7ef9b26efd8a0527818434d52c0658aeb9b5fdb84b6d2481ec1a147\": container with ID starting with 7d09e678f7ef9b26efd8a0527818434d52c0658aeb9b5fdb84b6d2481ec1a147 not found: ID does not exist" containerID="7d09e678f7ef9b26efd8a0527818434d52c0658aeb9b5fdb84b6d2481ec1a147" Jul 22 01:44:57.244922 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:57.244861 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d09e678f7ef9b26efd8a0527818434d52c0658aeb9b5fdb84b6d2481ec1a147"} err="failed to get container status \"7d09e678f7ef9b26efd8a0527818434d52c0658aeb9b5fdb84b6d2481ec1a147\": rpc error: code = NotFound desc = could not find container \"7d09e678f7ef9b26efd8a0527818434d52c0658aeb9b5fdb84b6d2481ec1a147\": container with ID starting with 7d09e678f7ef9b26efd8a0527818434d52c0658aeb9b5fdb84b6d2481ec1a147 not found: ID does not exist" Jul 22 01:44:57.261117 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:57.261074 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55c7m" podStartSLOduration=1.261059117 podStartE2EDuration="1.261059117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:44:57.258295153 +0000 UTC m=+467.851099537" watchObservedRunningTime="2026-07-22 01:44:57.261059117 +0000 UTC m=+467.853863480" Jul 22 01:44:57.275948 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:57.275912 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nmlh4"] Jul 22 01:44:57.284030 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:57.284007 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nmlh4"] Jul 22 01:44:57.930102 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:44:57.930061 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a53a039f-7cc9-4132-a639-168f9498322b" path="/var/lib/kubelet/pods/a53a039f-7cc9-4132-a639-168f9498322b/volumes" Jul 22 01:45:08.238316 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:08.238280 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55c7m" Jul 22 01:45:08.293251 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:08.293220 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-njhqn"] Jul 22 01:45:08.293453 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:08.293425 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-njhqn" podUID="9b55ebee-56f7-4c1c-b95e-85ad69d1ca68" containerName="manager" containerID="cri-o://a44a98dc298b183dc3c917779485d0682f4a73b8f022abe0981c1f01caf84a6b" gracePeriod=10 Jul 22 01:45:08.552186 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:08.552162 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-njhqn" Jul 22 01:45:08.618928 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:08.618899 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwfxf\" (UniqueName: \"kubernetes.io/projected/9b55ebee-56f7-4c1c-b95e-85ad69d1ca68-kube-api-access-wwfxf\") pod \"9b55ebee-56f7-4c1c-b95e-85ad69d1ca68\" (UID: \"9b55ebee-56f7-4c1c-b95e-85ad69d1ca68\") " Jul 22 01:45:08.619078 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:08.619048 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b55ebee-56f7-4c1c-b95e-85ad69d1ca68-extensions-socket-volume\") pod \"9b55ebee-56f7-4c1c-b95e-85ad69d1ca68\" (UID: \"9b55ebee-56f7-4c1c-b95e-85ad69d1ca68\") " Jul 22 01:45:08.619410 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:08.619385 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b55ebee-56f7-4c1c-b95e-85ad69d1ca68-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9b55ebee-56f7-4c1c-b95e-85ad69d1ca68" (UID: "9b55ebee-56f7-4c1c-b95e-85ad69d1ca68"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:45:08.621100 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:08.621078 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b55ebee-56f7-4c1c-b95e-85ad69d1ca68-kube-api-access-wwfxf" (OuterVolumeSpecName: "kube-api-access-wwfxf") pod "9b55ebee-56f7-4c1c-b95e-85ad69d1ca68" (UID: "9b55ebee-56f7-4c1c-b95e-85ad69d1ca68"). InnerVolumeSpecName "kube-api-access-wwfxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:45:08.720338 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:08.720305 2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b55ebee-56f7-4c1c-b95e-85ad69d1ca68-extensions-socket-volume\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:45:08.720338 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:08.720332 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwfxf\" (UniqueName: \"kubernetes.io/projected/9b55ebee-56f7-4c1c-b95e-85ad69d1ca68-kube-api-access-wwfxf\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:45:09.300572 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:09.300542 2561 generic.go:356] "Generic (PLEG): container finished" podID="9b55ebee-56f7-4c1c-b95e-85ad69d1ca68" containerID="a44a98dc298b183dc3c917779485d0682f4a73b8f022abe0981c1f01caf84a6b" exitCode=0 Jul 22 01:45:09.300977 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:09.300588 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-njhqn" Jul 22 01:45:09.300977 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:09.300634 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-njhqn" event={"ID":"9b55ebee-56f7-4c1c-b95e-85ad69d1ca68","Type":"ContainerDied","Data":"a44a98dc298b183dc3c917779485d0682f4a73b8f022abe0981c1f01caf84a6b"} Jul 22 01:45:09.300977 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:09.300688 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-njhqn" event={"ID":"9b55ebee-56f7-4c1c-b95e-85ad69d1ca68","Type":"ContainerDied","Data":"f1d60134b7aa177431cd5a89ed643a70eb2bfbde43c793db8799a779431f381b"} Jul 22 01:45:09.300977 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:09.300717 2561 scope.go:117] "RemoveContainer" containerID="a44a98dc298b183dc3c917779485d0682f4a73b8f022abe0981c1f01caf84a6b" Jul 22 01:45:09.316686 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:09.316663 2561 scope.go:117] "RemoveContainer" containerID="a44a98dc298b183dc3c917779485d0682f4a73b8f022abe0981c1f01caf84a6b" Jul 22 01:45:09.317064 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:45:09.317043 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a44a98dc298b183dc3c917779485d0682f4a73b8f022abe0981c1f01caf84a6b\": container with ID starting with a44a98dc298b183dc3c917779485d0682f4a73b8f022abe0981c1f01caf84a6b not found: ID does not exist" containerID="a44a98dc298b183dc3c917779485d0682f4a73b8f022abe0981c1f01caf84a6b" Jul 22 01:45:09.317173 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:09.317070 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a44a98dc298b183dc3c917779485d0682f4a73b8f022abe0981c1f01caf84a6b"} err="failed to get container status \"a44a98dc298b183dc3c917779485d0682f4a73b8f022abe0981c1f01caf84a6b\": rpc error: code = NotFound desc = could not find container \"a44a98dc298b183dc3c917779485d0682f4a73b8f022abe0981c1f01caf84a6b\": container with ID starting with a44a98dc298b183dc3c917779485d0682f4a73b8f022abe0981c1f01caf84a6b not found: ID does not exist" Jul 22 01:45:09.328654 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:09.328630 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-njhqn"] Jul 22 01:45:09.334910 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:09.334887 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-njhqn"] Jul 22 01:45:09.925159 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:09.925129 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b55ebee-56f7-4c1c-b95e-85ad69d1ca68" path="/var/lib/kubelet/pods/9b55ebee-56f7-4c1c-b95e-85ad69d1ca68/volumes" Jul 22 01:45:17.084987 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:17.084955 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f5fc4bdf6-xvr87"] Jul 22 01:45:27.220761 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:27.220728 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kwh5x"] Jul 22 01:45:27.221146 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:27.221127 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b55ebee-56f7-4c1c-b95e-85ad69d1ca68" containerName="manager" Jul 22 01:45:27.221146 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:27.221137 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b55ebee-56f7-4c1c-b95e-85ad69d1ca68" containerName="manager" Jul 22 01:45:27.221250 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:27.221195 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b55ebee-56f7-4c1c-b95e-85ad69d1ca68" containerName="manager" Jul 22 01:45:27.224493 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:27.224475 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-kwh5x" Jul 22 01:45:27.227739 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:27.227672 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 22 01:45:27.227871 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:27.227792 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2f55z\"" Jul 22 01:45:27.232614 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:27.232582 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kwh5x"] Jul 22 01:45:27.319587 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:27.319547 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kwh5x"] Jul 22 01:45:27.376258 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:27.376226 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf9zz\" (UniqueName: \"kubernetes.io/projected/5a55497f-0c13-487c-8a73-92ce2e459ad8-kube-api-access-tf9zz\") pod \"limitador-limitador-64c94fd64f-kwh5x\" (UID: \"5a55497f-0c13-487c-8a73-92ce2e459ad8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kwh5x" Jul 22 01:45:27.376472 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:27.376272 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5a55497f-0c13-487c-8a73-92ce2e459ad8-config-file\") pod \"limitador-limitador-64c94fd64f-kwh5x\" (UID: \"5a55497f-0c13-487c-8a73-92ce2e459ad8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kwh5x" Jul 22 01:45:27.477746 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:27.477644 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf9zz\" (UniqueName: \"kubernetes.io/projected/5a55497f-0c13-487c-8a73-92ce2e459ad8-kube-api-access-tf9zz\") pod \"limitador-limitador-64c94fd64f-kwh5x\" (UID: \"5a55497f-0c13-487c-8a73-92ce2e459ad8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kwh5x" Jul 22 01:45:27.477746 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:27.477690 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5a55497f-0c13-487c-8a73-92ce2e459ad8-config-file\") pod \"limitador-limitador-64c94fd64f-kwh5x\" (UID: \"5a55497f-0c13-487c-8a73-92ce2e459ad8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kwh5x" Jul 22 01:45:27.478264 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:27.478246 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5a55497f-0c13-487c-8a73-92ce2e459ad8-config-file\") pod \"limitador-limitador-64c94fd64f-kwh5x\" (UID: \"5a55497f-0c13-487c-8a73-92ce2e459ad8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kwh5x" Jul 22 01:45:27.487909 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:27.487880 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf9zz\" (UniqueName: \"kubernetes.io/projected/5a55497f-0c13-487c-8a73-92ce2e459ad8-kube-api-access-tf9zz\") pod \"limitador-limitador-64c94fd64f-kwh5x\" (UID: \"5a55497f-0c13-487c-8a73-92ce2e459ad8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kwh5x" Jul 22 01:45:27.536223 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:27.536189 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-kwh5x" Jul 22 01:45:27.671738 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:27.671689 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kwh5x"] Jul 22 01:45:27.673569 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:45:27.673544 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a55497f_0c13_487c_8a73_92ce2e459ad8.slice/crio-0007ccdc7b0dac3bd7b1519f0d0f753223f3f96a34abcdae91471426ee85af00 WatchSource:0}: Error finding container 0007ccdc7b0dac3bd7b1519f0d0f753223f3f96a34abcdae91471426ee85af00: Status 404 returned error can't find the container with id 0007ccdc7b0dac3bd7b1519f0d0f753223f3f96a34abcdae91471426ee85af00 Jul 22 01:45:28.287747 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:28.287707 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-q4ldl"] Jul 22 01:45:28.291163 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:28.291143 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-q4ldl" Jul 22 01:45:28.293835 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:28.293806 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7qvb4\"" Jul 22 01:45:28.297069 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:28.297022 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q4ldl"] Jul 22 01:45:28.385761 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:28.385570 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8q9b\" (UniqueName: \"kubernetes.io/projected/2734ec20-91f9-40f7-b209-7b39aaebeeec-kube-api-access-v8q9b\") pod \"authorino-75d556bd77-q4ldl\" (UID: \"2734ec20-91f9-40f7-b209-7b39aaebeeec\") " pod="kuadrant-system/authorino-75d556bd77-q4ldl" Jul 22 01:45:28.405823 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:28.405788 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-kwh5x" event={"ID":"5a55497f-0c13-487c-8a73-92ce2e459ad8","Type":"ContainerStarted","Data":"0007ccdc7b0dac3bd7b1519f0d0f753223f3f96a34abcdae91471426ee85af00"} Jul 22 01:45:28.487278 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:28.486818 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8q9b\" (UniqueName: \"kubernetes.io/projected/2734ec20-91f9-40f7-b209-7b39aaebeeec-kube-api-access-v8q9b\") pod \"authorino-75d556bd77-q4ldl\" (UID: \"2734ec20-91f9-40f7-b209-7b39aaebeeec\") " pod="kuadrant-system/authorino-75d556bd77-q4ldl" Jul 22 01:45:28.498413 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:28.498304 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8q9b\" (UniqueName: \"kubernetes.io/projected/2734ec20-91f9-40f7-b209-7b39aaebeeec-kube-api-access-v8q9b\") pod \"authorino-75d556bd77-q4ldl\" (UID: \"2734ec20-91f9-40f7-b209-7b39aaebeeec\") " pod="kuadrant-system/authorino-75d556bd77-q4ldl" Jul 22 01:45:28.606729 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:28.606636 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-q4ldl" Jul 22 01:45:28.779724 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:28.779679 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q4ldl"] Jul 22 01:45:28.781452 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:45:28.781409 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2734ec20_91f9_40f7_b209_7b39aaebeeec.slice/crio-782b7f7edba949a7804d12883c2fc3a3020c463cf3681efdcc9f14ab9e722a91 WatchSource:0}: Error finding container 782b7f7edba949a7804d12883c2fc3a3020c463cf3681efdcc9f14ab9e722a91: Status 404 returned error can't find the container with id 782b7f7edba949a7804d12883c2fc3a3020c463cf3681efdcc9f14ab9e722a91 Jul 22 01:45:29.413291 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:29.413254 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-q4ldl" event={"ID":"2734ec20-91f9-40f7-b209-7b39aaebeeec","Type":"ContainerStarted","Data":"782b7f7edba949a7804d12883c2fc3a3020c463cf3681efdcc9f14ab9e722a91"} Jul 22 01:45:31.427769 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:31.427728 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-kwh5x" event={"ID":"5a55497f-0c13-487c-8a73-92ce2e459ad8","Type":"ContainerStarted","Data":"770ab55a86be4886114ca2a573ec0df00a88f7d3541c4605ae30b6754b7adc39"} Jul 22 01:45:31.428144 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:31.427841 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-kwh5x" Jul 22 01:45:31.447878 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:31.447830 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-kwh5x" podStartSLOduration=1.713818541 podStartE2EDuration="4.447815248s" totalImagesPullingTime="2.733996707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:31.445326203 +0000 UTC m=+502.038130578" watchObservedRunningTime="2026-07-22 01:45:31.447815248 +0000 UTC m=+502.040619609" Jul 22 01:45:32.434904 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:32.434871 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-q4ldl" event={"ID":"2734ec20-91f9-40f7-b209-7b39aaebeeec","Type":"ContainerStarted","Data":"bf043b38f9171785bc8820ae60b0fffad7a9ebc18abf1371e12c3c5672aca947"} Jul 22 01:45:32.457155 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:32.457111 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-q4ldl" podStartSLOduration=0.992377897 podStartE2EDuration="4.457094815s" totalImagesPullingTime="3.464716918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:45:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:32.454229887 +0000 UTC m=+503.047034230" watchObservedRunningTime="2026-07-22 01:45:32.457094815 +0000 UTC m=+503.049899178" Jul 22 01:45:42.114164 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.114121 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f5fc4bdf6-xvr87" podUID="2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0" containerName="console" containerID="cri-o://c4628a6788e987b62bcd020cc69704b909e250a1200358021b65be73f92eba20" gracePeriod=15 Jul 22 01:45:42.363409 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.363381 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:45:42.407294 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.407226 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-console-config\") pod \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\" (UID: \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\") " Jul 22 01:45:42.407408 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.407318 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-console-serving-cert\") pod \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\" (UID: \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\") " Jul 22 01:45:42.407408 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.407346 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-console-oauth-config\") pod \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\" (UID: \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\") " Jul 22 01:45:42.407408 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.407381 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-trusted-ca-bundle\") pod \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\" (UID: \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\") " Jul 22 01:45:42.407544 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.407467 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-service-ca\") pod \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\" (UID: \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\") " Jul 22 01:45:42.407544 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.407496 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-oauth-serving-cert\") pod \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\" (UID: \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\") " Jul 22 01:45:42.407647 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.407551 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-console-config" (OuterVolumeSpecName: "console-config") pod "2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0" (UID: "2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:45:42.407647 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.407602 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwgng\" (UniqueName: \"kubernetes.io/projected/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-kube-api-access-pwgng\") pod \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\" (UID: \"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0\") " Jul 22 01:45:42.407910 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.407845 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-service-ca" (OuterVolumeSpecName: "service-ca") pod "2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0" (UID: "2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:45:42.407910 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.407879 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0" (UID: "2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:45:42.408082 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.407915 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0" (UID: "2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:45:42.408252 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.408162 2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-console-config\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:45:42.408371 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.408254 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-trusted-ca-bundle\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:45:42.408371 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.408268 2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-service-ca\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:45:42.408371 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.408282 2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-oauth-serving-cert\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:45:42.409661 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.409637 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0" (UID: "2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:45:42.409882 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.409758 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0" (UID: "2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:45:42.409958 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.409890 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-kube-api-access-pwgng" (OuterVolumeSpecName: "kube-api-access-pwgng") pod "2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0" (UID: "2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0"). InnerVolumeSpecName "kube-api-access-pwgng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:45:42.436607 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.436590 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-kwh5x" Jul 22 01:45:42.496062 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.496030 2561 generic.go:356] "Generic (PLEG): container finished" podID="2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0" containerID="c4628a6788e987b62bcd020cc69704b909e250a1200358021b65be73f92eba20" exitCode=0 Jul 22 01:45:42.496161 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.496077 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5fc4bdf6-xvr87" Jul 22 01:45:42.496161 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.496121 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5fc4bdf6-xvr87" event={"ID":"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0","Type":"ContainerDied","Data":"c4628a6788e987b62bcd020cc69704b909e250a1200358021b65be73f92eba20"} Jul 22 01:45:42.496239 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.496161 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5fc4bdf6-xvr87" event={"ID":"2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0","Type":"ContainerDied","Data":"968fc4e3dca8153b4e006a4d309fdeedfecd3c3d93b4ac9f647bbdb642c356bb"} Jul 22 01:45:42.496239 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.496182 2561 scope.go:117] "RemoveContainer" containerID="c4628a6788e987b62bcd020cc69704b909e250a1200358021b65be73f92eba20" Jul 22 01:45:42.505605 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.505587 2561 scope.go:117] "RemoveContainer" containerID="c4628a6788e987b62bcd020cc69704b909e250a1200358021b65be73f92eba20" Jul 22 01:45:42.505871 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:45:42.505853 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4628a6788e987b62bcd020cc69704b909e250a1200358021b65be73f92eba20\": container with ID starting with c4628a6788e987b62bcd020cc69704b909e250a1200358021b65be73f92eba20 not found: ID does not exist" containerID="c4628a6788e987b62bcd020cc69704b909e250a1200358021b65be73f92eba20" Jul 22 01:45:42.505922 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.505880 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4628a6788e987b62bcd020cc69704b909e250a1200358021b65be73f92eba20"} err="failed to get container status \"c4628a6788e987b62bcd020cc69704b909e250a1200358021b65be73f92eba20\": rpc error: code = NotFound desc = could not find container \"c4628a6788e987b62bcd020cc69704b909e250a1200358021b65be73f92eba20\": container with ID starting with c4628a6788e987b62bcd020cc69704b909e250a1200358021b65be73f92eba20 not found: ID does not exist" Jul 22 01:45:42.508856 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.508834 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwgng\" (UniqueName: \"kubernetes.io/projected/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-kube-api-access-pwgng\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:45:42.508926 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.508860 2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-console-serving-cert\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:45:42.508926 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.508874 2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0-console-oauth-config\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:45:42.518745 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.518723 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f5fc4bdf6-xvr87"] Jul 22 01:45:42.521074 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:42.521050 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f5fc4bdf6-xvr87"] Jul 22 01:45:43.924683 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:43.924650 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0" path="/var/lib/kubelet/pods/2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0/volumes" Jul 22 01:45:44.562389 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:44.562352 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kwh5x"] Jul 22 01:45:44.562595 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:44.562568 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-kwh5x" podUID="5a55497f-0c13-487c-8a73-92ce2e459ad8" containerName="limitador" containerID="cri-o://770ab55a86be4886114ca2a573ec0df00a88f7d3541c4605ae30b6754b7adc39" gracePeriod=30 Jul 22 01:45:45.118434 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:45.118410 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-kwh5x" Jul 22 01:45:45.229515 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:45.229464 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5a55497f-0c13-487c-8a73-92ce2e459ad8-config-file\") pod \"5a55497f-0c13-487c-8a73-92ce2e459ad8\" (UID: \"5a55497f-0c13-487c-8a73-92ce2e459ad8\") " Jul 22 01:45:45.229666 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:45.229546 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tf9zz\" (UniqueName: \"kubernetes.io/projected/5a55497f-0c13-487c-8a73-92ce2e459ad8-kube-api-access-tf9zz\") pod \"5a55497f-0c13-487c-8a73-92ce2e459ad8\" (UID: \"5a55497f-0c13-487c-8a73-92ce2e459ad8\") " Jul 22 01:45:45.229853 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:45.229831 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a55497f-0c13-487c-8a73-92ce2e459ad8-config-file" (OuterVolumeSpecName: "config-file") pod "5a55497f-0c13-487c-8a73-92ce2e459ad8" (UID: "5a55497f-0c13-487c-8a73-92ce2e459ad8"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:45:45.231751 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:45.231726 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a55497f-0c13-487c-8a73-92ce2e459ad8-kube-api-access-tf9zz" (OuterVolumeSpecName: "kube-api-access-tf9zz") pod "5a55497f-0c13-487c-8a73-92ce2e459ad8" (UID: "5a55497f-0c13-487c-8a73-92ce2e459ad8"). InnerVolumeSpecName "kube-api-access-tf9zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:45:45.330659 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:45.330617 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf9zz\" (UniqueName: \"kubernetes.io/projected/5a55497f-0c13-487c-8a73-92ce2e459ad8-kube-api-access-tf9zz\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:45:45.330659 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:45.330656 2561 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5a55497f-0c13-487c-8a73-92ce2e459ad8-config-file\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:45:45.515849 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:45.515760 2561 generic.go:356] "Generic (PLEG): container finished" podID="5a55497f-0c13-487c-8a73-92ce2e459ad8" containerID="770ab55a86be4886114ca2a573ec0df00a88f7d3541c4605ae30b6754b7adc39" exitCode=0 Jul 22 01:45:45.515849 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:45.515812 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-kwh5x" Jul 22 01:45:45.516059 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:45.515812 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-kwh5x" event={"ID":"5a55497f-0c13-487c-8a73-92ce2e459ad8","Type":"ContainerDied","Data":"770ab55a86be4886114ca2a573ec0df00a88f7d3541c4605ae30b6754b7adc39"} Jul 22 01:45:45.516059 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:45.515945 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-kwh5x" event={"ID":"5a55497f-0c13-487c-8a73-92ce2e459ad8","Type":"ContainerDied","Data":"0007ccdc7b0dac3bd7b1519f0d0f753223f3f96a34abcdae91471426ee85af00"} Jul 22 01:45:45.516059 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:45.515976 2561 scope.go:117] "RemoveContainer" containerID="770ab55a86be4886114ca2a573ec0df00a88f7d3541c4605ae30b6754b7adc39" Jul 22 01:45:45.525047 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:45.525032 2561 scope.go:117] "RemoveContainer" containerID="770ab55a86be4886114ca2a573ec0df00a88f7d3541c4605ae30b6754b7adc39" Jul 22 01:45:45.525286 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:45:45.525267 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"770ab55a86be4886114ca2a573ec0df00a88f7d3541c4605ae30b6754b7adc39\": container with ID starting with 770ab55a86be4886114ca2a573ec0df00a88f7d3541c4605ae30b6754b7adc39 not found: ID does not exist" containerID="770ab55a86be4886114ca2a573ec0df00a88f7d3541c4605ae30b6754b7adc39" Jul 22 01:45:45.525388 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:45.525295 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"770ab55a86be4886114ca2a573ec0df00a88f7d3541c4605ae30b6754b7adc39"} err="failed to get container status \"770ab55a86be4886114ca2a573ec0df00a88f7d3541c4605ae30b6754b7adc39\": rpc error: code = NotFound desc = could not find container \"770ab55a86be4886114ca2a573ec0df00a88f7d3541c4605ae30b6754b7adc39\": container with ID starting with 770ab55a86be4886114ca2a573ec0df00a88f7d3541c4605ae30b6754b7adc39 not found: ID does not exist" Jul 22 01:45:45.538164 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:45.538140 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kwh5x"] Jul 22 01:45:45.542239 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:45.542217 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kwh5x"] Jul 22 01:45:45.924485 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:45.924458 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a55497f-0c13-487c-8a73-92ce2e459ad8" path="/var/lib/kubelet/pods/5a55497f-0c13-487c-8a73-92ce2e459ad8/volumes" Jul 22 01:45:47.187926 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:47.187895 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-24rfr"] Jul 22 01:45:47.188317 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:47.188304 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0" containerName="console" Jul 22 01:45:47.188356 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:47.188319 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0" containerName="console" Jul 22 01:45:47.188356 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:47.188328 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a55497f-0c13-487c-8a73-92ce2e459ad8" containerName="limitador" Jul 22 01:45:47.188356 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:47.188338 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a55497f-0c13-487c-8a73-92ce2e459ad8" containerName="limitador" Jul 22 01:45:47.188447 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:47.188415 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e37b8a7-9a0b-4be8-8224-09e98ba6a7c0" containerName="console" Jul 22 01:45:47.188447 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:47.188426 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a55497f-0c13-487c-8a73-92ce2e459ad8" containerName="limitador" Jul 22 01:45:47.191208 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:47.191192 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-24rfr" Jul 22 01:45:47.194192 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:47.194173 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 22 01:45:47.195083 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:47.195057 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 22 01:45:47.199493 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:47.199476 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 01:45:47.199607 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:47.199528 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-ntsb6\"" Jul 22 01:45:47.209124 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:47.209104 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-24rfr"] Jul 22 01:45:47.245940 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:47.245914 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbbns\" (UniqueName: \"kubernetes.io/projected/4f1e2188-e71a-4611-a144-722d4f9a2454-kube-api-access-kbbns\") pod \"postgres-69fb4594cf-24rfr\" (UID: \"4f1e2188-e71a-4611-a144-722d4f9a2454\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-24rfr" Jul 22 01:45:47.246100 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:47.245980 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4f1e2188-e71a-4611-a144-722d4f9a2454-data\") pod \"postgres-69fb4594cf-24rfr\" (UID: \"4f1e2188-e71a-4611-a144-722d4f9a2454\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-24rfr" Jul 22 01:45:47.347253 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:47.347226 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbbns\" (UniqueName: \"kubernetes.io/projected/4f1e2188-e71a-4611-a144-722d4f9a2454-kube-api-access-kbbns\") pod \"postgres-69fb4594cf-24rfr\" (UID: \"4f1e2188-e71a-4611-a144-722d4f9a2454\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-24rfr" Jul 22 01:45:47.347459 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:47.347288 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4f1e2188-e71a-4611-a144-722d4f9a2454-data\") pod \"postgres-69fb4594cf-24rfr\" (UID: \"4f1e2188-e71a-4611-a144-722d4f9a2454\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-24rfr" Jul 22 01:45:47.347638 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:47.347620 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4f1e2188-e71a-4611-a144-722d4f9a2454-data\") pod \"postgres-69fb4594cf-24rfr\" (UID: \"4f1e2188-e71a-4611-a144-722d4f9a2454\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-24rfr" Jul 22 01:45:47.369821 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:47.369797 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbbns\" (UniqueName: \"kubernetes.io/projected/4f1e2188-e71a-4611-a144-722d4f9a2454-kube-api-access-kbbns\") pod \"postgres-69fb4594cf-24rfr\" (UID: \"4f1e2188-e71a-4611-a144-722d4f9a2454\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-24rfr" Jul 22 01:45:47.501023 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:47.500938 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-24rfr" Jul 22 01:45:47.630541 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:47.630503 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-24rfr"] Jul 22 01:45:47.632415 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:45:47.632381 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f1e2188_e71a_4611_a144_722d4f9a2454.slice/crio-af03bac2453a4cca2b5e8849018b2ff7828da24584b300f28225b27e55455c8e WatchSource:0}: Error finding container af03bac2453a4cca2b5e8849018b2ff7828da24584b300f28225b27e55455c8e: Status 404 returned error can't find the container with id af03bac2453a4cca2b5e8849018b2ff7828da24584b300f28225b27e55455c8e Jul 22 01:45:48.536457 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:48.536421 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-24rfr" event={"ID":"4f1e2188-e71a-4611-a144-722d4f9a2454","Type":"ContainerStarted","Data":"af03bac2453a4cca2b5e8849018b2ff7828da24584b300f28225b27e55455c8e"} Jul 22 01:45:54.573534 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:54.573499 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-24rfr" event={"ID":"4f1e2188-e71a-4611-a144-722d4f9a2454","Type":"ContainerStarted","Data":"1f4f66b1610e7b32ace11c2048c321d7407f89d10247c5ec6d5a25cf44b9510a"} Jul 22 01:45:54.573957 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:54.573656 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-24rfr" Jul 22 01:45:54.592889 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:45:54.592845 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-24rfr" podStartSLOduration=1.680978818 podStartE2EDuration="7.592832199s" totalImagesPullingTime="5.911853381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:45:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:54.590628464 +0000 UTC m=+525.183432827" watchObservedRunningTime="2026-07-22 01:45:54.592832199 +0000 UTC m=+525.185636562" Jul 22 01:46:00.609022 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:00.608946 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-24rfr" Jul 22 01:46:01.515498 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:01.515465 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-644445698b-nk9xx"] Jul 22 01:46:01.518987 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:01.518971 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-644445698b-nk9xx" Jul 22 01:46:01.521316 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:01.521299 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 22 01:46:01.534206 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:01.528859 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-644445698b-nk9xx"] Jul 22 01:46:01.570034 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:01.570004 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-449rj\" (UniqueName: \"kubernetes.io/projected/ed90b218-3a67-4a33-a572-19fa2561c8f5-kube-api-access-449rj\") pod \"authorino-644445698b-nk9xx\" (UID: \"ed90b218-3a67-4a33-a572-19fa2561c8f5\") " pod="kuadrant-system/authorino-644445698b-nk9xx" Jul 22 01:46:01.570145 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:01.570074 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ed90b218-3a67-4a33-a572-19fa2561c8f5-tls-cert\") pod \"authorino-644445698b-nk9xx\" (UID: \"ed90b218-3a67-4a33-a572-19fa2561c8f5\") " pod="kuadrant-system/authorino-644445698b-nk9xx" Jul 22 01:46:01.671450 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:01.671420 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ed90b218-3a67-4a33-a572-19fa2561c8f5-tls-cert\") pod \"authorino-644445698b-nk9xx\" (UID: \"ed90b218-3a67-4a33-a572-19fa2561c8f5\") " pod="kuadrant-system/authorino-644445698b-nk9xx" Jul 22 01:46:01.671809 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:01.671487 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-449rj\" (UniqueName: \"kubernetes.io/projected/ed90b218-3a67-4a33-a572-19fa2561c8f5-kube-api-access-449rj\") pod \"authorino-644445698b-nk9xx\" (UID: \"ed90b218-3a67-4a33-a572-19fa2561c8f5\") " pod="kuadrant-system/authorino-644445698b-nk9xx" Jul 22 01:46:01.674006 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:01.673979 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ed90b218-3a67-4a33-a572-19fa2561c8f5-tls-cert\") pod \"authorino-644445698b-nk9xx\" (UID: \"ed90b218-3a67-4a33-a572-19fa2561c8f5\") " pod="kuadrant-system/authorino-644445698b-nk9xx" Jul 22 01:46:01.680095 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:01.680071 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-449rj\" (UniqueName: \"kubernetes.io/projected/ed90b218-3a67-4a33-a572-19fa2561c8f5-kube-api-access-449rj\") pod \"authorino-644445698b-nk9xx\" (UID: \"ed90b218-3a67-4a33-a572-19fa2561c8f5\") " pod="kuadrant-system/authorino-644445698b-nk9xx" Jul 22 01:46:01.838155 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:01.838053 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-644445698b-nk9xx" Jul 22 01:46:01.974621 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:01.974592 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-644445698b-nk9xx"] Jul 22 01:46:01.976296 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:46:01.976272 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded90b218_3a67_4a33_a572_19fa2561c8f5.slice/crio-08ece096de1005afc447abfea69ccbea95966e685726fa10f8afea22aea46a07 WatchSource:0}: Error finding container 08ece096de1005afc447abfea69ccbea95966e685726fa10f8afea22aea46a07: Status 404 returned error can't find the container with id 08ece096de1005afc447abfea69ccbea95966e685726fa10f8afea22aea46a07 Jul 22 01:46:02.617456 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:02.617365 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-644445698b-nk9xx" event={"ID":"ed90b218-3a67-4a33-a572-19fa2561c8f5","Type":"ContainerStarted","Data":"609dd917bb84b0e36edf8142dae7b09fb45a0dd85b9aec6f7ce3457b93ad6ea7"} Jul 22 01:46:02.617456 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:02.617416 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-644445698b-nk9xx" event={"ID":"ed90b218-3a67-4a33-a572-19fa2561c8f5","Type":"ContainerStarted","Data":"08ece096de1005afc447abfea69ccbea95966e685726fa10f8afea22aea46a07"} Jul 22 01:46:02.634828 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:02.634777 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-644445698b-nk9xx" podStartSLOduration=1.247096987 podStartE2EDuration="1.634760988s" totalImagesPullingTime="387.664001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:46:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:02.633095594 +0000 UTC m=+533.225899956" watchObservedRunningTime="2026-07-22 01:46:02.634760988 +0000 UTC m=+533.227565350" Jul 22 01:46:02.662170 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:02.662133 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q4ldl"] Jul 22 01:46:02.662823 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:02.662359 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-q4ldl" podUID="2734ec20-91f9-40f7-b209-7b39aaebeeec" containerName="authorino" containerID="cri-o://bf043b38f9171785bc8820ae60b0fffad7a9ebc18abf1371e12c3c5672aca947" gracePeriod=30 Jul 22 01:46:02.918652 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:02.918626 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-q4ldl" Jul 22 01:46:02.982404 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:02.982381 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8q9b\" (UniqueName: \"kubernetes.io/projected/2734ec20-91f9-40f7-b209-7b39aaebeeec-kube-api-access-v8q9b\") pod \"2734ec20-91f9-40f7-b209-7b39aaebeeec\" (UID: \"2734ec20-91f9-40f7-b209-7b39aaebeeec\") " Jul 22 01:46:02.984521 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:02.984495 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2734ec20-91f9-40f7-b209-7b39aaebeeec-kube-api-access-v8q9b" (OuterVolumeSpecName: "kube-api-access-v8q9b") pod "2734ec20-91f9-40f7-b209-7b39aaebeeec" (UID: "2734ec20-91f9-40f7-b209-7b39aaebeeec"). InnerVolumeSpecName "kube-api-access-v8q9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:46:03.083369 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:03.083329 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8q9b\" (UniqueName: \"kubernetes.io/projected/2734ec20-91f9-40f7-b209-7b39aaebeeec-kube-api-access-v8q9b\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:46:03.624808 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:03.624771 2561 generic.go:356] "Generic (PLEG): container finished" podID="2734ec20-91f9-40f7-b209-7b39aaebeeec" containerID="bf043b38f9171785bc8820ae60b0fffad7a9ebc18abf1371e12c3c5672aca947" exitCode=0 Jul 22 01:46:03.624983 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:03.624815 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-q4ldl" Jul 22 01:46:03.624983 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:03.624851 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-q4ldl" event={"ID":"2734ec20-91f9-40f7-b209-7b39aaebeeec","Type":"ContainerDied","Data":"bf043b38f9171785bc8820ae60b0fffad7a9ebc18abf1371e12c3c5672aca947"} Jul 22 01:46:03.624983 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:03.624895 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-q4ldl" event={"ID":"2734ec20-91f9-40f7-b209-7b39aaebeeec","Type":"ContainerDied","Data":"782b7f7edba949a7804d12883c2fc3a3020c463cf3681efdcc9f14ab9e722a91"} Jul 22 01:46:03.624983 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:03.624914 2561 scope.go:117] "RemoveContainer" containerID="bf043b38f9171785bc8820ae60b0fffad7a9ebc18abf1371e12c3c5672aca947" Jul 22 01:46:03.636785 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:03.636767 2561 scope.go:117] "RemoveContainer" containerID="bf043b38f9171785bc8820ae60b0fffad7a9ebc18abf1371e12c3c5672aca947" Jul 22 01:46:03.637048 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:46:03.637031 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf043b38f9171785bc8820ae60b0fffad7a9ebc18abf1371e12c3c5672aca947\": container with ID starting with bf043b38f9171785bc8820ae60b0fffad7a9ebc18abf1371e12c3c5672aca947 not found: ID does not exist" containerID="bf043b38f9171785bc8820ae60b0fffad7a9ebc18abf1371e12c3c5672aca947" Jul 22 01:46:03.637093 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:03.637056 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf043b38f9171785bc8820ae60b0fffad7a9ebc18abf1371e12c3c5672aca947"} err="failed to get container status \"bf043b38f9171785bc8820ae60b0fffad7a9ebc18abf1371e12c3c5672aca947\": rpc error: code = NotFound desc = could not find container \"bf043b38f9171785bc8820ae60b0fffad7a9ebc18abf1371e12c3c5672aca947\": container with ID starting with bf043b38f9171785bc8820ae60b0fffad7a9ebc18abf1371e12c3c5672aca947 not found: ID does not exist" Jul 22 01:46:03.651387 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:03.651361 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q4ldl"] Jul 22 01:46:03.655184 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:03.655165 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q4ldl"] Jul 22 01:46:03.925339 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:03.925288 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2734ec20-91f9-40f7-b209-7b39aaebeeec" path="/var/lib/kubelet/pods/2734ec20-91f9-40f7-b209-7b39aaebeeec/volumes" Jul 22 01:46:07.549499 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:07.549456 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7948667f84-h2fvl"] Jul 22 01:46:07.550057 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:07.550035 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2734ec20-91f9-40f7-b209-7b39aaebeeec" containerName="authorino" Jul 22 01:46:07.550141 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:07.550059 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2734ec20-91f9-40f7-b209-7b39aaebeeec" containerName="authorino" Jul 22 01:46:07.550229 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:07.550215 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2734ec20-91f9-40f7-b209-7b39aaebeeec" containerName="authorino" Jul 22 01:46:07.555712 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:07.555678 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7948667f84-h2fvl" Jul 22 01:46:07.558506 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:07.558483 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mmdbr\"" Jul 22 01:46:07.558506 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:07.558503 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 01:46:07.558742 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:07.558483 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 22 01:46:07.564745 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:07.564722 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7948667f84-h2fvl"] Jul 22 01:46:07.621848 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:07.621819 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m46jh\" (UniqueName: \"kubernetes.io/projected/313780dd-1251-4220-b4c4-7a6d206df52a-kube-api-access-m46jh\") pod \"maas-controller-7948667f84-h2fvl\" (UID: \"313780dd-1251-4220-b4c4-7a6d206df52a\") " pod="opendatahub/maas-controller-7948667f84-h2fvl" Jul 22 01:46:07.622020 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:07.621887 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/313780dd-1251-4220-b4c4-7a6d206df52a-webhook-cert\") pod \"maas-controller-7948667f84-h2fvl\" (UID: \"313780dd-1251-4220-b4c4-7a6d206df52a\") " pod="opendatahub/maas-controller-7948667f84-h2fvl" Jul 22 01:46:07.723189 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:07.723159 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m46jh\" (UniqueName: \"kubernetes.io/projected/313780dd-1251-4220-b4c4-7a6d206df52a-kube-api-access-m46jh\") pod \"maas-controller-7948667f84-h2fvl\" (UID: \"313780dd-1251-4220-b4c4-7a6d206df52a\") " pod="opendatahub/maas-controller-7948667f84-h2fvl" Jul 22 01:46:07.723352 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:07.723218 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/313780dd-1251-4220-b4c4-7a6d206df52a-webhook-cert\") pod \"maas-controller-7948667f84-h2fvl\" (UID: \"313780dd-1251-4220-b4c4-7a6d206df52a\") " pod="opendatahub/maas-controller-7948667f84-h2fvl" Jul 22 01:46:07.726048 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:07.726016 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/313780dd-1251-4220-b4c4-7a6d206df52a-webhook-cert\") pod \"maas-controller-7948667f84-h2fvl\" (UID: \"313780dd-1251-4220-b4c4-7a6d206df52a\") " pod="opendatahub/maas-controller-7948667f84-h2fvl" Jul 22 01:46:07.732153 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:07.732126 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m46jh\" (UniqueName: \"kubernetes.io/projected/313780dd-1251-4220-b4c4-7a6d206df52a-kube-api-access-m46jh\") pod \"maas-controller-7948667f84-h2fvl\" (UID: \"313780dd-1251-4220-b4c4-7a6d206df52a\") " pod="opendatahub/maas-controller-7948667f84-h2fvl" Jul 22 01:46:07.735597 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:07.735566 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-9cc45ddbb-6wlnq"] Jul 22 01:46:07.738646 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:07.738629 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9cc45ddbb-6wlnq" Jul 22 01:46:07.749813 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:07.749793 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9cc45ddbb-6wlnq"] Jul 22 01:46:07.823957 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:07.823878 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x47pb\" (UniqueName: \"kubernetes.io/projected/645e2cfc-febd-41df-89de-f4b084140538-kube-api-access-x47pb\") pod \"maas-controller-9cc45ddbb-6wlnq\" (UID: \"645e2cfc-febd-41df-89de-f4b084140538\") " pod="opendatahub/maas-controller-9cc45ddbb-6wlnq" Jul 22 01:46:07.823957 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:07.823948 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/645e2cfc-febd-41df-89de-f4b084140538-webhook-cert\") pod \"maas-controller-9cc45ddbb-6wlnq\" (UID: \"645e2cfc-febd-41df-89de-f4b084140538\") " pod="opendatahub/maas-controller-9cc45ddbb-6wlnq" Jul 22 01:46:07.867818 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:07.867787 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7948667f84-h2fvl" Jul 22 01:46:07.924564 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:07.924541 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x47pb\" (UniqueName: \"kubernetes.io/projected/645e2cfc-febd-41df-89de-f4b084140538-kube-api-access-x47pb\") pod \"maas-controller-9cc45ddbb-6wlnq\" (UID: \"645e2cfc-febd-41df-89de-f4b084140538\") " pod="opendatahub/maas-controller-9cc45ddbb-6wlnq" Jul 22 01:46:07.924714 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:07.924621 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/645e2cfc-febd-41df-89de-f4b084140538-webhook-cert\") pod \"maas-controller-9cc45ddbb-6wlnq\" (UID: \"645e2cfc-febd-41df-89de-f4b084140538\") " pod="opendatahub/maas-controller-9cc45ddbb-6wlnq" Jul 22 01:46:07.927714 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:07.927641 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/645e2cfc-febd-41df-89de-f4b084140538-webhook-cert\") pod \"maas-controller-9cc45ddbb-6wlnq\" (UID: \"645e2cfc-febd-41df-89de-f4b084140538\") " pod="opendatahub/maas-controller-9cc45ddbb-6wlnq" Jul 22 01:46:07.934252 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:07.934180 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x47pb\" (UniqueName: \"kubernetes.io/projected/645e2cfc-febd-41df-89de-f4b084140538-kube-api-access-x47pb\") pod \"maas-controller-9cc45ddbb-6wlnq\" (UID: \"645e2cfc-febd-41df-89de-f4b084140538\") " pod="opendatahub/maas-controller-9cc45ddbb-6wlnq" Jul 22 01:46:08.007042 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:08.007017 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7948667f84-h2fvl"] Jul 22 01:46:08.008501 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:46:08.008475 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313780dd_1251_4220_b4c4_7a6d206df52a.slice/crio-5d644cdf584f2dafb1fbe35e16699ca39f8ce8459796771fc7a0a606a7d5770c WatchSource:0}: Error finding container 5d644cdf584f2dafb1fbe35e16699ca39f8ce8459796771fc7a0a606a7d5770c: Status 404 returned error can't find the container with id 5d644cdf584f2dafb1fbe35e16699ca39f8ce8459796771fc7a0a606a7d5770c Jul 22 01:46:08.052476 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:08.052449 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9cc45ddbb-6wlnq" Jul 22 01:46:08.183966 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:08.183942 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9cc45ddbb-6wlnq"] Jul 22 01:46:08.185820 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:46:08.185795 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645e2cfc_febd_41df_89de_f4b084140538.slice/crio-3f4e041fb7d693c11a0847a7626b8d3c9be12c414d362034e78f5e6c9e105f5c WatchSource:0}: Error finding container 3f4e041fb7d693c11a0847a7626b8d3c9be12c414d362034e78f5e6c9e105f5c: Status 404 returned error can't find the container with id 3f4e041fb7d693c11a0847a7626b8d3c9be12c414d362034e78f5e6c9e105f5c Jul 22 01:46:08.653378 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:08.653334 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7948667f84-h2fvl" event={"ID":"313780dd-1251-4220-b4c4-7a6d206df52a","Type":"ContainerStarted","Data":"5d644cdf584f2dafb1fbe35e16699ca39f8ce8459796771fc7a0a606a7d5770c"} Jul 22 01:46:08.654509 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:08.654486 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cc45ddbb-6wlnq" event={"ID":"645e2cfc-febd-41df-89de-f4b084140538","Type":"ContainerStarted","Data":"3f4e041fb7d693c11a0847a7626b8d3c9be12c414d362034e78f5e6c9e105f5c"} Jul 22 01:46:10.648323 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:10.648301 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 01:46:11.674061 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:11.674023 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7948667f84-h2fvl" event={"ID":"313780dd-1251-4220-b4c4-7a6d206df52a","Type":"ContainerStarted","Data":"65e1789cedcede8819cc6d2adb10e21ae453976cd87feac4048cc46d9fc38da9"} Jul 22 01:46:11.674511 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:11.674133 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7948667f84-h2fvl" Jul 22 01:46:11.675826 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:11.675789 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cc45ddbb-6wlnq" event={"ID":"645e2cfc-febd-41df-89de-f4b084140538","Type":"ContainerStarted","Data":"fa9a7a783f7dd43c2de1542da9fec5f33b470b679226d521d3b1721447599e8f"} Jul 22 01:46:11.675947 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:11.675839 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9cc45ddbb-6wlnq" Jul 22 01:46:11.696064 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:11.696019 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7948667f84-h2fvl" podStartSLOduration=2.059948751 podStartE2EDuration="4.696007546s" totalImagesPullingTime="2.636058795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:46:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:11.69360127 +0000 UTC m=+542.286405633" watchObservedRunningTime="2026-07-22 01:46:11.696007546 +0000 UTC m=+542.288811909" Jul 22 01:46:11.710624 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:11.710582 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-9cc45ddbb-6wlnq" podStartSLOduration=2.245482064 podStartE2EDuration="4.71057079s" totalImagesPullingTime="2.465088726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:46:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:11.709144037 +0000 UTC m=+542.301948400" watchObservedRunningTime="2026-07-22 01:46:11.71057079 +0000 UTC m=+542.303375153" Jul 22 01:46:22.684173 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:22.684140 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9cc45ddbb-6wlnq" Jul 22 01:46:22.684610 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:22.684221 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7948667f84-h2fvl" Jul 22 01:46:22.751665 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:22.751633 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7948667f84-h2fvl"] Jul 22 01:46:22.751934 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:22.751889 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7948667f84-h2fvl" podUID="313780dd-1251-4220-b4c4-7a6d206df52a" containerName="manager" containerID="cri-o://65e1789cedcede8819cc6d2adb10e21ae453976cd87feac4048cc46d9fc38da9" gracePeriod=30 Jul 22 01:46:23.562154 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:23.562130 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7948667f84-h2fvl" Jul 22 01:46:23.665975 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:23.665909 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/313780dd-1251-4220-b4c4-7a6d206df52a-webhook-cert\") pod \"313780dd-1251-4220-b4c4-7a6d206df52a\" (UID: \"313780dd-1251-4220-b4c4-7a6d206df52a\") " Jul 22 01:46:23.665975 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:23.665944 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m46jh\" (UniqueName: \"kubernetes.io/projected/313780dd-1251-4220-b4c4-7a6d206df52a-kube-api-access-m46jh\") pod \"313780dd-1251-4220-b4c4-7a6d206df52a\" (UID: \"313780dd-1251-4220-b4c4-7a6d206df52a\") " Jul 22 01:46:23.668158 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:23.668133 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/313780dd-1251-4220-b4c4-7a6d206df52a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "313780dd-1251-4220-b4c4-7a6d206df52a" (UID: "313780dd-1251-4220-b4c4-7a6d206df52a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:46:23.668234 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:23.668183 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/313780dd-1251-4220-b4c4-7a6d206df52a-kube-api-access-m46jh" (OuterVolumeSpecName: "kube-api-access-m46jh") pod "313780dd-1251-4220-b4c4-7a6d206df52a" (UID: "313780dd-1251-4220-b4c4-7a6d206df52a"). InnerVolumeSpecName "kube-api-access-m46jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:46:23.749439 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:23.749411 2561 generic.go:356] "Generic (PLEG): container finished" podID="313780dd-1251-4220-b4c4-7a6d206df52a" containerID="65e1789cedcede8819cc6d2adb10e21ae453976cd87feac4048cc46d9fc38da9" exitCode=0 Jul 22 01:46:23.749867 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:23.749459 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7948667f84-h2fvl" Jul 22 01:46:23.749867 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:23.749512 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7948667f84-h2fvl" event={"ID":"313780dd-1251-4220-b4c4-7a6d206df52a","Type":"ContainerDied","Data":"65e1789cedcede8819cc6d2adb10e21ae453976cd87feac4048cc46d9fc38da9"} Jul 22 01:46:23.749867 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:23.749575 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7948667f84-h2fvl" event={"ID":"313780dd-1251-4220-b4c4-7a6d206df52a","Type":"ContainerDied","Data":"5d644cdf584f2dafb1fbe35e16699ca39f8ce8459796771fc7a0a606a7d5770c"} Jul 22 01:46:23.749867 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:23.749608 2561 scope.go:117] "RemoveContainer" containerID="65e1789cedcede8819cc6d2adb10e21ae453976cd87feac4048cc46d9fc38da9" Jul 22 01:46:23.758835 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:23.758641 2561 scope.go:117] "RemoveContainer" containerID="65e1789cedcede8819cc6d2adb10e21ae453976cd87feac4048cc46d9fc38da9" Jul 22 01:46:23.758932 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:46:23.758916 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65e1789cedcede8819cc6d2adb10e21ae453976cd87feac4048cc46d9fc38da9\": container with ID starting with 65e1789cedcede8819cc6d2adb10e21ae453976cd87feac4048cc46d9fc38da9 not found: ID does not exist" containerID="65e1789cedcede8819cc6d2adb10e21ae453976cd87feac4048cc46d9fc38da9" Jul 22 01:46:23.758984 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:23.758938 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65e1789cedcede8819cc6d2adb10e21ae453976cd87feac4048cc46d9fc38da9"} err="failed to get container status \"65e1789cedcede8819cc6d2adb10e21ae453976cd87feac4048cc46d9fc38da9\": rpc error: code = NotFound desc = could not find container \"65e1789cedcede8819cc6d2adb10e21ae453976cd87feac4048cc46d9fc38da9\": container with ID starting with 65e1789cedcede8819cc6d2adb10e21ae453976cd87feac4048cc46d9fc38da9 not found: ID does not exist" Jul 22 01:46:23.766600 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:23.766583 2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/313780dd-1251-4220-b4c4-7a6d206df52a-webhook-cert\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:46:23.766679 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:23.766601 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m46jh\" (UniqueName: \"kubernetes.io/projected/313780dd-1251-4220-b4c4-7a6d206df52a-kube-api-access-m46jh\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:46:23.772564 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:23.772543 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7948667f84-h2fvl"] Jul 22 01:46:23.776763 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:23.776745 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7948667f84-h2fvl"] Jul 22 01:46:23.925671 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:23.925645 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="313780dd-1251-4220-b4c4-7a6d206df52a" path="/var/lib/kubelet/pods/313780dd-1251-4220-b4c4-7a6d206df52a/volumes" Jul 22 01:46:43.143602 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.143572 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6f898ff5d6-5kx2g"] Jul 22 01:46:43.144065 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.144033 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="313780dd-1251-4220-b4c4-7a6d206df52a" containerName="manager" Jul 22 01:46:43.144065 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.144046 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="313780dd-1251-4220-b4c4-7a6d206df52a" containerName="manager" Jul 22 01:46:43.144136 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.144122 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="313780dd-1251-4220-b4c4-7a6d206df52a" containerName="manager" Jul 22 01:46:43.148543 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.148525 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f898ff5d6-5kx2g" Jul 22 01:46:43.151505 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.151483 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 22 01:46:43.151628 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.151533 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-49lpj\"" Jul 22 01:46:43.162497 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.162473 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f898ff5d6-5kx2g"] Jul 22 01:46:43.223183 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.223155 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcsxk\" (UniqueName: \"kubernetes.io/projected/b65d3c5b-99a3-4526-b567-79106c0c9217-kube-api-access-lcsxk\") pod \"maas-api-6f898ff5d6-5kx2g\" (UID: \"b65d3c5b-99a3-4526-b567-79106c0c9217\") " pod="odh-ai-gateway-infra/maas-api-6f898ff5d6-5kx2g" Jul 22 01:46:43.223333 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.223216 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b65d3c5b-99a3-4526-b567-79106c0c9217-maas-api-tls\") pod \"maas-api-6f898ff5d6-5kx2g\" (UID: \"b65d3c5b-99a3-4526-b567-79106c0c9217\") " pod="odh-ai-gateway-infra/maas-api-6f898ff5d6-5kx2g" Jul 22 01:46:43.223333 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.223274 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b65d3c5b-99a3-4526-b567-79106c0c9217-openshift-service-ca\") pod \"maas-api-6f898ff5d6-5kx2g\" (UID: \"b65d3c5b-99a3-4526-b567-79106c0c9217\") " pod="odh-ai-gateway-infra/maas-api-6f898ff5d6-5kx2g" Jul 22 01:46:43.324308 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.324283 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b65d3c5b-99a3-4526-b567-79106c0c9217-openshift-service-ca\") pod \"maas-api-6f898ff5d6-5kx2g\" (UID: \"b65d3c5b-99a3-4526-b567-79106c0c9217\") " pod="odh-ai-gateway-infra/maas-api-6f898ff5d6-5kx2g" Jul 22 01:46:43.324428 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.324332 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcsxk\" (UniqueName: \"kubernetes.io/projected/b65d3c5b-99a3-4526-b567-79106c0c9217-kube-api-access-lcsxk\") pod \"maas-api-6f898ff5d6-5kx2g\" (UID: \"b65d3c5b-99a3-4526-b567-79106c0c9217\") " pod="odh-ai-gateway-infra/maas-api-6f898ff5d6-5kx2g" Jul 22 01:46:43.324428 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.324371 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b65d3c5b-99a3-4526-b567-79106c0c9217-maas-api-tls\") pod \"maas-api-6f898ff5d6-5kx2g\" (UID: \"b65d3c5b-99a3-4526-b567-79106c0c9217\") " pod="odh-ai-gateway-infra/maas-api-6f898ff5d6-5kx2g" Jul 22 01:46:43.325030 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.325013 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b65d3c5b-99a3-4526-b567-79106c0c9217-openshift-service-ca\") pod \"maas-api-6f898ff5d6-5kx2g\" (UID: \"b65d3c5b-99a3-4526-b567-79106c0c9217\") " pod="odh-ai-gateway-infra/maas-api-6f898ff5d6-5kx2g" Jul 22 01:46:43.326878 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.326856 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b65d3c5b-99a3-4526-b567-79106c0c9217-maas-api-tls\") pod \"maas-api-6f898ff5d6-5kx2g\" (UID: \"b65d3c5b-99a3-4526-b567-79106c0c9217\") " pod="odh-ai-gateway-infra/maas-api-6f898ff5d6-5kx2g" Jul 22 01:46:43.335391 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.335370 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcsxk\" (UniqueName: \"kubernetes.io/projected/b65d3c5b-99a3-4526-b567-79106c0c9217-kube-api-access-lcsxk\") pod \"maas-api-6f898ff5d6-5kx2g\" (UID: \"b65d3c5b-99a3-4526-b567-79106c0c9217\") " pod="odh-ai-gateway-infra/maas-api-6f898ff5d6-5kx2g" Jul 22 01:46:43.459115 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.459086 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f898ff5d6-5kx2g" Jul 22 01:46:43.604967 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.604940 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f898ff5d6-5kx2g"] Jul 22 01:46:43.606759 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:46:43.606732 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65d3c5b_99a3_4526_b567_79106c0c9217.slice/crio-f4cbb82302bdf1e2e043b80ea9b1877146451236cb855ddf14ffa0207089589c WatchSource:0}: Error finding container f4cbb82302bdf1e2e043b80ea9b1877146451236cb855ddf14ffa0207089589c: Status 404 returned error can't find the container with id f4cbb82302bdf1e2e043b80ea9b1877146451236cb855ddf14ffa0207089589c Jul 22 01:46:43.834155 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.834077 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-hqg49"] Jul 22 01:46:43.837487 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.837466 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-hqg49" Jul 22 01:46:43.839974 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.839951 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 22 01:46:43.848801 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.848776 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-hqg49"] Jul 22 01:46:43.869124 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.869079 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f898ff5d6-5kx2g" event={"ID":"b65d3c5b-99a3-4526-b567-79106c0c9217","Type":"ContainerStarted","Data":"f4cbb82302bdf1e2e043b80ea9b1877146451236cb855ddf14ffa0207089589c"} Jul 22 01:46:43.871492 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.871470 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-tsdbn"] Jul 22 01:46:43.874484 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.874466 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-tsdbn" Jul 22 01:46:43.877095 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.877070 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-j9wgs\"" Jul 22 01:46:43.879317 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.879259 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-tsdbn"] Jul 22 01:46:43.928506 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.928482 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c084ceb9-3866-47e4-888c-1a79d2a91e5f-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-hqg49\" (UID: \"c084ceb9-3866-47e4-888c-1a79d2a91e5f\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-hqg49" Jul 22 01:46:43.928653 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.928512 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/011f7fd0-7031-49ab-8eb2-ea5dc4a3f6ad-plugins-config-volume\") pod \"payload-processing-5bf47cd988-tsdbn\" (UID: \"011f7fd0-7031-49ab-8eb2-ea5dc4a3f6ad\") " pod="openshift-ingress/payload-processing-5bf47cd988-tsdbn" Jul 22 01:46:43.928653 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.928539 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5t9w\" (UniqueName: \"kubernetes.io/projected/c084ceb9-3866-47e4-888c-1a79d2a91e5f-kube-api-access-q5t9w\") pod \"payload-pre-processing-7677b774bb-hqg49\" (UID: \"c084ceb9-3866-47e4-888c-1a79d2a91e5f\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-hqg49" Jul 22 01:46:43.928653 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:43.928562 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx7h8\" (UniqueName: \"kubernetes.io/projected/011f7fd0-7031-49ab-8eb2-ea5dc4a3f6ad-kube-api-access-jx7h8\") pod \"payload-processing-5bf47cd988-tsdbn\" (UID: \"011f7fd0-7031-49ab-8eb2-ea5dc4a3f6ad\") " pod="openshift-ingress/payload-processing-5bf47cd988-tsdbn" Jul 22 01:46:44.030054 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:44.030006 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c084ceb9-3866-47e4-888c-1a79d2a91e5f-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-hqg49\" (UID: \"c084ceb9-3866-47e4-888c-1a79d2a91e5f\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-hqg49" Jul 22 01:46:44.030054 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:44.030053 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/011f7fd0-7031-49ab-8eb2-ea5dc4a3f6ad-plugins-config-volume\") pod \"payload-processing-5bf47cd988-tsdbn\" (UID: \"011f7fd0-7031-49ab-8eb2-ea5dc4a3f6ad\") " pod="openshift-ingress/payload-processing-5bf47cd988-tsdbn" Jul 22 01:46:44.030295 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:44.030105 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5t9w\" (UniqueName: \"kubernetes.io/projected/c084ceb9-3866-47e4-888c-1a79d2a91e5f-kube-api-access-q5t9w\") pod \"payload-pre-processing-7677b774bb-hqg49\" (UID: \"c084ceb9-3866-47e4-888c-1a79d2a91e5f\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-hqg49" Jul 22 01:46:44.030295 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:44.030134 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx7h8\" (UniqueName: \"kubernetes.io/projected/011f7fd0-7031-49ab-8eb2-ea5dc4a3f6ad-kube-api-access-jx7h8\") pod \"payload-processing-5bf47cd988-tsdbn\" (UID: \"011f7fd0-7031-49ab-8eb2-ea5dc4a3f6ad\") " pod="openshift-ingress/payload-processing-5bf47cd988-tsdbn" Jul 22 01:46:44.030801 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:44.030779 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/011f7fd0-7031-49ab-8eb2-ea5dc4a3f6ad-plugins-config-volume\") pod \"payload-processing-5bf47cd988-tsdbn\" (UID: \"011f7fd0-7031-49ab-8eb2-ea5dc4a3f6ad\") " pod="openshift-ingress/payload-processing-5bf47cd988-tsdbn" Jul 22 01:46:44.030908 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:44.030809 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c084ceb9-3866-47e4-888c-1a79d2a91e5f-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-hqg49\" (UID: \"c084ceb9-3866-47e4-888c-1a79d2a91e5f\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-hqg49" Jul 22 01:46:44.040630 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:44.040601 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5t9w\" (UniqueName: \"kubernetes.io/projected/c084ceb9-3866-47e4-888c-1a79d2a91e5f-kube-api-access-q5t9w\") pod \"payload-pre-processing-7677b774bb-hqg49\" (UID: \"c084ceb9-3866-47e4-888c-1a79d2a91e5f\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-hqg49" Jul 22 01:46:44.042530 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:44.042504 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx7h8\" (UniqueName: \"kubernetes.io/projected/011f7fd0-7031-49ab-8eb2-ea5dc4a3f6ad-kube-api-access-jx7h8\") pod \"payload-processing-5bf47cd988-tsdbn\" (UID: \"011f7fd0-7031-49ab-8eb2-ea5dc4a3f6ad\") " pod="openshift-ingress/payload-processing-5bf47cd988-tsdbn" Jul 22 01:46:44.151493 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:44.151424 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-hqg49" Jul 22 01:46:44.185757 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:44.185727 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-tsdbn" Jul 22 01:46:44.360387 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:44.360348 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-tsdbn"] Jul 22 01:46:44.361259 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:46:44.361229 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod011f7fd0_7031_49ab_8eb2_ea5dc4a3f6ad.slice/crio-a2d766819e85ddbc1d847d1c1fa3c36a55b28348c206b64b0a93b9b7717b425e WatchSource:0}: Error finding container a2d766819e85ddbc1d847d1c1fa3c36a55b28348c206b64b0a93b9b7717b425e: Status 404 returned error can't find the container with id a2d766819e85ddbc1d847d1c1fa3c36a55b28348c206b64b0a93b9b7717b425e Jul 22 01:46:44.535129 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:44.535104 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-hqg49"] Jul 22 01:46:44.537389 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:46:44.537360 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc084ceb9_3866_47e4_888c_1a79d2a91e5f.slice/crio-20b222a195be313fc90b00ac8b693ae6afa06b7500516cd2601fa44cf6c57511 WatchSource:0}: Error finding container 20b222a195be313fc90b00ac8b693ae6afa06b7500516cd2601fa44cf6c57511: Status 404 returned error can't find the container with id 20b222a195be313fc90b00ac8b693ae6afa06b7500516cd2601fa44cf6c57511 Jul 22 01:46:44.877344 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:44.877256 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-hqg49" event={"ID":"c084ceb9-3866-47e4-888c-1a79d2a91e5f","Type":"ContainerStarted","Data":"20b222a195be313fc90b00ac8b693ae6afa06b7500516cd2601fa44cf6c57511"} Jul 22 01:46:44.878859 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:44.878799 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-tsdbn" event={"ID":"011f7fd0-7031-49ab-8eb2-ea5dc4a3f6ad","Type":"ContainerStarted","Data":"a2d766819e85ddbc1d847d1c1fa3c36a55b28348c206b64b0a93b9b7717b425e"} Jul 22 01:46:45.890435 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:45.890398 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f898ff5d6-5kx2g" event={"ID":"b65d3c5b-99a3-4526-b567-79106c0c9217","Type":"ContainerStarted","Data":"ff604547cf709835c5f6dca8ed7104f2f1bb945b09e98523333032b5b125814b"} Jul 22 01:46:45.890908 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:45.890530 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6f898ff5d6-5kx2g" Jul 22 01:46:45.912853 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:45.912809 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6f898ff5d6-5kx2g" podStartSLOduration=1.354985234 podStartE2EDuration="2.912794918s" totalImagesPullingTime="1.557809684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:45.910657786 +0000 UTC m=+576.503462173" watchObservedRunningTime="2026-07-22 01:46:45.912794918 +0000 UTC m=+576.505599296" Jul 22 01:46:46.897458 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:46.897367 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-hqg49" event={"ID":"c084ceb9-3866-47e4-888c-1a79d2a91e5f","Type":"ContainerStarted","Data":"22987509692bca74aeab336d7480c2a1508f9f4f073316d4804452d09803b15b"} Jul 22 01:46:46.897874 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:46.897525 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-hqg49" Jul 22 01:46:46.899118 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:46.899083 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-tsdbn" event={"ID":"011f7fd0-7031-49ab-8eb2-ea5dc4a3f6ad","Type":"ContainerStarted","Data":"4bb2486c1333a105b4c41dd54e74d4362086b6261d9734092b49ec5985906f98"} Jul 22 01:46:46.899262 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:46.899213 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-tsdbn" Jul 22 01:46:46.917564 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:46.917513 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-hqg49" podStartSLOduration=1.964168452 podStartE2EDuration="3.917499916s" totalImagesPullingTime="1.953331464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:46.916340634 +0000 UTC m=+577.509145010" watchObservedRunningTime="2026-07-22 01:46:46.917499916 +0000 UTC m=+577.510304278" Jul 22 01:46:46.938805 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:46.938757 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-tsdbn" podStartSLOduration=1.796998009 podStartE2EDuration="3.938739568s" totalImagesPullingTime="2.141741559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:46.93437951 +0000 UTC m=+577.527183865" watchObservedRunningTime="2026-07-22 01:46:46.938739568 +0000 UTC m=+577.531543934" Jul 22 01:46:51.904714 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:51.904666 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6f898ff5d6-5kx2g" Jul 22 01:46:57.906759 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:57.906727 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-tsdbn" Jul 22 01:46:57.907243 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:46:57.907024 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-hqg49" Jul 22 01:47:09.388461 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:09.388428 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t"] Jul 22 01:47:09.391132 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:09.391115 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t" Jul 22 01:47:09.393773 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:09.393750 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 22 01:47:09.394864 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:09.394847 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-v8lsh\"" Jul 22 01:47:09.395259 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:09.394870 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 22 01:47:09.395259 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:09.394847 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 22 01:47:09.402537 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:09.402515 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t"] Jul 22 01:47:09.547447 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:09.547414 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nl64t\" (UID: \"9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t" Jul 22 01:47:09.547607 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:09.547452 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nl64t\" (UID: \"9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t" Jul 22 01:47:09.547607 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:09.547474 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nl64t\" (UID: \"9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t" Jul 22 01:47:09.547607 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:09.547565 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f54wm\" (UniqueName: \"kubernetes.io/projected/9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9-kube-api-access-f54wm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nl64t\" (UID: \"9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t" Jul 22 01:47:09.547607 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:09.547604 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nl64t\" (UID: \"9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t" Jul 22 01:47:09.547787 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:09.547639 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nl64t\" (UID: \"9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t" Jul 22 01:47:09.648955 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:09.648871 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nl64t\" (UID: \"9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t" Jul 22 01:47:09.648955 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:09.648932 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nl64t\" (UID: \"9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t" Jul 22 01:47:09.649162 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:09.648961 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nl64t\" (UID: \"9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t" Jul 22 01:47:09.649162 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:09.649008 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f54wm\" (UniqueName: \"kubernetes.io/projected/9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9-kube-api-access-f54wm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nl64t\" (UID: \"9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t" Jul 22 01:47:09.649274 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:09.649151 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nl64t\" (UID: \"9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t" Jul 22 01:47:09.649274 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:09.649215 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nl64t\" (UID: \"9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t" Jul 22 01:47:09.649371 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:09.649327 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nl64t\" (UID: \"9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t" Jul 22 01:47:09.649371 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:09.649352 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nl64t\" (UID: \"9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t" Jul 22 01:47:09.649488 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:09.649395 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nl64t\" (UID: \"9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t" Jul 22 01:47:09.651439 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:09.651414 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nl64t\" (UID: \"9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t" Jul 22 01:47:09.651712 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:09.651677 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nl64t\" (UID: \"9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t" Jul 22 01:47:09.661840 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:09.661813 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f54wm\" (UniqueName: \"kubernetes.io/projected/9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9-kube-api-access-f54wm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nl64t\" (UID: \"9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t" Jul 22 01:47:09.701533 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:09.701504 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t" Jul 22 01:47:09.852268 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:09.852241 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t"] Jul 22 01:47:09.852989 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:47:09.852965 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9be25cbc_bb3b_43f0_b158_a3ab08b7d6a9.slice/crio-897acfd2c05c428d5f9d1fd5079c18210a85ba42450131d2ef58860f43e19d6c WatchSource:0}: Error finding container 897acfd2c05c428d5f9d1fd5079c18210a85ba42450131d2ef58860f43e19d6c: Status 404 returned error can't find the container with id 897acfd2c05c428d5f9d1fd5079c18210a85ba42450131d2ef58860f43e19d6c Jul 22 01:47:10.027739 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:10.027612 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t" event={"ID":"9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9","Type":"ContainerStarted","Data":"897acfd2c05c428d5f9d1fd5079c18210a85ba42450131d2ef58860f43e19d6c"} Jul 22 01:47:11.038076 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:11.038036 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t" event={"ID":"9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9","Type":"ContainerStarted","Data":"a47ec3205cf0b7d5a979f7d8864f088b956163549ac1d66cd92915ee59c04e9d"} Jul 22 01:47:11.063207 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:11.063150 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t" podStartSLOduration=0.97498957 podStartE2EDuration="2.063130467s" totalImagesPullingTime="1.088140897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:47:11.062832742 +0000 UTC m=+601.655637132" watchObservedRunningTime="2026-07-22 01:47:11.063130467 +0000 UTC m=+601.655934859" Jul 22 01:47:19.702272 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:19.702224 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t" Jul 22 01:47:19.702272 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:19.702279 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t" Jul 22 01:47:19.714979 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:19.714910 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t" Jul 22 01:47:20.097792 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:20.097725 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nl64t" Jul 22 01:47:29.910925 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:29.910891 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx"] Jul 22 01:47:29.914378 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:29.914358 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx" Jul 22 01:47:29.917054 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:29.917034 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 22 01:47:29.927130 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:29.927106 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx"] Jul 22 01:47:29.936337 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:29.936300 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zvrdx\" (UID: \"0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx" Jul 22 01:47:29.936427 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:29.936377 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zvrdx\" (UID: \"0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx" Jul 22 01:47:29.936491 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:29.936475 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zvrdx\" (UID: \"0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx" Jul 22 01:47:29.936532 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:29.936500 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zvrdx\" (UID: \"0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx" Jul 22 01:47:29.936583 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:29.936535 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zvrdx\" (UID: \"0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx" Jul 22 01:47:29.936583 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:29.936563 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgktb\" (UniqueName: \"kubernetes.io/projected/0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424-kube-api-access-bgktb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zvrdx\" (UID: \"0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx" Jul 22 01:47:30.037512 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:30.037474 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zvrdx\" (UID: \"0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx" Jul 22 01:47:30.037666 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:30.037523 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgktb\" (UniqueName: \"kubernetes.io/projected/0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424-kube-api-access-bgktb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zvrdx\" (UID: \"0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx" Jul 22 01:47:30.037666 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:30.037606 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zvrdx\" (UID: \"0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx" Jul 22 01:47:30.037801 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:30.037664 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zvrdx\" (UID: \"0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx" Jul 22 01:47:30.037801 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:30.037740 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zvrdx\" (UID: \"0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx" Jul 22 01:47:30.037801 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:30.037764 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zvrdx\" (UID: \"0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx" Jul 22 01:47:30.038166 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:30.038131 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zvrdx\" (UID: \"0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx" Jul 22 01:47:30.038166 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:30.038157 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zvrdx\" (UID: \"0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx" Jul 22 01:47:30.038328 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:30.038212 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zvrdx\" (UID: \"0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx" Jul 22 01:47:30.039853 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:30.039831 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zvrdx\" (UID: \"0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx" Jul 22 01:47:30.040345 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:30.040326 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zvrdx\" (UID: \"0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx" Jul 22 01:47:30.047907 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:30.047884 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgktb\" (UniqueName: \"kubernetes.io/projected/0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424-kube-api-access-bgktb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zvrdx\" (UID: \"0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx" Jul 22 01:47:30.225682 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:30.225652 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx" Jul 22 01:47:30.367570 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:30.367540 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx"] Jul 22 01:47:30.368093 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:47:30.368071 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cbbc5fd_55ba_4f44_ac9b_bf9ccc347424.slice/crio-1389c1e8eb6823ff6fdf0f14b3b1c4f15ff093d88110a38a8141e6bb21c2aa01 WatchSource:0}: Error finding container 1389c1e8eb6823ff6fdf0f14b3b1c4f15ff093d88110a38a8141e6bb21c2aa01: Status 404 returned error can't find the container with id 1389c1e8eb6823ff6fdf0f14b3b1c4f15ff093d88110a38a8141e6bb21c2aa01 Jul 22 01:47:30.369553 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:30.369537 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 01:47:31.149774 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:31.149738 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx" event={"ID":"0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424","Type":"ContainerStarted","Data":"864c8abb9f4dff9f0d06102fb072131a846ee64a1d37c648d4f235a0e91b83ee"} Jul 22 01:47:31.150137 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:31.149803 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx" event={"ID":"0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424","Type":"ContainerStarted","Data":"1389c1e8eb6823ff6fdf0f14b3b1c4f15ff093d88110a38a8141e6bb21c2aa01"} Jul 22 01:47:31.171493 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:31.171447 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx" podStartSLOduration=2.011359349 podStartE2EDuration="2.171431578s" totalImagesPullingTime="160.072229ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:47:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:47:31.170012587 +0000 UTC m=+621.762816949" watchObservedRunningTime="2026-07-22 01:47:31.171431578 +0000 UTC m=+621.764235942" Jul 22 01:47:40.226431 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:40.226398 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx" Jul 22 01:47:40.226935 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:40.226473 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx" Jul 22 01:47:40.244051 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:40.244026 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx" Jul 22 01:47:41.217299 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:47:41.217272 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zvrdx" Jul 22 01:49:30.484516 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:49:30.484476 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-644445698b-nk9xx"] Jul 22 01:49:30.485125 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:49:30.484753 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-644445698b-nk9xx" podUID="ed90b218-3a67-4a33-a572-19fa2561c8f5" containerName="authorino" containerID="cri-o://609dd917bb84b0e36edf8142dae7b09fb45a0dd85b9aec6f7ce3457b93ad6ea7" gracePeriod=30 Jul 22 01:49:30.751596 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:49:30.751573 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-644445698b-nk9xx" Jul 22 01:49:30.846793 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:49:30.846757 2561 generic.go:356] "Generic (PLEG): container finished" podID="ed90b218-3a67-4a33-a572-19fa2561c8f5" containerID="609dd917bb84b0e36edf8142dae7b09fb45a0dd85b9aec6f7ce3457b93ad6ea7" exitCode=0 Jul 22 01:49:30.846958 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:49:30.846822 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-644445698b-nk9xx" Jul 22 01:49:30.846958 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:49:30.846851 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-644445698b-nk9xx" event={"ID":"ed90b218-3a67-4a33-a572-19fa2561c8f5","Type":"ContainerDied","Data":"609dd917bb84b0e36edf8142dae7b09fb45a0dd85b9aec6f7ce3457b93ad6ea7"} Jul 22 01:49:30.846958 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:49:30.846900 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-644445698b-nk9xx" event={"ID":"ed90b218-3a67-4a33-a572-19fa2561c8f5","Type":"ContainerDied","Data":"08ece096de1005afc447abfea69ccbea95966e685726fa10f8afea22aea46a07"} Jul 22 01:49:30.846958 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:49:30.846923 2561 scope.go:117] "RemoveContainer" containerID="609dd917bb84b0e36edf8142dae7b09fb45a0dd85b9aec6f7ce3457b93ad6ea7" Jul 22 01:49:30.856289 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:49:30.856270 2561 scope.go:117] "RemoveContainer" containerID="609dd917bb84b0e36edf8142dae7b09fb45a0dd85b9aec6f7ce3457b93ad6ea7" Jul 22 01:49:30.856557 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:49:30.856538 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"609dd917bb84b0e36edf8142dae7b09fb45a0dd85b9aec6f7ce3457b93ad6ea7\": container with ID starting with 609dd917bb84b0e36edf8142dae7b09fb45a0dd85b9aec6f7ce3457b93ad6ea7 not found: ID does not exist" containerID="609dd917bb84b0e36edf8142dae7b09fb45a0dd85b9aec6f7ce3457b93ad6ea7" Jul 22 01:49:30.856617 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:49:30.856566 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"609dd917bb84b0e36edf8142dae7b09fb45a0dd85b9aec6f7ce3457b93ad6ea7"} err="failed to get container status \"609dd917bb84b0e36edf8142dae7b09fb45a0dd85b9aec6f7ce3457b93ad6ea7\": rpc error: code = NotFound desc = could not find container \"609dd917bb84b0e36edf8142dae7b09fb45a0dd85b9aec6f7ce3457b93ad6ea7\": container with ID starting with 609dd917bb84b0e36edf8142dae7b09fb45a0dd85b9aec6f7ce3457b93ad6ea7 not found: ID does not exist" Jul 22 01:49:30.864949 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:49:30.864924 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ed90b218-3a67-4a33-a572-19fa2561c8f5-tls-cert\") pod \"ed90b218-3a67-4a33-a572-19fa2561c8f5\" (UID: \"ed90b218-3a67-4a33-a572-19fa2561c8f5\") " Jul 22 01:49:30.865050 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:49:30.865037 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-449rj\" (UniqueName: \"kubernetes.io/projected/ed90b218-3a67-4a33-a572-19fa2561c8f5-kube-api-access-449rj\") pod \"ed90b218-3a67-4a33-a572-19fa2561c8f5\" (UID: \"ed90b218-3a67-4a33-a572-19fa2561c8f5\") " Jul 22 01:49:30.867439 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:49:30.867413 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed90b218-3a67-4a33-a572-19fa2561c8f5-kube-api-access-449rj" (OuterVolumeSpecName: "kube-api-access-449rj") pod "ed90b218-3a67-4a33-a572-19fa2561c8f5" (UID: "ed90b218-3a67-4a33-a572-19fa2561c8f5"). InnerVolumeSpecName "kube-api-access-449rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:49:30.878414 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:49:30.878383 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed90b218-3a67-4a33-a572-19fa2561c8f5-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ed90b218-3a67-4a33-a572-19fa2561c8f5" (UID: "ed90b218-3a67-4a33-a572-19fa2561c8f5"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:49:30.966372 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:49:30.966349 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-449rj\" (UniqueName: \"kubernetes.io/projected/ed90b218-3a67-4a33-a572-19fa2561c8f5-kube-api-access-449rj\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:49:30.966372 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:49:30.966371 2561 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ed90b218-3a67-4a33-a572-19fa2561c8f5-tls-cert\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:49:31.169363 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:49:31.169330 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-644445698b-nk9xx"] Jul 22 01:49:31.175365 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:49:31.175344 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-644445698b-nk9xx"] Jul 22 01:49:31.924833 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:49:31.924801 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed90b218-3a67-4a33-a572-19fa2561c8f5" path="/var/lib/kubelet/pods/ed90b218-3a67-4a33-a572-19fa2561c8f5/volumes" Jul 22 01:50:16.439502 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:50:16.439460 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9cc45ddbb-6wlnq"] Jul 22 01:50:16.439981 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:50:16.439725 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-9cc45ddbb-6wlnq" podUID="645e2cfc-febd-41df-89de-f4b084140538" containerName="manager" containerID="cri-o://fa9a7a783f7dd43c2de1542da9fec5f33b470b679226d521d3b1721447599e8f" gracePeriod=30 Jul 22 01:50:16.693515 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:50:16.693457 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9cc45ddbb-6wlnq" Jul 22 01:50:16.752008 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:50:16.751979 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x47pb\" (UniqueName: \"kubernetes.io/projected/645e2cfc-febd-41df-89de-f4b084140538-kube-api-access-x47pb\") pod \"645e2cfc-febd-41df-89de-f4b084140538\" (UID: \"645e2cfc-febd-41df-89de-f4b084140538\") " Jul 22 01:50:16.752189 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:50:16.752032 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/645e2cfc-febd-41df-89de-f4b084140538-webhook-cert\") pod \"645e2cfc-febd-41df-89de-f4b084140538\" (UID: \"645e2cfc-febd-41df-89de-f4b084140538\") " Jul 22 01:50:16.754232 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:50:16.754206 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645e2cfc-febd-41df-89de-f4b084140538-kube-api-access-x47pb" (OuterVolumeSpecName: "kube-api-access-x47pb") pod "645e2cfc-febd-41df-89de-f4b084140538" (UID: "645e2cfc-febd-41df-89de-f4b084140538"). InnerVolumeSpecName "kube-api-access-x47pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:50:16.754340 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:50:16.754299 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/645e2cfc-febd-41df-89de-f4b084140538-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "645e2cfc-febd-41df-89de-f4b084140538" (UID: "645e2cfc-febd-41df-89de-f4b084140538"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:50:16.853364 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:50:16.853340 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x47pb\" (UniqueName: \"kubernetes.io/projected/645e2cfc-febd-41df-89de-f4b084140538-kube-api-access-x47pb\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:50:16.853472 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:50:16.853364 2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/645e2cfc-febd-41df-89de-f4b084140538-webhook-cert\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:50:17.105500 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:50:17.105412 2561 generic.go:356] "Generic (PLEG): container finished" podID="645e2cfc-febd-41df-89de-f4b084140538" containerID="fa9a7a783f7dd43c2de1542da9fec5f33b470b679226d521d3b1721447599e8f" exitCode=0 Jul 22 01:50:17.105500 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:50:17.105458 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9cc45ddbb-6wlnq" Jul 22 01:50:17.105718 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:50:17.105494 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cc45ddbb-6wlnq" event={"ID":"645e2cfc-febd-41df-89de-f4b084140538","Type":"ContainerDied","Data":"fa9a7a783f7dd43c2de1542da9fec5f33b470b679226d521d3b1721447599e8f"} Jul 22 01:50:17.105718 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:50:17.105547 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cc45ddbb-6wlnq" event={"ID":"645e2cfc-febd-41df-89de-f4b084140538","Type":"ContainerDied","Data":"3f4e041fb7d693c11a0847a7626b8d3c9be12c414d362034e78f5e6c9e105f5c"} Jul 22 01:50:17.105718 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:50:17.105569 2561 scope.go:117] "RemoveContainer" containerID="fa9a7a783f7dd43c2de1542da9fec5f33b470b679226d521d3b1721447599e8f" Jul 22 01:50:17.115734 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:50:17.115680 2561 scope.go:117] "RemoveContainer" containerID="fa9a7a783f7dd43c2de1542da9fec5f33b470b679226d521d3b1721447599e8f" Jul 22 01:50:17.115981 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:50:17.115963 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa9a7a783f7dd43c2de1542da9fec5f33b470b679226d521d3b1721447599e8f\": container with ID starting with fa9a7a783f7dd43c2de1542da9fec5f33b470b679226d521d3b1721447599e8f not found: ID does not exist" containerID="fa9a7a783f7dd43c2de1542da9fec5f33b470b679226d521d3b1721447599e8f" Jul 22 01:50:17.116027 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:50:17.115991 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa9a7a783f7dd43c2de1542da9fec5f33b470b679226d521d3b1721447599e8f"} err="failed to get container status \"fa9a7a783f7dd43c2de1542da9fec5f33b470b679226d521d3b1721447599e8f\": rpc error: code = NotFound desc = could not find container \"fa9a7a783f7dd43c2de1542da9fec5f33b470b679226d521d3b1721447599e8f\": container with ID starting with fa9a7a783f7dd43c2de1542da9fec5f33b470b679226d521d3b1721447599e8f not found: ID does not exist" Jul 22 01:50:17.128434 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:50:17.128413 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9cc45ddbb-6wlnq"] Jul 22 01:50:17.135219 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:50:17.135196 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-9cc45ddbb-6wlnq"] Jul 22 01:50:17.925500 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:50:17.925464 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="645e2cfc-febd-41df-89de-f4b084140538" path="/var/lib/kubelet/pods/645e2cfc-febd-41df-89de-f4b084140538/volumes" Jul 22 01:51:17.962773 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:17.962745 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-9cc45ddbb-6q8dd"] Jul 22 01:51:17.963240 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:17.963183 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="645e2cfc-febd-41df-89de-f4b084140538" containerName="manager" Jul 22 01:51:17.963240 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:17.963196 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="645e2cfc-febd-41df-89de-f4b084140538" containerName="manager" Jul 22 01:51:17.963240 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:17.963208 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed90b218-3a67-4a33-a572-19fa2561c8f5" containerName="authorino" Jul 22 01:51:17.963240 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:17.963213 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed90b218-3a67-4a33-a572-19fa2561c8f5" containerName="authorino" Jul 22 01:51:17.963373 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:17.963301 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed90b218-3a67-4a33-a572-19fa2561c8f5" containerName="authorino" Jul 22 01:51:17.963373 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:17.963311 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="645e2cfc-febd-41df-89de-f4b084140538" containerName="manager" Jul 22 01:51:17.967674 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:17.967656 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9cc45ddbb-6q8dd" Jul 22 01:51:17.970887 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:17.970868 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mmdbr\"" Jul 22 01:51:17.970887 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:17.970882 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 01:51:17.971010 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:17.970886 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 22 01:51:17.976758 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:17.976734 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9cc45ddbb-6q8dd"] Jul 22 01:51:18.073896 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:18.073866 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xbs2\" (UniqueName: \"kubernetes.io/projected/cad5271d-31a1-434a-abcf-8e6ebc175523-kube-api-access-8xbs2\") pod \"maas-controller-9cc45ddbb-6q8dd\" (UID: \"cad5271d-31a1-434a-abcf-8e6ebc175523\") " pod="opendatahub/maas-controller-9cc45ddbb-6q8dd" Jul 22 01:51:18.074021 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:18.073989 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cad5271d-31a1-434a-abcf-8e6ebc175523-webhook-cert\") pod \"maas-controller-9cc45ddbb-6q8dd\" (UID: \"cad5271d-31a1-434a-abcf-8e6ebc175523\") " pod="opendatahub/maas-controller-9cc45ddbb-6q8dd" Jul 22 01:51:18.174612 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:18.174584 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cad5271d-31a1-434a-abcf-8e6ebc175523-webhook-cert\") pod \"maas-controller-9cc45ddbb-6q8dd\" (UID: \"cad5271d-31a1-434a-abcf-8e6ebc175523\") " pod="opendatahub/maas-controller-9cc45ddbb-6q8dd" Jul 22 01:51:18.174750 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:18.174631 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xbs2\" (UniqueName: \"kubernetes.io/projected/cad5271d-31a1-434a-abcf-8e6ebc175523-kube-api-access-8xbs2\") pod \"maas-controller-9cc45ddbb-6q8dd\" (UID: \"cad5271d-31a1-434a-abcf-8e6ebc175523\") " pod="opendatahub/maas-controller-9cc45ddbb-6q8dd" Jul 22 01:51:18.177241 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:18.177219 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cad5271d-31a1-434a-abcf-8e6ebc175523-webhook-cert\") pod \"maas-controller-9cc45ddbb-6q8dd\" (UID: \"cad5271d-31a1-434a-abcf-8e6ebc175523\") " pod="opendatahub/maas-controller-9cc45ddbb-6q8dd" Jul 22 01:51:18.183349 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:18.183328 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xbs2\" (UniqueName: \"kubernetes.io/projected/cad5271d-31a1-434a-abcf-8e6ebc175523-kube-api-access-8xbs2\") pod \"maas-controller-9cc45ddbb-6q8dd\" (UID: \"cad5271d-31a1-434a-abcf-8e6ebc175523\") " pod="opendatahub/maas-controller-9cc45ddbb-6q8dd" Jul 22 01:51:18.279346 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:18.279246 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9cc45ddbb-6q8dd" Jul 22 01:51:18.415568 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:18.415541 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9cc45ddbb-6q8dd"] Jul 22 01:51:18.416772 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:51:18.416743 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad5271d_31a1_434a_abcf_8e6ebc175523.slice/crio-71b7131d0d66dcd25953907fc871b7dcc14d6378e851b5dc70bfb2b453761471 WatchSource:0}: Error finding container 71b7131d0d66dcd25953907fc871b7dcc14d6378e851b5dc70bfb2b453761471: Status 404 returned error can't find the container with id 71b7131d0d66dcd25953907fc871b7dcc14d6378e851b5dc70bfb2b453761471 Jul 22 01:51:18.466129 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:18.466095 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cc45ddbb-6q8dd" event={"ID":"cad5271d-31a1-434a-abcf-8e6ebc175523","Type":"ContainerStarted","Data":"71b7131d0d66dcd25953907fc871b7dcc14d6378e851b5dc70bfb2b453761471"} Jul 22 01:51:19.473084 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:19.473045 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cc45ddbb-6q8dd" event={"ID":"cad5271d-31a1-434a-abcf-8e6ebc175523","Type":"ContainerStarted","Data":"e422e1bba1a32bcff3548da5805e0b8331145969d50e4bc5a1a243a02a5e5c1f"} Jul 22 01:51:19.473550 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:19.473116 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9cc45ddbb-6q8dd" Jul 22 01:51:19.489931 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:19.489887 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-9cc45ddbb-6q8dd" podStartSLOduration=2.044336019 podStartE2EDuration="2.489872614s" totalImagesPullingTime="445.536595ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:51:19.489061163 +0000 UTC m=+850.081865530" watchObservedRunningTime="2026-07-22 01:51:19.489872614 +0000 UTC m=+850.082676977" Jul 22 01:51:30.481244 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:30.481216 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9cc45ddbb-6q8dd" Jul 22 01:51:35.202884 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:35.202840 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9cc45ddbb-6q8dd"] Jul 22 01:51:35.203352 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:35.203151 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-9cc45ddbb-6q8dd" podUID="cad5271d-31a1-434a-abcf-8e6ebc175523" containerName="manager" containerID="cri-o://e422e1bba1a32bcff3548da5805e0b8331145969d50e4bc5a1a243a02a5e5c1f" gracePeriod=30 Jul 22 01:51:35.563517 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:35.563479 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9cc45ddbb-6q8dd" Jul 22 01:51:35.564845 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:35.564823 2561 generic.go:356] "Generic (PLEG): container finished" podID="cad5271d-31a1-434a-abcf-8e6ebc175523" containerID="e422e1bba1a32bcff3548da5805e0b8331145969d50e4bc5a1a243a02a5e5c1f" exitCode=0 Jul 22 01:51:35.564982 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:35.564905 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cc45ddbb-6q8dd" event={"ID":"cad5271d-31a1-434a-abcf-8e6ebc175523","Type":"ContainerDied","Data":"e422e1bba1a32bcff3548da5805e0b8331145969d50e4bc5a1a243a02a5e5c1f"} Jul 22 01:51:35.564982 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:35.564970 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cc45ddbb-6q8dd" event={"ID":"cad5271d-31a1-434a-abcf-8e6ebc175523","Type":"ContainerDied","Data":"71b7131d0d66dcd25953907fc871b7dcc14d6378e851b5dc70bfb2b453761471"} Jul 22 01:51:35.565069 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:35.565038 2561 scope.go:117] "RemoveContainer" containerID="e422e1bba1a32bcff3548da5805e0b8331145969d50e4bc5a1a243a02a5e5c1f" Jul 22 01:51:35.576511 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:35.576482 2561 scope.go:117] "RemoveContainer" containerID="e422e1bba1a32bcff3548da5805e0b8331145969d50e4bc5a1a243a02a5e5c1f" Jul 22 01:51:35.577048 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:51:35.577026 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e422e1bba1a32bcff3548da5805e0b8331145969d50e4bc5a1a243a02a5e5c1f\": container with ID starting with e422e1bba1a32bcff3548da5805e0b8331145969d50e4bc5a1a243a02a5e5c1f not found: ID does not exist" containerID="e422e1bba1a32bcff3548da5805e0b8331145969d50e4bc5a1a243a02a5e5c1f" Jul 22 01:51:35.577171 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:35.577064 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e422e1bba1a32bcff3548da5805e0b8331145969d50e4bc5a1a243a02a5e5c1f"} err="failed to get container status \"e422e1bba1a32bcff3548da5805e0b8331145969d50e4bc5a1a243a02a5e5c1f\": rpc error: code = NotFound desc = could not find container \"e422e1bba1a32bcff3548da5805e0b8331145969d50e4bc5a1a243a02a5e5c1f\": container with ID starting with e422e1bba1a32bcff3548da5805e0b8331145969d50e4bc5a1a243a02a5e5c1f not found: ID does not exist" Jul 22 01:51:35.627497 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:35.627468 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xbs2\" (UniqueName: \"kubernetes.io/projected/cad5271d-31a1-434a-abcf-8e6ebc175523-kube-api-access-8xbs2\") pod \"cad5271d-31a1-434a-abcf-8e6ebc175523\" (UID: \"cad5271d-31a1-434a-abcf-8e6ebc175523\") " Jul 22 01:51:35.627650 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:35.627541 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cad5271d-31a1-434a-abcf-8e6ebc175523-webhook-cert\") pod \"cad5271d-31a1-434a-abcf-8e6ebc175523\" (UID: \"cad5271d-31a1-434a-abcf-8e6ebc175523\") " Jul 22 01:51:35.629826 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:35.629778 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cad5271d-31a1-434a-abcf-8e6ebc175523-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "cad5271d-31a1-434a-abcf-8e6ebc175523" (UID: "cad5271d-31a1-434a-abcf-8e6ebc175523"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:51:35.629958 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:35.629778 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad5271d-31a1-434a-abcf-8e6ebc175523-kube-api-access-8xbs2" (OuterVolumeSpecName: "kube-api-access-8xbs2") pod "cad5271d-31a1-434a-abcf-8e6ebc175523" (UID: "cad5271d-31a1-434a-abcf-8e6ebc175523"). InnerVolumeSpecName "kube-api-access-8xbs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:51:35.728504 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:35.728442 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xbs2\" (UniqueName: \"kubernetes.io/projected/cad5271d-31a1-434a-abcf-8e6ebc175523-kube-api-access-8xbs2\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:51:35.728504 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:35.728468 2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cad5271d-31a1-434a-abcf-8e6ebc175523-webhook-cert\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:51:36.571783 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:36.571754 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9cc45ddbb-6q8dd" Jul 22 01:51:36.590432 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:36.590401 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9cc45ddbb-6q8dd"] Jul 22 01:51:36.594932 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:36.594907 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-9cc45ddbb-6q8dd"] Jul 22 01:51:37.446149 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:37.446118 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-9cc45ddbb-qtw4v"] Jul 22 01:51:37.446790 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:37.446756 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cad5271d-31a1-434a-abcf-8e6ebc175523" containerName="manager" Jul 22 01:51:37.446790 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:37.446778 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cad5271d-31a1-434a-abcf-8e6ebc175523" containerName="manager" Jul 22 01:51:37.446977 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:37.446913 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cad5271d-31a1-434a-abcf-8e6ebc175523" containerName="manager" Jul 22 01:51:37.450829 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:37.450812 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9cc45ddbb-qtw4v" Jul 22 01:51:37.454161 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:37.454136 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mmdbr\"" Jul 22 01:51:37.454257 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:37.454140 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 22 01:51:37.454257 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:37.454143 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 01:51:37.461304 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:37.461280 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9cc45ddbb-qtw4v"] Jul 22 01:51:37.545367 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:37.545331 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f068eb0-f72d-408d-af6c-ccc299da8009-webhook-cert\") pod \"maas-controller-9cc45ddbb-qtw4v\" (UID: \"2f068eb0-f72d-408d-af6c-ccc299da8009\") " pod="opendatahub/maas-controller-9cc45ddbb-qtw4v" Jul 22 01:51:37.545515 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:37.545392 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zz9t\" (UniqueName: \"kubernetes.io/projected/2f068eb0-f72d-408d-af6c-ccc299da8009-kube-api-access-7zz9t\") pod \"maas-controller-9cc45ddbb-qtw4v\" (UID: \"2f068eb0-f72d-408d-af6c-ccc299da8009\") " pod="opendatahub/maas-controller-9cc45ddbb-qtw4v" Jul 22 01:51:37.646218 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:37.646188 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zz9t\" (UniqueName: \"kubernetes.io/projected/2f068eb0-f72d-408d-af6c-ccc299da8009-kube-api-access-7zz9t\") pod \"maas-controller-9cc45ddbb-qtw4v\" (UID: \"2f068eb0-f72d-408d-af6c-ccc299da8009\") " pod="opendatahub/maas-controller-9cc45ddbb-qtw4v" Jul 22 01:51:37.646553 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:37.646272 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f068eb0-f72d-408d-af6c-ccc299da8009-webhook-cert\") pod \"maas-controller-9cc45ddbb-qtw4v\" (UID: \"2f068eb0-f72d-408d-af6c-ccc299da8009\") " pod="opendatahub/maas-controller-9cc45ddbb-qtw4v" Jul 22 01:51:37.648939 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:37.648919 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f068eb0-f72d-408d-af6c-ccc299da8009-webhook-cert\") pod \"maas-controller-9cc45ddbb-qtw4v\" (UID: \"2f068eb0-f72d-408d-af6c-ccc299da8009\") " pod="opendatahub/maas-controller-9cc45ddbb-qtw4v" Jul 22 01:51:37.655621 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:37.655598 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zz9t\" (UniqueName: \"kubernetes.io/projected/2f068eb0-f72d-408d-af6c-ccc299da8009-kube-api-access-7zz9t\") pod \"maas-controller-9cc45ddbb-qtw4v\" (UID: \"2f068eb0-f72d-408d-af6c-ccc299da8009\") " pod="opendatahub/maas-controller-9cc45ddbb-qtw4v" Jul 22 01:51:37.761587 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:37.761519 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9cc45ddbb-qtw4v" Jul 22 01:51:37.894852 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:37.894827 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9cc45ddbb-qtw4v"] Jul 22 01:51:37.895801 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:51:37.895774 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f068eb0_f72d_408d_af6c_ccc299da8009.slice/crio-f652fbd23ebe0973a3068f84b75956da8e5bb7d022db79625009fc71e804a82f WatchSource:0}: Error finding container f652fbd23ebe0973a3068f84b75956da8e5bb7d022db79625009fc71e804a82f: Status 404 returned error can't find the container with id f652fbd23ebe0973a3068f84b75956da8e5bb7d022db79625009fc71e804a82f Jul 22 01:51:37.925829 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:37.925801 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cad5271d-31a1-434a-abcf-8e6ebc175523" path="/var/lib/kubelet/pods/cad5271d-31a1-434a-abcf-8e6ebc175523/volumes" Jul 22 01:51:38.593239 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:38.593149 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cc45ddbb-qtw4v" event={"ID":"2f068eb0-f72d-408d-af6c-ccc299da8009","Type":"ContainerStarted","Data":"075e28aee0773b6adb77defe110d4c50faa66599bff9441df4800bdcdc8e4b52"} Jul 22 01:51:38.593389 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:38.593302 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9cc45ddbb-qtw4v" Jul 22 01:51:38.593389 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:38.593343 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9cc45ddbb-qtw4v" event={"ID":"2f068eb0-f72d-408d-af6c-ccc299da8009","Type":"ContainerStarted","Data":"f652fbd23ebe0973a3068f84b75956da8e5bb7d022db79625009fc71e804a82f"} Jul 22 01:51:38.609198 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:38.609160 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-9cc45ddbb-qtw4v" podStartSLOduration=1.244366792 podStartE2EDuration="1.609146052s" totalImagesPullingTime="364.77926ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:51:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:51:38.609038774 +0000 UTC m=+869.201843137" watchObservedRunningTime="2026-07-22 01:51:38.609146052 +0000 UTC m=+869.201950415" Jul 22 01:51:49.601355 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:51:49.601323 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9cc45ddbb-qtw4v" Jul 22 01:56:18.583585 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:56:18.583554 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55c7m"] Jul 22 01:56:18.584265 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:56:18.583806 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55c7m" podUID="378be3c0-a6b3-4335-836d-c7d1a461750d" containerName="manager" containerID="cri-o://59a076a0377c6d09f0bd92fb7b2cfc540a486e86d2fa9e81c993eee2db54258e" gracePeriod=10 Jul 22 01:56:21.837670 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:56:21.837647 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55c7m" Jul 22 01:56:21.936365 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:56:21.936332 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mq68g\" (UniqueName: \"kubernetes.io/projected/378be3c0-a6b3-4335-836d-c7d1a461750d-kube-api-access-mq68g\") pod \"378be3c0-a6b3-4335-836d-c7d1a461750d\" (UID: \"378be3c0-a6b3-4335-836d-c7d1a461750d\") " Jul 22 01:56:21.936532 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:56:21.936385 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/378be3c0-a6b3-4335-836d-c7d1a461750d-extensions-socket-volume\") pod \"378be3c0-a6b3-4335-836d-c7d1a461750d\" (UID: \"378be3c0-a6b3-4335-836d-c7d1a461750d\") " Jul 22 01:56:21.936797 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:56:21.936771 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378be3c0-a6b3-4335-836d-c7d1a461750d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "378be3c0-a6b3-4335-836d-c7d1a461750d" (UID: "378be3c0-a6b3-4335-836d-c7d1a461750d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 01:56:21.938528 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:56:21.938503 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/378be3c0-a6b3-4335-836d-c7d1a461750d-kube-api-access-mq68g" (OuterVolumeSpecName: "kube-api-access-mq68g") pod "378be3c0-a6b3-4335-836d-c7d1a461750d" (UID: "378be3c0-a6b3-4335-836d-c7d1a461750d"). InnerVolumeSpecName "kube-api-access-mq68g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:56:22.038125 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:56:22.038062 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq68g\" (UniqueName: \"kubernetes.io/projected/378be3c0-a6b3-4335-836d-c7d1a461750d-kube-api-access-mq68g\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:56:22.038125 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:56:22.038086 2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/378be3c0-a6b3-4335-836d-c7d1a461750d-extensions-socket-volume\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 01:56:22.216928 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:56:22.216893 2561 generic.go:356] "Generic (PLEG): container finished" podID="378be3c0-a6b3-4335-836d-c7d1a461750d" containerID="59a076a0377c6d09f0bd92fb7b2cfc540a486e86d2fa9e81c993eee2db54258e" exitCode=0 Jul 22 01:56:22.217116 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:56:22.216937 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55c7m" Jul 22 01:56:22.217116 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:56:22.216986 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55c7m" event={"ID":"378be3c0-a6b3-4335-836d-c7d1a461750d","Type":"ContainerDied","Data":"59a076a0377c6d09f0bd92fb7b2cfc540a486e86d2fa9e81c993eee2db54258e"} Jul 22 01:56:22.217116 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:56:22.217044 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55c7m" event={"ID":"378be3c0-a6b3-4335-836d-c7d1a461750d","Type":"ContainerDied","Data":"b728a720e21cb21d221ddc6e199e6ef375d0bd6044b89874649e2e6f1f59e131"} Jul 22 01:56:22.217116 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:56:22.217073 2561 scope.go:117] "RemoveContainer" containerID="59a076a0377c6d09f0bd92fb7b2cfc540a486e86d2fa9e81c993eee2db54258e" Jul 22 01:56:22.227875 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:56:22.227859 2561 scope.go:117] "RemoveContainer" containerID="59a076a0377c6d09f0bd92fb7b2cfc540a486e86d2fa9e81c993eee2db54258e" Jul 22 01:56:22.228108 ip-10-0-131-56 kubenswrapper[2561]: E0722 01:56:22.228086 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59a076a0377c6d09f0bd92fb7b2cfc540a486e86d2fa9e81c993eee2db54258e\": container with ID starting with 59a076a0377c6d09f0bd92fb7b2cfc540a486e86d2fa9e81c993eee2db54258e not found: ID does not exist" containerID="59a076a0377c6d09f0bd92fb7b2cfc540a486e86d2fa9e81c993eee2db54258e" Jul 22 01:56:22.228186 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:56:22.228115 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59a076a0377c6d09f0bd92fb7b2cfc540a486e86d2fa9e81c993eee2db54258e"} err="failed to get container status \"59a076a0377c6d09f0bd92fb7b2cfc540a486e86d2fa9e81c993eee2db54258e\": rpc error: code = NotFound desc = could not find container \"59a076a0377c6d09f0bd92fb7b2cfc540a486e86d2fa9e81c993eee2db54258e\": container with ID starting with 59a076a0377c6d09f0bd92fb7b2cfc540a486e86d2fa9e81c993eee2db54258e not found: ID does not exist" Jul 22 01:56:22.240368 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:56:22.240346 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55c7m"] Jul 22 01:56:22.244252 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:56:22.244220 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55c7m"] Jul 22 01:56:23.925668 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:56:23.925628 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="378be3c0-a6b3-4335-836d-c7d1a461750d" path="/var/lib/kubelet/pods/378be3c0-a6b3-4335-836d-c7d1a461750d/volumes" Jul 22 01:57:19.846995 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:57:19.846954 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-glwnx"] Jul 22 01:57:19.847597 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:57:19.847567 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="378be3c0-a6b3-4335-836d-c7d1a461750d" containerName="manager" Jul 22 01:57:19.847597 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:57:19.847595 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="378be3c0-a6b3-4335-836d-c7d1a461750d" containerName="manager" Jul 22 01:57:19.847788 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:57:19.847673 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="378be3c0-a6b3-4335-836d-c7d1a461750d" containerName="manager" Jul 22 01:57:19.851068 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:57:19.851053 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-glwnx" Jul 22 01:57:19.853373 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:57:19.853355 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zxvtc\"" Jul 22 01:57:19.863773 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:57:19.863752 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-glwnx"] Jul 22 01:57:19.941376 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:57:19.941353 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/54c07509-dea0-4ab0-a2fc-e90694a5ca50-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-glwnx\" (UID: \"54c07509-dea0-4ab0-a2fc-e90694a5ca50\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-glwnx" Jul 22 01:57:19.941493 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:57:19.941396 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9tbd\" (UniqueName: \"kubernetes.io/projected/54c07509-dea0-4ab0-a2fc-e90694a5ca50-kube-api-access-v9tbd\") pod \"kuadrant-operator-controller-manager-86878c769f-glwnx\" (UID: \"54c07509-dea0-4ab0-a2fc-e90694a5ca50\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-glwnx" Jul 22 01:57:20.042676 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:57:20.042652 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/54c07509-dea0-4ab0-a2fc-e90694a5ca50-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-glwnx\" (UID: \"54c07509-dea0-4ab0-a2fc-e90694a5ca50\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-glwnx" Jul 22 01:57:20.042794 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:57:20.042718 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9tbd\" (UniqueName: \"kubernetes.io/projected/54c07509-dea0-4ab0-a2fc-e90694a5ca50-kube-api-access-v9tbd\") pod \"kuadrant-operator-controller-manager-86878c769f-glwnx\" (UID: \"54c07509-dea0-4ab0-a2fc-e90694a5ca50\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-glwnx" Jul 22 01:57:20.043087 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:57:20.043065 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/54c07509-dea0-4ab0-a2fc-e90694a5ca50-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-glwnx\" (UID: \"54c07509-dea0-4ab0-a2fc-e90694a5ca50\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-glwnx" Jul 22 01:57:20.052119 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:57:20.052097 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9tbd\" (UniqueName: \"kubernetes.io/projected/54c07509-dea0-4ab0-a2fc-e90694a5ca50-kube-api-access-v9tbd\") pod \"kuadrant-operator-controller-manager-86878c769f-glwnx\" (UID: \"54c07509-dea0-4ab0-a2fc-e90694a5ca50\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-glwnx" Jul 22 01:57:20.160904 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:57:20.160847 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-glwnx" Jul 22 01:57:20.313927 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:57:20.313902 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-glwnx"] Jul 22 01:57:20.315116 ip-10-0-131-56 kubenswrapper[2561]: W0722 01:57:20.315085 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c07509_dea0_4ab0_a2fc_e90694a5ca50.slice/crio-766e98e92bdfea1d31bc306b1751889862ae7c801ed5300a2a271e399e131c5d WatchSource:0}: Error finding container 766e98e92bdfea1d31bc306b1751889862ae7c801ed5300a2a271e399e131c5d: Status 404 returned error can't find the container with id 766e98e92bdfea1d31bc306b1751889862ae7c801ed5300a2a271e399e131c5d Jul 22 01:57:20.543814 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:57:20.543771 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-glwnx" event={"ID":"54c07509-dea0-4ab0-a2fc-e90694a5ca50","Type":"ContainerStarted","Data":"738614fde6ffceb4806e3828e5870d89c469c728a6aaab30c06975eba9e6dd9b"} Jul 22 01:57:20.543979 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:57:20.543824 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-glwnx" event={"ID":"54c07509-dea0-4ab0-a2fc-e90694a5ca50","Type":"ContainerStarted","Data":"766e98e92bdfea1d31bc306b1751889862ae7c801ed5300a2a271e399e131c5d"} Jul 22 01:57:20.543979 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:57:20.543862 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-glwnx" Jul 22 01:57:20.565876 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:57:20.565824 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-glwnx" podStartSLOduration=1.5658077019999999 podStartE2EDuration="1.565807702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:57:20.564679276 +0000 UTC m=+1211.157483701" watchObservedRunningTime="2026-07-22 01:57:20.565807702 +0000 UTC m=+1211.158612134" Jul 22 01:57:31.552392 ip-10-0-131-56 kubenswrapper[2561]: I0722 01:57:31.552362 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-glwnx" Jul 22 02:04:26.838522 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:04:26.838488 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f2f0e842-66585f759b-7jkht"] Jul 22 02:04:26.842433 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:04:26.842409 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f2f0e842-66585f759b-7jkht" Jul 22 02:04:26.845053 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:04:26.845029 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f2f0e842-serving-cert\"" Jul 22 02:04:26.853542 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:04:26.853517 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f2f0e842-66585f759b-7jkht"] Jul 22 02:04:26.942043 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:04:26.942008 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd81515b-85a7-4d9c-8df6-5380c881bdda-openshift-service-ca\") pod \"maas-api-e2e-ait-f2f0e842-66585f759b-7jkht\" (UID: \"cd81515b-85a7-4d9c-8df6-5380c881bdda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f2f0e842-66585f759b-7jkht" Jul 22 02:04:26.942213 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:04:26.942123 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbthj\" (UniqueName: \"kubernetes.io/projected/cd81515b-85a7-4d9c-8df6-5380c881bdda-kube-api-access-pbthj\") pod \"maas-api-e2e-ait-f2f0e842-66585f759b-7jkht\" (UID: \"cd81515b-85a7-4d9c-8df6-5380c881bdda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f2f0e842-66585f759b-7jkht" Jul 22 02:04:26.942257 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:04:26.942236 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd81515b-85a7-4d9c-8df6-5380c881bdda-maas-api-tls\") pod \"maas-api-e2e-ait-f2f0e842-66585f759b-7jkht\" (UID: \"cd81515b-85a7-4d9c-8df6-5380c881bdda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f2f0e842-66585f759b-7jkht" Jul 22 02:04:27.043497 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:04:27.043463 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd81515b-85a7-4d9c-8df6-5380c881bdda-maas-api-tls\") pod \"maas-api-e2e-ait-f2f0e842-66585f759b-7jkht\" (UID: \"cd81515b-85a7-4d9c-8df6-5380c881bdda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f2f0e842-66585f759b-7jkht" Jul 22 02:04:27.043680 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:04:27.043541 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd81515b-85a7-4d9c-8df6-5380c881bdda-openshift-service-ca\") pod \"maas-api-e2e-ait-f2f0e842-66585f759b-7jkht\" (UID: \"cd81515b-85a7-4d9c-8df6-5380c881bdda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f2f0e842-66585f759b-7jkht" Jul 22 02:04:27.043680 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:04:27.043584 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbthj\" (UniqueName: \"kubernetes.io/projected/cd81515b-85a7-4d9c-8df6-5380c881bdda-kube-api-access-pbthj\") pod \"maas-api-e2e-ait-f2f0e842-66585f759b-7jkht\" (UID: \"cd81515b-85a7-4d9c-8df6-5380c881bdda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f2f0e842-66585f759b-7jkht" Jul 22 02:04:27.044226 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:04:27.044193 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd81515b-85a7-4d9c-8df6-5380c881bdda-openshift-service-ca\") pod \"maas-api-e2e-ait-f2f0e842-66585f759b-7jkht\" (UID: \"cd81515b-85a7-4d9c-8df6-5380c881bdda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f2f0e842-66585f759b-7jkht" Jul 22 02:04:27.046197 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:04:27.046174 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd81515b-85a7-4d9c-8df6-5380c881bdda-maas-api-tls\") pod \"maas-api-e2e-ait-f2f0e842-66585f759b-7jkht\" (UID: \"cd81515b-85a7-4d9c-8df6-5380c881bdda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f2f0e842-66585f759b-7jkht" Jul 22 02:04:27.053416 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:04:27.053390 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbthj\" (UniqueName: \"kubernetes.io/projected/cd81515b-85a7-4d9c-8df6-5380c881bdda-kube-api-access-pbthj\") pod \"maas-api-e2e-ait-f2f0e842-66585f759b-7jkht\" (UID: \"cd81515b-85a7-4d9c-8df6-5380c881bdda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f2f0e842-66585f759b-7jkht" Jul 22 02:04:27.154799 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:04:27.154712 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f2f0e842-66585f759b-7jkht" Jul 22 02:04:27.309513 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:04:27.307146 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f2f0e842-66585f759b-7jkht"] Jul 22 02:04:27.310156 ip-10-0-131-56 kubenswrapper[2561]: W0722 02:04:27.310095 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd81515b_85a7_4d9c_8df6_5380c881bdda.slice/crio-733fd78217cbc41d412860a1ae4308fc6226707687f3077af889128175c50e25 WatchSource:0}: Error finding container 733fd78217cbc41d412860a1ae4308fc6226707687f3077af889128175c50e25: Status 404 returned error can't find the container with id 733fd78217cbc41d412860a1ae4308fc6226707687f3077af889128175c50e25 Jul 22 02:04:27.311672 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:04:27.311650 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 02:04:27.973811 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:04:27.973570 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f2f0e842-66585f759b-7jkht" event={"ID":"cd81515b-85a7-4d9c-8df6-5380c881bdda","Type":"ContainerStarted","Data":"f0949e15a6ed1df2e37f3ec2d3d81c39e50a9b1707ec7a71e896dc4dfce42129"} Jul 22 02:04:27.973811 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:04:27.973726 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f2f0e842-66585f759b-7jkht" event={"ID":"cd81515b-85a7-4d9c-8df6-5380c881bdda","Type":"ContainerStarted","Data":"733fd78217cbc41d412860a1ae4308fc6226707687f3077af889128175c50e25"} Jul 22 02:04:27.974431 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:04:27.974362 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f2f0e842-66585f759b-7jkht" Jul 22 02:04:27.994257 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:04:27.994212 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f2f0e842-66585f759b-7jkht" podStartSLOduration=1.4914914989999999 podStartE2EDuration="1.99419919s" totalImagesPullingTime="502.707691ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:04:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:04:27.993461878 +0000 UTC m=+1638.586266238" watchObservedRunningTime="2026-07-22 02:04:27.99419919 +0000 UTC m=+1638.587003553" Jul 22 02:04:33.985500 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:04:33.985471 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f2f0e842-66585f759b-7jkht" Jul 22 02:05:02.948322 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:02.948288 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f2f0e842-66585f759b-7jkht"] Jul 22 02:05:02.948817 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:02.948632 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f2f0e842-66585f759b-7jkht" podUID="cd81515b-85a7-4d9c-8df6-5380c881bdda" containerName="maas-api" containerID="cri-o://f0949e15a6ed1df2e37f3ec2d3d81c39e50a9b1707ec7a71e896dc4dfce42129" gracePeriod=30 Jul 22 02:05:03.175782 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:03.175745 2561 generic.go:356] "Generic (PLEG): container finished" podID="cd81515b-85a7-4d9c-8df6-5380c881bdda" containerID="f0949e15a6ed1df2e37f3ec2d3d81c39e50a9b1707ec7a71e896dc4dfce42129" exitCode=0 Jul 22 02:05:03.175938 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:03.175817 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f2f0e842-66585f759b-7jkht" event={"ID":"cd81515b-85a7-4d9c-8df6-5380c881bdda","Type":"ContainerDied","Data":"f0949e15a6ed1df2e37f3ec2d3d81c39e50a9b1707ec7a71e896dc4dfce42129"} Jul 22 02:05:03.204129 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:03.204084 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f2f0e842-66585f759b-7jkht" Jul 22 02:05:03.360596 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:03.360565 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd81515b-85a7-4d9c-8df6-5380c881bdda-maas-api-tls\") pod \"cd81515b-85a7-4d9c-8df6-5380c881bdda\" (UID: \"cd81515b-85a7-4d9c-8df6-5380c881bdda\") " Jul 22 02:05:03.360803 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:03.360602 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbthj\" (UniqueName: \"kubernetes.io/projected/cd81515b-85a7-4d9c-8df6-5380c881bdda-kube-api-access-pbthj\") pod \"cd81515b-85a7-4d9c-8df6-5380c881bdda\" (UID: \"cd81515b-85a7-4d9c-8df6-5380c881bdda\") " Jul 22 02:05:03.360803 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:03.360646 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd81515b-85a7-4d9c-8df6-5380c881bdda-openshift-service-ca\") pod \"cd81515b-85a7-4d9c-8df6-5380c881bdda\" (UID: \"cd81515b-85a7-4d9c-8df6-5380c881bdda\") " Jul 22 02:05:03.361019 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:03.360995 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd81515b-85a7-4d9c-8df6-5380c881bdda-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cd81515b-85a7-4d9c-8df6-5380c881bdda" (UID: "cd81515b-85a7-4d9c-8df6-5380c881bdda"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:05:03.362972 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:03.362949 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd81515b-85a7-4d9c-8df6-5380c881bdda-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cd81515b-85a7-4d9c-8df6-5380c881bdda" (UID: "cd81515b-85a7-4d9c-8df6-5380c881bdda"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:05:03.363125 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:03.363105 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd81515b-85a7-4d9c-8df6-5380c881bdda-kube-api-access-pbthj" (OuterVolumeSpecName: "kube-api-access-pbthj") pod "cd81515b-85a7-4d9c-8df6-5380c881bdda" (UID: "cd81515b-85a7-4d9c-8df6-5380c881bdda"). InnerVolumeSpecName "kube-api-access-pbthj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:05:03.461472 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:03.461391 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd81515b-85a7-4d9c-8df6-5380c881bdda-maas-api-tls\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:05:03.461472 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:03.461416 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbthj\" (UniqueName: \"kubernetes.io/projected/cd81515b-85a7-4d9c-8df6-5380c881bdda-kube-api-access-pbthj\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:05:03.461472 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:03.461426 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd81515b-85a7-4d9c-8df6-5380c881bdda-openshift-service-ca\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:05:04.182971 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:04.182895 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f2f0e842-66585f759b-7jkht" Jul 22 02:05:04.182971 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:04.182942 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f2f0e842-66585f759b-7jkht" event={"ID":"cd81515b-85a7-4d9c-8df6-5380c881bdda","Type":"ContainerDied","Data":"733fd78217cbc41d412860a1ae4308fc6226707687f3077af889128175c50e25"} Jul 22 02:05:04.183407 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:04.183000 2561 scope.go:117] "RemoveContainer" containerID="f0949e15a6ed1df2e37f3ec2d3d81c39e50a9b1707ec7a71e896dc4dfce42129" Jul 22 02:05:04.201055 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:04.201029 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f2f0e842-66585f759b-7jkht"] Jul 22 02:05:04.206057 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:04.206037 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f2f0e842-66585f759b-7jkht"] Jul 22 02:05:05.924943 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:05.924901 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd81515b-85a7-4d9c-8df6-5380c881bdda" path="/var/lib/kubelet/pods/cd81515b-85a7-4d9c-8df6-5380c881bdda/volumes" Jul 22 02:05:11.154099 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:11.154068 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt"] Jul 22 02:05:11.154741 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:11.154719 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd81515b-85a7-4d9c-8df6-5380c881bdda" containerName="maas-api" Jul 22 02:05:11.154741 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:11.154740 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd81515b-85a7-4d9c-8df6-5380c881bdda" containerName="maas-api" Jul 22 02:05:11.154916 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:11.154898 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd81515b-85a7-4d9c-8df6-5380c881bdda" containerName="maas-api" Jul 22 02:05:11.159345 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:11.159322 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt" Jul 22 02:05:11.161595 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:11.161574 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-0dd62ba0-serving-cert\"" Jul 22 02:05:11.169900 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:11.169876 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt"] Jul 22 02:05:11.330754 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:11.330720 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e224278-6d9b-486c-9cab-cc1a68794f39-maas-api-tls\") pod \"maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt\" (UID: \"8e224278-6d9b-486c-9cab-cc1a68794f39\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt" Jul 22 02:05:11.330916 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:11.330767 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r825\" (UniqueName: \"kubernetes.io/projected/8e224278-6d9b-486c-9cab-cc1a68794f39-kube-api-access-7r825\") pod \"maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt\" (UID: \"8e224278-6d9b-486c-9cab-cc1a68794f39\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt" Jul 22 02:05:11.330916 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:11.330825 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e224278-6d9b-486c-9cab-cc1a68794f39-openshift-service-ca\") pod \"maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt\" (UID: \"8e224278-6d9b-486c-9cab-cc1a68794f39\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt" Jul 22 02:05:11.431685 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:11.431650 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r825\" (UniqueName: \"kubernetes.io/projected/8e224278-6d9b-486c-9cab-cc1a68794f39-kube-api-access-7r825\") pod \"maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt\" (UID: \"8e224278-6d9b-486c-9cab-cc1a68794f39\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt" Jul 22 02:05:11.431858 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:11.431738 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e224278-6d9b-486c-9cab-cc1a68794f39-openshift-service-ca\") pod \"maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt\" (UID: \"8e224278-6d9b-486c-9cab-cc1a68794f39\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt" Jul 22 02:05:11.431858 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:11.431824 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e224278-6d9b-486c-9cab-cc1a68794f39-maas-api-tls\") pod \"maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt\" (UID: \"8e224278-6d9b-486c-9cab-cc1a68794f39\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt" Jul 22 02:05:11.431938 ip-10-0-131-56 kubenswrapper[2561]: E0722 02:05:11.431925 2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-serving-cert: secret "maas-api-e2e-migrate-0dd62ba0-serving-cert" not found Jul 22 02:05:11.432009 ip-10-0-131-56 kubenswrapper[2561]: E0722 02:05:11.431998 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e224278-6d9b-486c-9cab-cc1a68794f39-maas-api-tls podName:8e224278-6d9b-486c-9cab-cc1a68794f39 nodeName:}" failed. No retries permitted until 2026-07-22 02:05:11.931977846 +0000 UTC m=+1682.524782186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8e224278-6d9b-486c-9cab-cc1a68794f39-maas-api-tls") pod "maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt" (UID: "8e224278-6d9b-486c-9cab-cc1a68794f39") : secret "maas-api-e2e-migrate-0dd62ba0-serving-cert" not found Jul 22 02:05:11.432376 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:11.432358 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e224278-6d9b-486c-9cab-cc1a68794f39-openshift-service-ca\") pod \"maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt\" (UID: \"8e224278-6d9b-486c-9cab-cc1a68794f39\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt" Jul 22 02:05:11.443424 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:11.443394 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r825\" (UniqueName: \"kubernetes.io/projected/8e224278-6d9b-486c-9cab-cc1a68794f39-kube-api-access-7r825\") pod \"maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt\" (UID: \"8e224278-6d9b-486c-9cab-cc1a68794f39\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt" Jul 22 02:05:11.936863 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:11.936832 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e224278-6d9b-486c-9cab-cc1a68794f39-maas-api-tls\") pod \"maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt\" (UID: \"8e224278-6d9b-486c-9cab-cc1a68794f39\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt" Jul 22 02:05:11.939382 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:11.939353 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e224278-6d9b-486c-9cab-cc1a68794f39-maas-api-tls\") pod \"maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt\" (UID: \"8e224278-6d9b-486c-9cab-cc1a68794f39\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt" Jul 22 02:05:12.072811 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:12.072778 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt" Jul 22 02:05:12.423363 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:12.423330 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt"] Jul 22 02:05:12.424606 ip-10-0-131-56 kubenswrapper[2561]: W0722 02:05:12.424579 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e224278_6d9b_486c_9cab_cc1a68794f39.slice/crio-35bbf45f868ee06024200fd02c691c384ee84f1c703ad08db8055b97ea8c249e WatchSource:0}: Error finding container 35bbf45f868ee06024200fd02c691c384ee84f1c703ad08db8055b97ea8c249e: Status 404 returned error can't find the container with id 35bbf45f868ee06024200fd02c691c384ee84f1c703ad08db8055b97ea8c249e Jul 22 02:05:13.240964 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:13.240926 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt" event={"ID":"8e224278-6d9b-486c-9cab-cc1a68794f39","Type":"ContainerStarted","Data":"897367176bb3bb2765d90e1677e89ba9624fe0c9c4604d15cedfc0478786ec43"} Jul 22 02:05:13.241128 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:13.240986 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt" Jul 22 02:05:13.241128 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:13.241004 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt" event={"ID":"8e224278-6d9b-486c-9cab-cc1a68794f39","Type":"ContainerStarted","Data":"35bbf45f868ee06024200fd02c691c384ee84f1c703ad08db8055b97ea8c249e"} Jul 22 02:05:13.262009 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:13.261960 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt" podStartSLOduration=1.894767965 podStartE2EDuration="2.261945285s" totalImagesPullingTime="367.17732ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:05:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:05:13.260632758 +0000 UTC m=+1683.853437120" watchObservedRunningTime="2026-07-22 02:05:13.261945285 +0000 UTC m=+1683.854749648" Jul 22 02:05:19.252471 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:19.252439 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt" Jul 22 02:05:46.664445 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:46.664413 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt"] Jul 22 02:05:46.664954 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:46.664607 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt" podUID="8e224278-6d9b-486c-9cab-cc1a68794f39" containerName="maas-api" containerID="cri-o://897367176bb3bb2765d90e1677e89ba9624fe0c9c4604d15cedfc0478786ec43" gracePeriod=30 Jul 22 02:05:46.925068 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:46.925046 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt" Jul 22 02:05:47.043423 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:47.043386 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7r825\" (UniqueName: \"kubernetes.io/projected/8e224278-6d9b-486c-9cab-cc1a68794f39-kube-api-access-7r825\") pod \"8e224278-6d9b-486c-9cab-cc1a68794f39\" (UID: \"8e224278-6d9b-486c-9cab-cc1a68794f39\") " Jul 22 02:05:47.043618 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:47.043450 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e224278-6d9b-486c-9cab-cc1a68794f39-openshift-service-ca\") pod \"8e224278-6d9b-486c-9cab-cc1a68794f39\" (UID: \"8e224278-6d9b-486c-9cab-cc1a68794f39\") " Jul 22 02:05:47.043618 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:47.043523 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e224278-6d9b-486c-9cab-cc1a68794f39-maas-api-tls\") pod \"8e224278-6d9b-486c-9cab-cc1a68794f39\" (UID: \"8e224278-6d9b-486c-9cab-cc1a68794f39\") " Jul 22 02:05:47.043866 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:47.043829 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e224278-6d9b-486c-9cab-cc1a68794f39-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8e224278-6d9b-486c-9cab-cc1a68794f39" (UID: "8e224278-6d9b-486c-9cab-cc1a68794f39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:05:47.044001 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:47.043955 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e224278-6d9b-486c-9cab-cc1a68794f39-openshift-service-ca\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:05:47.045842 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:47.045817 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e224278-6d9b-486c-9cab-cc1a68794f39-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8e224278-6d9b-486c-9cab-cc1a68794f39" (UID: "8e224278-6d9b-486c-9cab-cc1a68794f39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:05:47.045950 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:47.045841 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e224278-6d9b-486c-9cab-cc1a68794f39-kube-api-access-7r825" (OuterVolumeSpecName: "kube-api-access-7r825") pod "8e224278-6d9b-486c-9cab-cc1a68794f39" (UID: "8e224278-6d9b-486c-9cab-cc1a68794f39"). InnerVolumeSpecName "kube-api-access-7r825". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:05:47.144710 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:47.144652 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e224278-6d9b-486c-9cab-cc1a68794f39-maas-api-tls\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:05:47.144710 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:47.144686 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r825\" (UniqueName: \"kubernetes.io/projected/8e224278-6d9b-486c-9cab-cc1a68794f39-kube-api-access-7r825\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:05:47.436730 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:47.436663 2561 generic.go:356] "Generic (PLEG): container finished" podID="8e224278-6d9b-486c-9cab-cc1a68794f39" containerID="897367176bb3bb2765d90e1677e89ba9624fe0c9c4604d15cedfc0478786ec43" exitCode=0 Jul 22 02:05:47.436928 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:47.436738 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt" Jul 22 02:05:47.436928 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:47.436741 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt" event={"ID":"8e224278-6d9b-486c-9cab-cc1a68794f39","Type":"ContainerDied","Data":"897367176bb3bb2765d90e1677e89ba9624fe0c9c4604d15cedfc0478786ec43"} Jul 22 02:05:47.436928 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:47.436796 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt" event={"ID":"8e224278-6d9b-486c-9cab-cc1a68794f39","Type":"ContainerDied","Data":"35bbf45f868ee06024200fd02c691c384ee84f1c703ad08db8055b97ea8c249e"} Jul 22 02:05:47.436928 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:47.436819 2561 scope.go:117] "RemoveContainer" containerID="897367176bb3bb2765d90e1677e89ba9624fe0c9c4604d15cedfc0478786ec43" Jul 22 02:05:47.447072 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:47.447051 2561 scope.go:117] "RemoveContainer" containerID="897367176bb3bb2765d90e1677e89ba9624fe0c9c4604d15cedfc0478786ec43" Jul 22 02:05:47.447331 ip-10-0-131-56 kubenswrapper[2561]: E0722 02:05:47.447309 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"897367176bb3bb2765d90e1677e89ba9624fe0c9c4604d15cedfc0478786ec43\": container with ID starting with 897367176bb3bb2765d90e1677e89ba9624fe0c9c4604d15cedfc0478786ec43 not found: ID does not exist" containerID="897367176bb3bb2765d90e1677e89ba9624fe0c9c4604d15cedfc0478786ec43" Jul 22 02:05:47.447428 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:47.447338 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"897367176bb3bb2765d90e1677e89ba9624fe0c9c4604d15cedfc0478786ec43"} err="failed to get container status \"897367176bb3bb2765d90e1677e89ba9624fe0c9c4604d15cedfc0478786ec43\": rpc error: code = NotFound desc = could not find container \"897367176bb3bb2765d90e1677e89ba9624fe0c9c4604d15cedfc0478786ec43\": container with ID starting with 897367176bb3bb2765d90e1677e89ba9624fe0c9c4604d15cedfc0478786ec43 not found: ID does not exist" Jul 22 02:05:47.460027 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:47.460006 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt"] Jul 22 02:05:47.464000 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:47.463971 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0dd62ba0-c4488dc59-crrzt"] Jul 22 02:05:47.925409 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:47.925371 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e224278-6d9b-486c-9cab-cc1a68794f39" path="/var/lib/kubelet/pods/8e224278-6d9b-486c-9cab-cc1a68794f39/volumes" Jul 22 02:05:54.348968 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:54.348923 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l"] Jul 22 02:05:54.349635 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:54.349614 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e224278-6d9b-486c-9cab-cc1a68794f39" containerName="maas-api" Jul 22 02:05:54.349741 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:54.349639 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e224278-6d9b-486c-9cab-cc1a68794f39" containerName="maas-api" Jul 22 02:05:54.349806 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:54.349781 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e224278-6d9b-486c-9cab-cc1a68794f39" containerName="maas-api" Jul 22 02:05:54.355063 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:54.355033 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l" Jul 22 02:05:54.357502 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:54.357473 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8662596b-serving-cert\"" Jul 22 02:05:54.364582 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:54.364548 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l"] Jul 22 02:05:54.507641 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:54.507615 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/263bea81-29da-4699-88d7-152eb7a12b5e-maas-api-tls\") pod \"maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l\" (UID: \"263bea81-29da-4699-88d7-152eb7a12b5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l" Jul 22 02:05:54.507906 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:54.507735 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktttn\" (UniqueName: \"kubernetes.io/projected/263bea81-29da-4699-88d7-152eb7a12b5e-kube-api-access-ktttn\") pod \"maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l\" (UID: \"263bea81-29da-4699-88d7-152eb7a12b5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l" Jul 22 02:05:54.507906 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:54.507770 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/263bea81-29da-4699-88d7-152eb7a12b5e-openshift-service-ca\") pod \"maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l\" (UID: \"263bea81-29da-4699-88d7-152eb7a12b5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l" Jul 22 02:05:54.608977 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:54.608892 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktttn\" (UniqueName: \"kubernetes.io/projected/263bea81-29da-4699-88d7-152eb7a12b5e-kube-api-access-ktttn\") pod \"maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l\" (UID: \"263bea81-29da-4699-88d7-152eb7a12b5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l" Jul 22 02:05:54.608977 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:54.608933 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/263bea81-29da-4699-88d7-152eb7a12b5e-openshift-service-ca\") pod \"maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l\" (UID: \"263bea81-29da-4699-88d7-152eb7a12b5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l" Jul 22 02:05:54.609164 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:54.609049 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/263bea81-29da-4699-88d7-152eb7a12b5e-maas-api-tls\") pod \"maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l\" (UID: \"263bea81-29da-4699-88d7-152eb7a12b5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l" Jul 22 02:05:54.609482 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:54.609466 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/263bea81-29da-4699-88d7-152eb7a12b5e-openshift-service-ca\") pod \"maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l\" (UID: \"263bea81-29da-4699-88d7-152eb7a12b5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l" Jul 22 02:05:54.611578 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:54.611558 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/263bea81-29da-4699-88d7-152eb7a12b5e-maas-api-tls\") pod \"maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l\" (UID: \"263bea81-29da-4699-88d7-152eb7a12b5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l" Jul 22 02:05:54.617071 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:54.617047 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktttn\" (UniqueName: \"kubernetes.io/projected/263bea81-29da-4699-88d7-152eb7a12b5e-kube-api-access-ktttn\") pod \"maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l\" (UID: \"263bea81-29da-4699-88d7-152eb7a12b5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l" Jul 22 02:05:54.668070 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:54.668038 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l" Jul 22 02:05:54.821259 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:54.821230 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l"] Jul 22 02:05:54.822881 ip-10-0-131-56 kubenswrapper[2561]: W0722 02:05:54.822853 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263bea81_29da_4699_88d7_152eb7a12b5e.slice/crio-12dd0c3ffc688e51adb569a0c8613fabcc259b508c656c7d64edf2c6e1ecb0da WatchSource:0}: Error finding container 12dd0c3ffc688e51adb569a0c8613fabcc259b508c656c7d64edf2c6e1ecb0da: Status 404 returned error can't find the container with id 12dd0c3ffc688e51adb569a0c8613fabcc259b508c656c7d64edf2c6e1ecb0da Jul 22 02:05:55.486106 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:55.486062 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l" event={"ID":"263bea81-29da-4699-88d7-152eb7a12b5e","Type":"ContainerStarted","Data":"132fbfc9f4f275bec4592da8c106b2a123c0b62b3dc1a09f6394cac7cb86c919"} Jul 22 02:05:55.486510 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:55.486139 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l" event={"ID":"263bea81-29da-4699-88d7-152eb7a12b5e","Type":"ContainerStarted","Data":"12dd0c3ffc688e51adb569a0c8613fabcc259b508c656c7d64edf2c6e1ecb0da"} Jul 22 02:05:55.486510 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:55.486231 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l" Jul 22 02:05:55.501925 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:05:55.501878 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l" podStartSLOduration=1.028165832 podStartE2EDuration="1.501860188s" totalImagesPullingTime="473.694356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:05:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:05:55.501217814 +0000 UTC m=+1726.094022191" watchObservedRunningTime="2026-07-22 02:05:55.501860188 +0000 UTC m=+1726.094664551" Jul 22 02:06:01.497854 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:01.497823 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l" Jul 22 02:06:20.167637 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:20.167607 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8662596b-wh962"] Jul 22 02:06:20.171027 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:20.171009 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8662596b-wh962" Jul 22 02:06:20.173272 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:20.173255 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cqdw8\"" Jul 22 02:06:20.178776 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:20.178741 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8662596b-wh962"] Jul 22 02:06:20.239287 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:20.239258 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3f4af3-0003-4da3-b86e-fc729d680714-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8662596b-wh962\" (UID: \"1e3f4af3-0003-4da3-b86e-fc729d680714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8662596b-wh962" Jul 22 02:06:20.340819 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:20.340795 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3f4af3-0003-4da3-b86e-fc729d680714-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8662596b-wh962\" (UID: \"1e3f4af3-0003-4da3-b86e-fc729d680714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8662596b-wh962" Jul 22 02:06:20.341486 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:20.341468 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3f4af3-0003-4da3-b86e-fc729d680714-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8662596b-wh962\" (UID: \"1e3f4af3-0003-4da3-b86e-fc729d680714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8662596b-wh962" Jul 22 02:06:20.485961 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:20.485928 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8662596b-wh962" Jul 22 02:06:20.630542 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:20.630517 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8662596b-wh962"] Jul 22 02:06:20.632630 ip-10-0-131-56 kubenswrapper[2561]: W0722 02:06:20.632604 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3f4af3_0003_4da3_b86e_fc729d680714.slice/crio-aaaa1e15c9545726854f3a2359acfd6ce6a75acf017b9c9e3d05670c9e7dd9d6 WatchSource:0}: Error finding container aaaa1e15c9545726854f3a2359acfd6ce6a75acf017b9c9e3d05670c9e7dd9d6: Status 404 returned error can't find the container with id aaaa1e15c9545726854f3a2359acfd6ce6a75acf017b9c9e3d05670c9e7dd9d6 Jul 22 02:06:21.637232 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:21.637199 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8662596b-wh962" event={"ID":"1e3f4af3-0003-4da3-b86e-fc729d680714","Type":"ContainerStarted","Data":"aaaa1e15c9545726854f3a2359acfd6ce6a75acf017b9c9e3d05670c9e7dd9d6"} Jul 22 02:06:26.666981 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:26.666948 2561 generic.go:356] "Generic (PLEG): container finished" podID="1e3f4af3-0003-4da3-b86e-fc729d680714" containerID="b3c5226a0ee4a12c05de0a26230cf5f512ce6c4cc692f0257cce5767184ae56f" exitCode=0 Jul 22 02:06:26.667350 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:26.667033 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8662596b-wh962" event={"ID":"1e3f4af3-0003-4da3-b86e-fc729d680714","Type":"ContainerDied","Data":"b3c5226a0ee4a12c05de0a26230cf5f512ce6c4cc692f0257cce5767184ae56f"} Jul 22 02:06:27.810485 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:27.810465 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8662596b-wh962" Jul 22 02:06:27.911174 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:27.911146 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3f4af3-0003-4da3-b86e-fc729d680714-maas-api-service-ca\") pod \"1e3f4af3-0003-4da3-b86e-fc729d680714\" (UID: \"1e3f4af3-0003-4da3-b86e-fc729d680714\") " Jul 22 02:06:27.911453 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:27.911429 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e3f4af3-0003-4da3-b86e-fc729d680714-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1e3f4af3-0003-4da3-b86e-fc729d680714" (UID: "1e3f4af3-0003-4da3-b86e-fc729d680714"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:06:28.012022 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:28.011993 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3f4af3-0003-4da3-b86e-fc729d680714-maas-api-service-ca\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:06:28.679042 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:28.679012 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8662596b-wh962" Jul 22 02:06:28.679042 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:28.679030 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8662596b-wh962" event={"ID":"1e3f4af3-0003-4da3-b86e-fc729d680714","Type":"ContainerDied","Data":"aaaa1e15c9545726854f3a2359acfd6ce6a75acf017b9c9e3d05670c9e7dd9d6"} Jul 22 02:06:28.679298 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:28.679073 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaaa1e15c9545726854f3a2359acfd6ce6a75acf017b9c9e3d05670c9e7dd9d6" Jul 22 02:06:30.647404 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:30.647366 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l"] Jul 22 02:06:30.647903 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:30.647610 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l" podUID="263bea81-29da-4699-88d7-152eb7a12b5e" containerName="maas-api" containerID="cri-o://132fbfc9f4f275bec4592da8c106b2a123c0b62b3dc1a09f6394cac7cb86c919" gracePeriod=30 Jul 22 02:06:30.901337 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:30.901278 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l" Jul 22 02:06:31.051946 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:31.051913 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/263bea81-29da-4699-88d7-152eb7a12b5e-maas-api-tls\") pod \"263bea81-29da-4699-88d7-152eb7a12b5e\" (UID: \"263bea81-29da-4699-88d7-152eb7a12b5e\") " Jul 22 02:06:31.052098 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:31.052022 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/263bea81-29da-4699-88d7-152eb7a12b5e-openshift-service-ca\") pod \"263bea81-29da-4699-88d7-152eb7a12b5e\" (UID: \"263bea81-29da-4699-88d7-152eb7a12b5e\") " Jul 22 02:06:31.052098 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:31.052070 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktttn\" (UniqueName: \"kubernetes.io/projected/263bea81-29da-4699-88d7-152eb7a12b5e-kube-api-access-ktttn\") pod \"263bea81-29da-4699-88d7-152eb7a12b5e\" (UID: \"263bea81-29da-4699-88d7-152eb7a12b5e\") " Jul 22 02:06:31.052358 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:31.052332 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/263bea81-29da-4699-88d7-152eb7a12b5e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "263bea81-29da-4699-88d7-152eb7a12b5e" (UID: "263bea81-29da-4699-88d7-152eb7a12b5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:06:31.054215 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:31.054191 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/263bea81-29da-4699-88d7-152eb7a12b5e-kube-api-access-ktttn" (OuterVolumeSpecName: "kube-api-access-ktttn") pod "263bea81-29da-4699-88d7-152eb7a12b5e" (UID: "263bea81-29da-4699-88d7-152eb7a12b5e"). InnerVolumeSpecName "kube-api-access-ktttn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:06:31.054215 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:31.054200 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/263bea81-29da-4699-88d7-152eb7a12b5e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "263bea81-29da-4699-88d7-152eb7a12b5e" (UID: "263bea81-29da-4699-88d7-152eb7a12b5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:06:31.153505 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:31.153434 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktttn\" (UniqueName: \"kubernetes.io/projected/263bea81-29da-4699-88d7-152eb7a12b5e-kube-api-access-ktttn\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:06:31.153505 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:31.153462 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/263bea81-29da-4699-88d7-152eb7a12b5e-maas-api-tls\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:06:31.153505 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:31.153475 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/263bea81-29da-4699-88d7-152eb7a12b5e-openshift-service-ca\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:06:31.699192 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:31.699157 2561 generic.go:356] "Generic (PLEG): container finished" podID="263bea81-29da-4699-88d7-152eb7a12b5e" containerID="132fbfc9f4f275bec4592da8c106b2a123c0b62b3dc1a09f6394cac7cb86c919" exitCode=0 Jul 22 02:06:31.699607 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:31.699201 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l" Jul 22 02:06:31.699607 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:31.699252 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l" event={"ID":"263bea81-29da-4699-88d7-152eb7a12b5e","Type":"ContainerDied","Data":"132fbfc9f4f275bec4592da8c106b2a123c0b62b3dc1a09f6394cac7cb86c919"} Jul 22 02:06:31.699607 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:31.699308 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l" event={"ID":"263bea81-29da-4699-88d7-152eb7a12b5e","Type":"ContainerDied","Data":"12dd0c3ffc688e51adb569a0c8613fabcc259b508c656c7d64edf2c6e1ecb0da"} Jul 22 02:06:31.699607 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:31.699335 2561 scope.go:117] "RemoveContainer" containerID="132fbfc9f4f275bec4592da8c106b2a123c0b62b3dc1a09f6394cac7cb86c919" Jul 22 02:06:31.710794 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:31.710738 2561 scope.go:117] "RemoveContainer" containerID="132fbfc9f4f275bec4592da8c106b2a123c0b62b3dc1a09f6394cac7cb86c919" Jul 22 02:06:31.710984 ip-10-0-131-56 kubenswrapper[2561]: E0722 02:06:31.710966 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"132fbfc9f4f275bec4592da8c106b2a123c0b62b3dc1a09f6394cac7cb86c919\": container with ID starting with 132fbfc9f4f275bec4592da8c106b2a123c0b62b3dc1a09f6394cac7cb86c919 not found: ID does not exist" containerID="132fbfc9f4f275bec4592da8c106b2a123c0b62b3dc1a09f6394cac7cb86c919" Jul 22 02:06:31.711034 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:31.710996 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"132fbfc9f4f275bec4592da8c106b2a123c0b62b3dc1a09f6394cac7cb86c919"} err="failed to get container status \"132fbfc9f4f275bec4592da8c106b2a123c0b62b3dc1a09f6394cac7cb86c919\": rpc error: code = NotFound desc = could not find container \"132fbfc9f4f275bec4592da8c106b2a123c0b62b3dc1a09f6394cac7cb86c919\": container with ID starting with 132fbfc9f4f275bec4592da8c106b2a123c0b62b3dc1a09f6394cac7cb86c919 not found: ID does not exist" Jul 22 02:06:31.722647 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:31.722626 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l"] Jul 22 02:06:31.727625 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:31.727601 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8662596b-5cdf87fcdf-rws7l"] Jul 22 02:06:31.925916 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:31.925889 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="263bea81-29da-4699-88d7-152eb7a12b5e" path="/var/lib/kubelet/pods/263bea81-29da-4699-88d7-152eb7a12b5e/volumes" Jul 22 02:06:35.255524 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:35.255490 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8662596b-wh962"] Jul 22 02:06:35.257828 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:35.257808 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8662596b-wh962"] Jul 22 02:06:35.925009 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:35.924972 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e3f4af3-0003-4da3-b86e-fc729d680714" path="/var/lib/kubelet/pods/1e3f4af3-0003-4da3-b86e-fc729d680714/volumes" Jul 22 02:06:37.307510 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.307473 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh"] Jul 22 02:06:37.308079 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.308061 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="263bea81-29da-4699-88d7-152eb7a12b5e" containerName="maas-api" Jul 22 02:06:37.308157 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.308084 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="263bea81-29da-4699-88d7-152eb7a12b5e" containerName="maas-api" Jul 22 02:06:37.308157 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.308105 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e3f4af3-0003-4da3-b86e-fc729d680714" containerName="revoke-keys" Jul 22 02:06:37.308157 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.308111 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e3f4af3-0003-4da3-b86e-fc729d680714" containerName="revoke-keys" Jul 22 02:06:37.308292 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.308193 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="263bea81-29da-4699-88d7-152eb7a12b5e" containerName="maas-api" Jul 22 02:06:37.308292 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.308205 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e3f4af3-0003-4da3-b86e-fc729d680714" containerName="revoke-keys" Jul 22 02:06:37.312725 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.312682 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.315547 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.315527 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-c62fc15d-openshift-default-dockercfg-tfrlf\"" Jul 22 02:06:37.315547 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.315537 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 22 02:06:37.326439 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.326414 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh"] Jul 22 02:06:37.409119 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.409089 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istio-token\") pod \"e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.409266 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.409139 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-workload-socket\") pod \"e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.409266 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.409173 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-workload-certs\") pod \"e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.409266 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.409214 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istiod-ca-cert\") pod \"e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.409374 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.409263 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istio-podinfo\") pod \"e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.409374 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.409309 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istio-data\") pod \"e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.409446 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.409391 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-credential-socket\") pod \"e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.409446 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.409417 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk8rf\" (UniqueName: \"kubernetes.io/projected/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-kube-api-access-mk8rf\") pod \"e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.409446 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.409435 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istio-envoy\") pod \"e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.510392 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.510363 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk8rf\" (UniqueName: \"kubernetes.io/projected/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-kube-api-access-mk8rf\") pod \"e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.510520 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.510396 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istio-envoy\") pod \"e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.510520 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.510422 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istio-token\") pod \"e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.510520 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.510456 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-workload-socket\") pod \"e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.510520 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.510477 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-workload-certs\") pod \"e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.510520 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.510509 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istiod-ca-cert\") pod \"e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.510807 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.510538 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istio-podinfo\") pod \"e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.510807 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.510583 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istio-data\") pod \"e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.510807 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.510747 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-credential-socket\") pod \"e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.510970 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.510949 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-workload-socket\") pod \"e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.511045 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.511028 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-workload-certs\") pod \"e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.511105 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.511062 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istio-data\") pod \"e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.511213 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.511127 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-credential-socket\") pod \"e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.511341 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.511320 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istiod-ca-cert\") pod \"e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.512842 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.512823 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istio-envoy\") pod \"e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.513097 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.513078 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istio-podinfo\") pod \"e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.521262 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.521243 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk8rf\" (UniqueName: \"kubernetes.io/projected/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-kube-api-access-mk8rf\") pod \"e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.522901 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.522878 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istio-token\") pod \"e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.623086 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.623032 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:37.771856 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:37.771835 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh"] Jul 22 02:06:37.773158 ip-10-0-131-56 kubenswrapper[2561]: W0722 02:06:37.773129 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e1b3113_7e6d_4dd5_a0ff_911c0409b902.slice/crio-e9fdac7243d4d272776cf8f4f08d68e1d54e8b22f8285ba0a474e62b73617205 WatchSource:0}: Error finding container e9fdac7243d4d272776cf8f4f08d68e1d54e8b22f8285ba0a474e62b73617205: Status 404 returned error can't find the container with id e9fdac7243d4d272776cf8f4f08d68e1d54e8b22f8285ba0a474e62b73617205 Jul 22 02:06:38.743768 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:38.743731 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" event={"ID":"7e1b3113-7e6d-4dd5-a0ff-911c0409b902","Type":"ContainerStarted","Data":"e9fdac7243d4d272776cf8f4f08d68e1d54e8b22f8285ba0a474e62b73617205"} Jul 22 02:06:44.135464 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:44.135419 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 22 02:06:44.135774 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:44.135497 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 22 02:06:44.135774 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:44.135525 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 22 02:06:44.783971 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:44.783934 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" event={"ID":"7e1b3113-7e6d-4dd5-a0ff-911c0409b902","Type":"ContainerStarted","Data":"1ad07c065be9fcfb146803d28587ecfbae92c08b55bc70485cd59695a3337a7e"} Jul 22 02:06:44.808544 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:44.806818 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" podStartSLOduration=1.446690612 podStartE2EDuration="7.80679386s" totalImagesPullingTime="6.360103248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:06:44.805245363 +0000 UTC m=+1775.398049726" watchObservedRunningTime="2026-07-22 02:06:44.80679386 +0000 UTC m=+1775.399598238" Jul 22 02:06:45.624080 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:45.624047 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:45.629111 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:45.629085 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:45.789838 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:45.789805 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:45.790600 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:45.790581 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:06:58.875773 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:58.875665 2561 generic.go:356] "Generic (PLEG): container finished" podID="7e1b3113-7e6d-4dd5-a0ff-911c0409b902" containerID="1ad07c065be9fcfb146803d28587ecfbae92c08b55bc70485cd59695a3337a7e" exitCode=0 Jul 22 02:06:58.875773 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:58.875738 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" event={"ID":"7e1b3113-7e6d-4dd5-a0ff-911c0409b902","Type":"ContainerDied","Data":"1ad07c065be9fcfb146803d28587ecfbae92c08b55bc70485cd59695a3337a7e"} Jul 22 02:06:58.876211 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:58.876196 2561 scope.go:117] "RemoveContainer" containerID="1ad07c065be9fcfb146803d28587ecfbae92c08b55bc70485cd59695a3337a7e" Jul 22 02:06:58.877215 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:58.877183 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 22 02:06:58.877315 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:58.877270 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 22 02:06:58.877382 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:58.877319 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 22 02:06:59.883061 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:06:59.883022 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" event={"ID":"7e1b3113-7e6d-4dd5-a0ff-911c0409b902","Type":"ContainerStarted","Data":"fd39b2b3daaac6530a07b39a2636270596af445891cd24146817dcd472443ec0"} Jul 22 02:07:00.623808 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:00.623735 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:07:00.624922 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:00.624897 2561 patch_prober.go:28] interesting pod/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Jul 22 02:07:00.625027 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:00.624950 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" podUID="7e1b3113-7e6d-4dd5-a0ff-911c0409b902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Jul 22 02:07:00.790784 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:00.790750 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:07:01.624366 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:01.624334 2561 patch_prober.go:28] interesting pod/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Jul 22 02:07:01.624787 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:01.624393 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" podUID="7e1b3113-7e6d-4dd5-a0ff-911c0409b902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Jul 22 02:07:02.623605 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:02.623571 2561 patch_prober.go:28] interesting pod/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Jul 22 02:07:02.623805 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:02.623623 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" podUID="7e1b3113-7e6d-4dd5-a0ff-911c0409b902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Jul 22 02:07:03.624211 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:03.624176 2561 patch_prober.go:28] interesting pod/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Jul 22 02:07:03.624683 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:03.624241 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" podUID="7e1b3113-7e6d-4dd5-a0ff-911c0409b902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Jul 22 02:07:04.627358 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:04.627322 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:07:04.628373 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:04.628354 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:07:11.117139 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:11.117115 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c62fc15d-w8r8f"] Jul 22 02:07:11.119953 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:11.119935 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c62fc15d-w8r8f" Jul 22 02:07:11.122492 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:11.122462 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cqdw8\"" Jul 22 02:07:11.125546 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:11.125499 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c62fc15d-w8r8f"] Jul 22 02:07:11.229329 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:11.229297 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a28fc94e-e8a2-4d1c-9b96-97e9e4df5319-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c62fc15d-w8r8f\" (UID: \"a28fc94e-e8a2-4d1c-9b96-97e9e4df5319\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c62fc15d-w8r8f" Jul 22 02:07:11.330604 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:11.330573 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a28fc94e-e8a2-4d1c-9b96-97e9e4df5319-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c62fc15d-w8r8f\" (UID: \"a28fc94e-e8a2-4d1c-9b96-97e9e4df5319\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c62fc15d-w8r8f" Jul 22 02:07:11.331200 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:11.331182 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a28fc94e-e8a2-4d1c-9b96-97e9e4df5319-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c62fc15d-w8r8f\" (UID: \"a28fc94e-e8a2-4d1c-9b96-97e9e4df5319\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c62fc15d-w8r8f" Jul 22 02:07:11.433650 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:11.433625 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c62fc15d-w8r8f" Jul 22 02:07:11.779440 ip-10-0-131-56 kubenswrapper[2561]: W0722 02:07:11.779409 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda28fc94e_e8a2_4d1c_9b96_97e9e4df5319.slice/crio-bb14b796b4fd432bb48d9af564801ffb5f4c0e5db9a9fa6255079e0c29225f4a WatchSource:0}: Error finding container bb14b796b4fd432bb48d9af564801ffb5f4c0e5db9a9fa6255079e0c29225f4a: Status 404 returned error can't find the container with id bb14b796b4fd432bb48d9af564801ffb5f4c0e5db9a9fa6255079e0c29225f4a Jul 22 02:07:11.779567 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:11.779489 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c62fc15d-w8r8f"] Jul 22 02:07:11.951418 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:11.951387 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c62fc15d-w8r8f" event={"ID":"a28fc94e-e8a2-4d1c-9b96-97e9e4df5319","Type":"ContainerStarted","Data":"bb14b796b4fd432bb48d9af564801ffb5f4c0e5db9a9fa6255079e0c29225f4a"} Jul 22 02:07:12.958480 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:12.958445 2561 generic.go:356] "Generic (PLEG): container finished" podID="a28fc94e-e8a2-4d1c-9b96-97e9e4df5319" containerID="3ed42dacd27db06178536d8a8017f9e90205af03c11dde3cf8135f57a66aacaa" exitCode=0 Jul 22 02:07:12.958972 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:12.958503 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c62fc15d-w8r8f" event={"ID":"a28fc94e-e8a2-4d1c-9b96-97e9e4df5319","Type":"ContainerDied","Data":"3ed42dacd27db06178536d8a8017f9e90205af03c11dde3cf8135f57a66aacaa"} Jul 22 02:07:14.106148 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:14.106125 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c62fc15d-w8r8f" Jul 22 02:07:14.254682 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:14.254608 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a28fc94e-e8a2-4d1c-9b96-97e9e4df5319-maas-api-service-ca\") pod \"a28fc94e-e8a2-4d1c-9b96-97e9e4df5319\" (UID: \"a28fc94e-e8a2-4d1c-9b96-97e9e4df5319\") " Jul 22 02:07:14.254947 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:14.254926 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a28fc94e-e8a2-4d1c-9b96-97e9e4df5319-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a28fc94e-e8a2-4d1c-9b96-97e9e4df5319" (UID: "a28fc94e-e8a2-4d1c-9b96-97e9e4df5319"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:07:14.355924 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:14.355895 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a28fc94e-e8a2-4d1c-9b96-97e9e4df5319-maas-api-service-ca\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:07:14.971208 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:14.971173 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c62fc15d-w8r8f" event={"ID":"a28fc94e-e8a2-4d1c-9b96-97e9e4df5319","Type":"ContainerDied","Data":"bb14b796b4fd432bb48d9af564801ffb5f4c0e5db9a9fa6255079e0c29225f4a"} Jul 22 02:07:14.971208 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:14.971202 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c62fc15d-w8r8f" Jul 22 02:07:14.971416 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:14.971218 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb14b796b4fd432bb48d9af564801ffb5f4c0e5db9a9fa6255079e0c29225f4a" Jul 22 02:07:26.249600 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:26.249566 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c62fc15d-w8r8f"] Jul 22 02:07:26.255107 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:26.255079 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c62fc15d-w8r8f"] Jul 22 02:07:26.394446 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:26.394411 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh"] Jul 22 02:07:26.394727 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:26.394657 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" podUID="7e1b3113-7e6d-4dd5-a0ff-911c0409b902" containerName="istio-proxy" containerID="cri-o://fd39b2b3daaac6530a07b39a2636270596af445891cd24146817dcd472443ec0" gracePeriod=30 Jul 22 02:07:27.925466 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:27.925431 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a28fc94e-e8a2-4d1c-9b96-97e9e4df5319" path="/var/lib/kubelet/pods/a28fc94e-e8a2-4d1c-9b96-97e9e4df5319/volumes" Jul 22 02:07:31.652582 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:31.652558 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:07:31.716241 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:31.716212 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istio-podinfo\") pod \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " Jul 22 02:07:31.716380 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:31.716252 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istiod-ca-cert\") pod \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " Jul 22 02:07:31.716380 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:31.716285 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-credential-socket\") pod \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " Jul 22 02:07:31.716380 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:31.716328 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istio-envoy\") pod \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " Jul 22 02:07:31.716380 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:31.716362 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mk8rf\" (UniqueName: \"kubernetes.io/projected/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-kube-api-access-mk8rf\") pod \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " Jul 22 02:07:31.716589 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:31.716388 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istio-token\") pod \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " Jul 22 02:07:31.716589 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:31.716418 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istio-data\") pod \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " Jul 22 02:07:31.719406 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:31.717162 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-workload-socket\") pod \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " Jul 22 02:07:31.719406 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:31.717268 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7e1b3113-7e6d-4dd5-a0ff-911c0409b902" (UID: "7e1b3113-7e6d-4dd5-a0ff-911c0409b902"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:07:31.719406 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:31.717732 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7e1b3113-7e6d-4dd5-a0ff-911c0409b902" (UID: "7e1b3113-7e6d-4dd5-a0ff-911c0409b902"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:07:31.719406 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:31.717986 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7e1b3113-7e6d-4dd5-a0ff-911c0409b902" (UID: "7e1b3113-7e6d-4dd5-a0ff-911c0409b902"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:07:31.719406 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:31.718004 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istio-data" (OuterVolumeSpecName: "istio-data") pod "7e1b3113-7e6d-4dd5-a0ff-911c0409b902" (UID: "7e1b3113-7e6d-4dd5-a0ff-911c0409b902"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:07:31.719406 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:31.717355 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-workload-certs\") pod \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\" (UID: \"7e1b3113-7e6d-4dd5-a0ff-911c0409b902\") " Jul 22 02:07:31.719406 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:31.718111 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7e1b3113-7e6d-4dd5-a0ff-911c0409b902" (UID: "7e1b3113-7e6d-4dd5-a0ff-911c0409b902"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:07:31.719406 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:31.718745 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-workload-certs\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:07:31.719406 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:31.718766 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istiod-ca-cert\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:07:31.719406 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:31.718788 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-credential-socket\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:07:31.719406 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:31.718803 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istio-data\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:07:31.719406 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:31.718818 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-workload-socket\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:07:31.719406 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:31.719303 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7e1b3113-7e6d-4dd5-a0ff-911c0409b902" (UID: "7e1b3113-7e6d-4dd5-a0ff-911c0409b902"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 02:07:31.720131 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:31.719839 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-kube-api-access-mk8rf" (OuterVolumeSpecName: "kube-api-access-mk8rf") pod "7e1b3113-7e6d-4dd5-a0ff-911c0409b902" (UID: "7e1b3113-7e6d-4dd5-a0ff-911c0409b902"). InnerVolumeSpecName "kube-api-access-mk8rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:07:31.726193 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:31.726168 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istio-token" (OuterVolumeSpecName: "istio-token") pod "7e1b3113-7e6d-4dd5-a0ff-911c0409b902" (UID: "7e1b3113-7e6d-4dd5-a0ff-911c0409b902"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:07:31.726290 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:31.726210 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7e1b3113-7e6d-4dd5-a0ff-911c0409b902" (UID: "7e1b3113-7e6d-4dd5-a0ff-911c0409b902"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:07:31.820103 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:31.820080 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istio-envoy\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:07:31.820103 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:31.820102 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk8rf\" (UniqueName: \"kubernetes.io/projected/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-kube-api-access-mk8rf\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:07:31.820247 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:31.820111 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istio-token\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:07:31.820247 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:31.820119 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e1b3113-7e6d-4dd5-a0ff-911c0409b902-istio-podinfo\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:07:32.069536 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:32.069442 2561 generic.go:356] "Generic (PLEG): container finished" podID="7e1b3113-7e6d-4dd5-a0ff-911c0409b902" containerID="fd39b2b3daaac6530a07b39a2636270596af445891cd24146817dcd472443ec0" exitCode=0 Jul 22 02:07:32.069536 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:32.069494 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" event={"ID":"7e1b3113-7e6d-4dd5-a0ff-911c0409b902","Type":"ContainerDied","Data":"fd39b2b3daaac6530a07b39a2636270596af445891cd24146817dcd472443ec0"} Jul 22 02:07:32.069536 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:32.069516 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" Jul 22 02:07:32.069536 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:32.069537 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh" event={"ID":"7e1b3113-7e6d-4dd5-a0ff-911c0409b902","Type":"ContainerDied","Data":"e9fdac7243d4d272776cf8f4f08d68e1d54e8b22f8285ba0a474e62b73617205"} Jul 22 02:07:32.069901 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:32.069558 2561 scope.go:117] "RemoveContainer" containerID="fd39b2b3daaac6530a07b39a2636270596af445891cd24146817dcd472443ec0" Jul 22 02:07:32.078907 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:32.078890 2561 scope.go:117] "RemoveContainer" containerID="1ad07c065be9fcfb146803d28587ecfbae92c08b55bc70485cd59695a3337a7e" Jul 22 02:07:32.087138 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:32.087116 2561 scope.go:117] "RemoveContainer" containerID="fd39b2b3daaac6530a07b39a2636270596af445891cd24146817dcd472443ec0" Jul 22 02:07:32.087397 ip-10-0-131-56 kubenswrapper[2561]: E0722 02:07:32.087376 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd39b2b3daaac6530a07b39a2636270596af445891cd24146817dcd472443ec0\": container with ID starting with fd39b2b3daaac6530a07b39a2636270596af445891cd24146817dcd472443ec0 not found: ID does not exist" containerID="fd39b2b3daaac6530a07b39a2636270596af445891cd24146817dcd472443ec0" Jul 22 02:07:32.087475 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:32.087407 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd39b2b3daaac6530a07b39a2636270596af445891cd24146817dcd472443ec0"} err="failed to get container status \"fd39b2b3daaac6530a07b39a2636270596af445891cd24146817dcd472443ec0\": rpc error: code = NotFound desc = could not find container \"fd39b2b3daaac6530a07b39a2636270596af445891cd24146817dcd472443ec0\": container with ID starting with fd39b2b3daaac6530a07b39a2636270596af445891cd24146817dcd472443ec0 not found: ID does not exist" Jul 22 02:07:32.087475 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:32.087428 2561 scope.go:117] "RemoveContainer" containerID="1ad07c065be9fcfb146803d28587ecfbae92c08b55bc70485cd59695a3337a7e" Jul 22 02:07:32.087683 ip-10-0-131-56 kubenswrapper[2561]: E0722 02:07:32.087656 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ad07c065be9fcfb146803d28587ecfbae92c08b55bc70485cd59695a3337a7e\": container with ID starting with 1ad07c065be9fcfb146803d28587ecfbae92c08b55bc70485cd59695a3337a7e not found: ID does not exist" containerID="1ad07c065be9fcfb146803d28587ecfbae92c08b55bc70485cd59695a3337a7e" Jul 22 02:07:32.087755 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:32.087704 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ad07c065be9fcfb146803d28587ecfbae92c08b55bc70485cd59695a3337a7e"} err="failed to get container status \"1ad07c065be9fcfb146803d28587ecfbae92c08b55bc70485cd59695a3337a7e\": rpc error: code = NotFound desc = could not find container \"1ad07c065be9fcfb146803d28587ecfbae92c08b55bc70485cd59695a3337a7e\": container with ID starting with 1ad07c065be9fcfb146803d28587ecfbae92c08b55bc70485cd59695a3337a7e not found: ID does not exist" Jul 22 02:07:32.093138 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:32.093116 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh"] Jul 22 02:07:32.095455 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:32.095436 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-c62fc15d-openshift-default-99fc64b9f-67jhh"] Jul 22 02:07:33.926102 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:07:33.926069 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e1b3113-7e6d-4dd5-a0ff-911c0409b902" path="/var/lib/kubelet/pods/7e1b3113-7e6d-4dd5-a0ff-911c0409b902/volumes" Jul 22 02:09:21.207353 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:21.207311 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32942ebc-qshtq"] Jul 22 02:09:21.207824 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:21.207808 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e1b3113-7e6d-4dd5-a0ff-911c0409b902" containerName="istio-proxy" Jul 22 02:09:21.207879 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:21.207826 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e1b3113-7e6d-4dd5-a0ff-911c0409b902" containerName="istio-proxy" Jul 22 02:09:21.207879 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:21.207850 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e1b3113-7e6d-4dd5-a0ff-911c0409b902" containerName="istio-proxy" Jul 22 02:09:21.207879 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:21.207855 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e1b3113-7e6d-4dd5-a0ff-911c0409b902" containerName="istio-proxy" Jul 22 02:09:21.207879 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:21.207868 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a28fc94e-e8a2-4d1c-9b96-97e9e4df5319" containerName="revoke-keys" Jul 22 02:09:21.207879 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:21.207877 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a28fc94e-e8a2-4d1c-9b96-97e9e4df5319" containerName="revoke-keys" Jul 22 02:09:21.208031 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:21.207940 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a28fc94e-e8a2-4d1c-9b96-97e9e4df5319" containerName="revoke-keys" Jul 22 02:09:21.208031 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:21.207953 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e1b3113-7e6d-4dd5-a0ff-911c0409b902" containerName="istio-proxy" Jul 22 02:09:21.211195 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:21.211179 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32942ebc-qshtq" Jul 22 02:09:21.213445 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:21.213423 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cqdw8\"" Jul 22 02:09:21.219561 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:21.219536 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32942ebc-qshtq"] Jul 22 02:09:21.263063 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:21.263031 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d789fd-0306-4667-8b4c-1d788d994146-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-32942ebc-qshtq\" (UID: \"73d789fd-0306-4667-8b4c-1d788d994146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32942ebc-qshtq" Jul 22 02:09:21.364274 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:21.364243 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d789fd-0306-4667-8b4c-1d788d994146-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-32942ebc-qshtq\" (UID: \"73d789fd-0306-4667-8b4c-1d788d994146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32942ebc-qshtq" Jul 22 02:09:21.365001 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:21.364982 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d789fd-0306-4667-8b4c-1d788d994146-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-32942ebc-qshtq\" (UID: \"73d789fd-0306-4667-8b4c-1d788d994146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32942ebc-qshtq" Jul 22 02:09:21.524902 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:21.524834 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32942ebc-qshtq" Jul 22 02:09:21.673178 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:21.673145 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32942ebc-qshtq"] Jul 22 02:09:21.674231 ip-10-0-131-56 kubenswrapper[2561]: W0722 02:09:21.674209 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d789fd_0306_4667_8b4c_1d788d994146.slice/crio-eca5a76e2cb5a7dd6abd3de8abda6a099ff0fe0f017f1a4d55d881007003dd32 WatchSource:0}: Error finding container eca5a76e2cb5a7dd6abd3de8abda6a099ff0fe0f017f1a4d55d881007003dd32: Status 404 returned error can't find the container with id eca5a76e2cb5a7dd6abd3de8abda6a099ff0fe0f017f1a4d55d881007003dd32 Jul 22 02:09:21.696441 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:21.696407 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32942ebc-qshtq" event={"ID":"73d789fd-0306-4667-8b4c-1d788d994146","Type":"ContainerStarted","Data":"eca5a76e2cb5a7dd6abd3de8abda6a099ff0fe0f017f1a4d55d881007003dd32"} Jul 22 02:09:22.704155 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:22.704122 2561 generic.go:356] "Generic (PLEG): container finished" podID="73d789fd-0306-4667-8b4c-1d788d994146" containerID="e7799e6a7af13c6310cef07fad1151e1e49490d3a35c93535182b7929d8a9615" exitCode=0 Jul 22 02:09:22.704528 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:22.704210 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32942ebc-qshtq" event={"ID":"73d789fd-0306-4667-8b4c-1d788d994146","Type":"ContainerDied","Data":"e7799e6a7af13c6310cef07fad1151e1e49490d3a35c93535182b7929d8a9615"} Jul 22 02:09:23.843245 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:23.843223 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32942ebc-qshtq" Jul 22 02:09:23.882891 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:23.882868 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d789fd-0306-4667-8b4c-1d788d994146-maas-api-service-ca\") pod \"73d789fd-0306-4667-8b4c-1d788d994146\" (UID: \"73d789fd-0306-4667-8b4c-1d788d994146\") " Jul 22 02:09:23.883206 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:23.883184 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73d789fd-0306-4667-8b4c-1d788d994146-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "73d789fd-0306-4667-8b4c-1d788d994146" (UID: "73d789fd-0306-4667-8b4c-1d788d994146"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:09:23.983605 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:23.983533 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d789fd-0306-4667-8b4c-1d788d994146-maas-api-service-ca\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:09:24.716753 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:24.716722 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32942ebc-qshtq" Jul 22 02:09:24.716930 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:24.716725 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32942ebc-qshtq" event={"ID":"73d789fd-0306-4667-8b4c-1d788d994146","Type":"ContainerDied","Data":"eca5a76e2cb5a7dd6abd3de8abda6a099ff0fe0f017f1a4d55d881007003dd32"} Jul 22 02:09:24.716930 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:24.716843 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eca5a76e2cb5a7dd6abd3de8abda6a099ff0fe0f017f1a4d55d881007003dd32" Jul 22 02:09:36.290338 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:36.290296 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32942ebc-qshtq"] Jul 22 02:09:36.292826 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:36.292800 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-32942ebc-qshtq"] Jul 22 02:09:37.925539 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:09:37.925496 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73d789fd-0306-4667-8b4c-1d788d994146" path="/var/lib/kubelet/pods/73d789fd-0306-4667-8b4c-1d788d994146/volumes" Jul 22 02:10:33.952583 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:10:33.952553 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb9ad51b-zkp8r"] Jul 22 02:10:33.953082 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:10:33.953038 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73d789fd-0306-4667-8b4c-1d788d994146" containerName="revoke-keys" Jul 22 02:10:33.953082 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:10:33.953050 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="73d789fd-0306-4667-8b4c-1d788d994146" containerName="revoke-keys" Jul 22 02:10:33.953171 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:10:33.953128 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e1b3113-7e6d-4dd5-a0ff-911c0409b902" containerName="istio-proxy" Jul 22 02:10:33.953171 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:10:33.953139 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="73d789fd-0306-4667-8b4c-1d788d994146" containerName="revoke-keys" Jul 22 02:10:33.956318 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:10:33.956301 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb9ad51b-zkp8r" Jul 22 02:10:33.958705 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:10:33.958674 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cqdw8\"" Jul 22 02:10:33.964416 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:10:33.964058 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb9ad51b-zkp8r"] Jul 22 02:10:33.985970 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:10:33.985940 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07234ff5-eb14-4807-b83c-2f3f0d6550fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb9ad51b-zkp8r\" (UID: \"07234ff5-eb14-4807-b83c-2f3f0d6550fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb9ad51b-zkp8r" Jul 22 02:10:34.086443 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:10:34.086405 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07234ff5-eb14-4807-b83c-2f3f0d6550fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb9ad51b-zkp8r\" (UID: \"07234ff5-eb14-4807-b83c-2f3f0d6550fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb9ad51b-zkp8r" Jul 22 02:10:34.087096 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:10:34.087071 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07234ff5-eb14-4807-b83c-2f3f0d6550fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb9ad51b-zkp8r\" (UID: \"07234ff5-eb14-4807-b83c-2f3f0d6550fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb9ad51b-zkp8r" Jul 22 02:10:34.269399 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:10:34.269325 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb9ad51b-zkp8r" Jul 22 02:10:34.616369 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:10:34.616343 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb9ad51b-zkp8r"] Jul 22 02:10:34.618020 ip-10-0-131-56 kubenswrapper[2561]: W0722 02:10:34.617995 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07234ff5_eb14_4807_b83c_2f3f0d6550fe.slice/crio-2f6f3eb5efc42ac3bf3aff01a989174734756afb87bb45b2580071fa4801376d WatchSource:0}: Error finding container 2f6f3eb5efc42ac3bf3aff01a989174734756afb87bb45b2580071fa4801376d: Status 404 returned error can't find the container with id 2f6f3eb5efc42ac3bf3aff01a989174734756afb87bb45b2580071fa4801376d Jul 22 02:10:35.131370 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:10:35.131321 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb9ad51b-zkp8r" event={"ID":"07234ff5-eb14-4807-b83c-2f3f0d6550fe","Type":"ContainerStarted","Data":"e71213406191e93fbdf401a376c07690f5f72b06a8387d7bdb9f751de252e532"} Jul 22 02:10:35.131852 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:10:35.131388 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb9ad51b-zkp8r" event={"ID":"07234ff5-eb14-4807-b83c-2f3f0d6550fe","Type":"ContainerStarted","Data":"2f6f3eb5efc42ac3bf3aff01a989174734756afb87bb45b2580071fa4801376d"} Jul 22 02:10:35.148958 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:10:35.148899 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb9ad51b-zkp8r" podStartSLOduration=2.148879588 podStartE2EDuration="2.148879588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:10:35.147156561 +0000 UTC m=+2005.739960946" watchObservedRunningTime="2026-07-22 02:10:35.148879588 +0000 UTC m=+2005.741683955" Jul 22 02:10:36.139119 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:10:36.139083 2561 generic.go:356] "Generic (PLEG): container finished" podID="07234ff5-eb14-4807-b83c-2f3f0d6550fe" containerID="e71213406191e93fbdf401a376c07690f5f72b06a8387d7bdb9f751de252e532" exitCode=0 Jul 22 02:10:36.139551 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:10:36.139176 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb9ad51b-zkp8r" event={"ID":"07234ff5-eb14-4807-b83c-2f3f0d6550fe","Type":"ContainerDied","Data":"e71213406191e93fbdf401a376c07690f5f72b06a8387d7bdb9f751de252e532"} Jul 22 02:10:37.283080 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:10:37.283054 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb9ad51b-zkp8r" Jul 22 02:10:37.417555 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:10:37.417481 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07234ff5-eb14-4807-b83c-2f3f0d6550fe-maas-api-service-ca\") pod \"07234ff5-eb14-4807-b83c-2f3f0d6550fe\" (UID: \"07234ff5-eb14-4807-b83c-2f3f0d6550fe\") " Jul 22 02:10:37.417881 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:10:37.417860 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07234ff5-eb14-4807-b83c-2f3f0d6550fe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "07234ff5-eb14-4807-b83c-2f3f0d6550fe" (UID: "07234ff5-eb14-4807-b83c-2f3f0d6550fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:10:37.518919 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:10:37.518892 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07234ff5-eb14-4807-b83c-2f3f0d6550fe-maas-api-service-ca\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:10:38.152134 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:10:38.152107 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb9ad51b-zkp8r" Jul 22 02:10:38.152134 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:10:38.152118 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb9ad51b-zkp8r" event={"ID":"07234ff5-eb14-4807-b83c-2f3f0d6550fe","Type":"ContainerDied","Data":"2f6f3eb5efc42ac3bf3aff01a989174734756afb87bb45b2580071fa4801376d"} Jul 22 02:10:38.152323 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:10:38.152169 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f6f3eb5efc42ac3bf3aff01a989174734756afb87bb45b2580071fa4801376d" Jul 22 02:10:54.041665 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:10:54.041627 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb9ad51b-zkp8r"] Jul 22 02:10:54.045718 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:10:54.045678 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb9ad51b-zkp8r"] Jul 22 02:10:55.924847 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:10:55.924810 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07234ff5-eb14-4807-b83c-2f3f0d6550fe" path="/var/lib/kubelet/pods/07234ff5-eb14-4807-b83c-2f3f0d6550fe/volumes" Jul 22 02:12:50.150924 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.150887 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf"] Jul 22 02:12:50.151613 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.151517 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07234ff5-eb14-4807-b83c-2f3f0d6550fe" containerName="revoke-keys" Jul 22 02:12:50.151613 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.151533 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="07234ff5-eb14-4807-b83c-2f3f0d6550fe" containerName="revoke-keys" Jul 22 02:12:50.151613 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.151603 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="07234ff5-eb14-4807-b83c-2f3f0d6550fe" containerName="revoke-keys" Jul 22 02:12:50.154763 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.154737 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.159128 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.159111 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 22 02:12:50.159254 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.159162 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1ad649-openshift-default-dockercfg-4spgr\"" Jul 22 02:12:50.172504 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.172475 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf"] Jul 22 02:12:50.265978 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.265943 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-istio-envoy\") pod \"e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.266151 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.265990 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwdzd\" (UniqueName: \"kubernetes.io/projected/ce498ada-4f85-437f-827f-0e5b0a438bd0-kube-api-access-cwdzd\") pod \"e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.266151 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.266052 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-workload-certs\") pod \"e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.266151 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.266110 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce498ada-4f85-437f-827f-0e5b0a438bd0-istio-token\") pod \"e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.266151 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.266134 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-workload-socket\") pod \"e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.266376 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.266156 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-istio-data\") pod \"e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.266376 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.266235 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-credential-socket\") pod \"e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.266376 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.266269 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce498ada-4f85-437f-827f-0e5b0a438bd0-istio-podinfo\") pod \"e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.266376 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.266341 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce498ada-4f85-437f-827f-0e5b0a438bd0-istiod-ca-cert\") pod \"e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.367311 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.367275 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce498ada-4f85-437f-827f-0e5b0a438bd0-istiod-ca-cert\") pod \"e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.367462 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.367352 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-istio-envoy\") pod \"e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.367462 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.367393 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwdzd\" (UniqueName: \"kubernetes.io/projected/ce498ada-4f85-437f-827f-0e5b0a438bd0-kube-api-access-cwdzd\") pod \"e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.367582 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.367523 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-workload-certs\") pod \"e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.367582 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.367566 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce498ada-4f85-437f-827f-0e5b0a438bd0-istio-token\") pod \"e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.367687 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.367592 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-workload-socket\") pod \"e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.367687 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.367615 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-istio-data\") pod \"e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.367687 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.367657 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-credential-socket\") pod \"e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.367687 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.367690 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce498ada-4f85-437f-827f-0e5b0a438bd0-istio-podinfo\") pod \"e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.367958 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.367877 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-workload-certs\") pod \"e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.367958 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.367945 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-workload-socket\") pod \"e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.368073 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.368042 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce498ada-4f85-437f-827f-0e5b0a438bd0-istiod-ca-cert\") pod \"e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.368073 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.368063 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-istio-data\") pod \"e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.368334 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.368311 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-credential-socket\") pod \"e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.369806 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.369787 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-istio-envoy\") pod \"e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.370169 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.370152 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce498ada-4f85-437f-827f-0e5b0a438bd0-istio-podinfo\") pod \"e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.376866 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.376839 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwdzd\" (UniqueName: \"kubernetes.io/projected/ce498ada-4f85-437f-827f-0e5b0a438bd0-kube-api-access-cwdzd\") pod \"e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.377213 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.377198 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce498ada-4f85-437f-827f-0e5b0a438bd0-istio-token\") pod \"e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.465377 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.465350 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:50.619559 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.619531 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf"] Jul 22 02:12:50.620504 ip-10-0-131-56 kubenswrapper[2561]: W0722 02:12:50.620474 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce498ada_4f85_437f_827f_0e5b0a438bd0.slice/crio-6b7cf41c78339786b13e6652a7e05777cc952b31327d280d65071aea9568d9b8 WatchSource:0}: Error finding container 6b7cf41c78339786b13e6652a7e05777cc952b31327d280d65071aea9568d9b8: Status 404 returned error can't find the container with id 6b7cf41c78339786b13e6652a7e05777cc952b31327d280d65071aea9568d9b8 Jul 22 02:12:50.622787 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.622756 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 22 02:12:50.622862 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.622822 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 22 02:12:50.622862 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.622849 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 22 02:12:50.935648 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.935609 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" event={"ID":"ce498ada-4f85-437f-827f-0e5b0a438bd0","Type":"ContainerStarted","Data":"be07df7a83c28af92859b3ad365d4e865deca2da1a7c3ac7c7e1ea01d887682d"} Jul 22 02:12:50.935852 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.935675 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" event={"ID":"ce498ada-4f85-437f-827f-0e5b0a438bd0","Type":"ContainerStarted","Data":"6b7cf41c78339786b13e6652a7e05777cc952b31327d280d65071aea9568d9b8"} Jul 22 02:12:50.958658 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:50.958613 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" podStartSLOduration=0.958597847 podStartE2EDuration="958.597847ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:12:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:12:50.956879478 +0000 UTC m=+2141.549683862" watchObservedRunningTime="2026-07-22 02:12:50.958597847 +0000 UTC m=+2141.551402285" Jul 22 02:12:51.465865 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:51.465832 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:51.471079 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:51.471055 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:51.942299 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:51.942261 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:12:51.943129 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:12:51.943109 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:13:05.019668 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:13:05.019631 2561 generic.go:356] "Generic (PLEG): container finished" podID="ce498ada-4f85-437f-827f-0e5b0a438bd0" containerID="be07df7a83c28af92859b3ad365d4e865deca2da1a7c3ac7c7e1ea01d887682d" exitCode=0 Jul 22 02:13:05.020073 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:13:05.019730 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" event={"ID":"ce498ada-4f85-437f-827f-0e5b0a438bd0","Type":"ContainerDied","Data":"be07df7a83c28af92859b3ad365d4e865deca2da1a7c3ac7c7e1ea01d887682d"} Jul 22 02:13:05.020145 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:13:05.020128 2561 scope.go:117] "RemoveContainer" containerID="be07df7a83c28af92859b3ad365d4e865deca2da1a7c3ac7c7e1ea01d887682d" Jul 22 02:13:05.021121 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:13:05.021097 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 22 02:13:05.021195 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:13:05.021175 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 22 02:13:05.021244 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:13:05.021214 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 22 02:13:06.027317 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:13:06.027272 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" event={"ID":"ce498ada-4f85-437f-827f-0e5b0a438bd0","Type":"ContainerStarted","Data":"66f5871c7c32c84b548b44dfc11d2d5f51123221bc330842054a3cf3953fa036"} Jul 22 02:13:06.466020 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:13:06.465980 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:13:06.467066 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:13:06.467037 2561 patch_prober.go:28] interesting pod/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Jul 22 02:13:06.467188 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:13:06.467091 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" podUID="ce498ada-4f85-437f-827f-0e5b0a438bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Jul 22 02:13:06.942393 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:13:06.942350 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:13:07.466204 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:13:07.466168 2561 patch_prober.go:28] interesting pod/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Jul 22 02:13:07.466586 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:13:07.466240 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" podUID="ce498ada-4f85-437f-827f-0e5b0a438bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Jul 22 02:13:08.465862 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:13:08.465820 2561 patch_prober.go:28] interesting pod/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Jul 22 02:13:08.466049 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:13:08.465899 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" podUID="ce498ada-4f85-437f-827f-0e5b0a438bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Jul 22 02:13:09.466455 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:13:09.466418 2561 patch_prober.go:28] interesting pod/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Jul 22 02:13:09.466880 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:13:09.466470 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" podUID="ce498ada-4f85-437f-827f-0e5b0a438bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Jul 22 02:13:10.469743 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:13:10.469708 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:13:10.470798 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:13:10.470782 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:13:11.178019 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:13:11.177987 2561 scope.go:117] "RemoveContainer" containerID="b3c5226a0ee4a12c05de0a26230cf5f512ce6c4cc692f0257cce5767184ae56f" Jul 22 02:14:11.199646 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:11.199613 2561 scope.go:117] "RemoveContainer" containerID="3ed42dacd27db06178536d8a8017f9e90205af03c11dde3cf8135f57a66aacaa" Jul 22 02:14:29.345082 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:29.345045 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0ac34-q7xjv"] Jul 22 02:14:29.349538 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:29.349516 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0ac34-q7xjv" Jul 22 02:14:29.352205 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:29.352183 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cqdw8\"" Jul 22 02:14:29.359401 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:29.359380 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0ac34-q7xjv"] Jul 22 02:14:29.440496 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:29.440461 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70b94c0-5d2d-44df-9bb5-7041cc66b6d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b0ac34-q7xjv\" (UID: \"f70b94c0-5d2d-44df-9bb5-7041cc66b6d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0ac34-q7xjv" Jul 22 02:14:29.541620 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:29.541584 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70b94c0-5d2d-44df-9bb5-7041cc66b6d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b0ac34-q7xjv\" (UID: \"f70b94c0-5d2d-44df-9bb5-7041cc66b6d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0ac34-q7xjv" Jul 22 02:14:29.542217 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:29.542198 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70b94c0-5d2d-44df-9bb5-7041cc66b6d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b0ac34-q7xjv\" (UID: \"f70b94c0-5d2d-44df-9bb5-7041cc66b6d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0ac34-q7xjv" Jul 22 02:14:29.663545 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:29.663457 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0ac34-q7xjv" Jul 22 02:14:29.823715 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:29.823672 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0ac34-q7xjv"] Jul 22 02:14:29.824900 ip-10-0-131-56 kubenswrapper[2561]: W0722 02:14:29.824871 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70b94c0_5d2d_44df_9bb5_7041cc66b6d3.slice/crio-69bf849a522972b976dd4d756bbba5d427936d42ed635d98ade06a94ea10eda0 WatchSource:0}: Error finding container 69bf849a522972b976dd4d756bbba5d427936d42ed635d98ade06a94ea10eda0: Status 404 returned error can't find the container with id 69bf849a522972b976dd4d756bbba5d427936d42ed635d98ade06a94ea10eda0 Jul 22 02:14:30.531245 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:30.531207 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0ac34-q7xjv" event={"ID":"f70b94c0-5d2d-44df-9bb5-7041cc66b6d3","Type":"ContainerStarted","Data":"6fa5c994ac72f619d6bc292b6db80de88c72567a2ef80ba9f76af9e598125806"} Jul 22 02:14:30.531833 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:30.531796 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0ac34-q7xjv" event={"ID":"f70b94c0-5d2d-44df-9bb5-7041cc66b6d3","Type":"ContainerStarted","Data":"69bf849a522972b976dd4d756bbba5d427936d42ed635d98ade06a94ea10eda0"} Jul 22 02:14:30.557260 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:30.557208 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0ac34-q7xjv" podStartSLOduration=1.557190442 podStartE2EDuration="1.557190442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:14:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:14:30.555189169 +0000 UTC m=+2241.147993559" watchObservedRunningTime="2026-07-22 02:14:30.557190442 +0000 UTC m=+2241.149994807" Jul 22 02:14:31.538813 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:31.538781 2561 generic.go:356] "Generic (PLEG): container finished" podID="f70b94c0-5d2d-44df-9bb5-7041cc66b6d3" containerID="6fa5c994ac72f619d6bc292b6db80de88c72567a2ef80ba9f76af9e598125806" exitCode=0 Jul 22 02:14:31.539205 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:31.538868 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0ac34-q7xjv" event={"ID":"f70b94c0-5d2d-44df-9bb5-7041cc66b6d3","Type":"ContainerDied","Data":"6fa5c994ac72f619d6bc292b6db80de88c72567a2ef80ba9f76af9e598125806"} Jul 22 02:14:32.685827 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:32.685800 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0ac34-q7xjv" Jul 22 02:14:32.769645 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:32.769609 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70b94c0-5d2d-44df-9bb5-7041cc66b6d3-maas-api-service-ca\") pod \"f70b94c0-5d2d-44df-9bb5-7041cc66b6d3\" (UID: \"f70b94c0-5d2d-44df-9bb5-7041cc66b6d3\") " Jul 22 02:14:32.769963 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:32.769940 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70b94c0-5d2d-44df-9bb5-7041cc66b6d3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f70b94c0-5d2d-44df-9bb5-7041cc66b6d3" (UID: "f70b94c0-5d2d-44df-9bb5-7041cc66b6d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:14:32.870979 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:32.870916 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70b94c0-5d2d-44df-9bb5-7041cc66b6d3-maas-api-service-ca\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:14:33.552159 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:33.552127 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0ac34-q7xjv" Jul 22 02:14:33.552159 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:33.552140 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0ac34-q7xjv" event={"ID":"f70b94c0-5d2d-44df-9bb5-7041cc66b6d3","Type":"ContainerDied","Data":"69bf849a522972b976dd4d756bbba5d427936d42ed635d98ade06a94ea10eda0"} Jul 22 02:14:33.552367 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:33.552180 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69bf849a522972b976dd4d756bbba5d427936d42ed635d98ade06a94ea10eda0" Jul 22 02:14:44.466087 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:44.466050 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0ac34-q7xjv"] Jul 22 02:14:44.471250 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:44.471226 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b0ac34-q7xjv"] Jul 22 02:14:45.925055 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:45.925018 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f70b94c0-5d2d-44df-9bb5-7041cc66b6d3" path="/var/lib/kubelet/pods/f70b94c0-5d2d-44df-9bb5-7041cc66b6d3/volumes" Jul 22 02:14:52.153757 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:52.153712 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1ad649-2nfjv"] Jul 22 02:14:52.154200 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:52.154159 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70b94c0-5d2d-44df-9bb5-7041cc66b6d3" containerName="revoke-keys" Jul 22 02:14:52.154200 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:52.154172 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70b94c0-5d2d-44df-9bb5-7041cc66b6d3" containerName="revoke-keys" Jul 22 02:14:52.154272 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:52.154259 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f70b94c0-5d2d-44df-9bb5-7041cc66b6d3" containerName="revoke-keys" Jul 22 02:14:52.157566 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:52.157547 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1ad649-2nfjv" Jul 22 02:14:52.160763 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:52.160746 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cqdw8\"" Jul 22 02:14:52.168061 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:52.168039 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1ad649-2nfjv"] Jul 22 02:14:52.246954 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:52.246920 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/190925df-1eec-40fb-b4b6-705fa2657212-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1ad649-2nfjv\" (UID: \"190925df-1eec-40fb-b4b6-705fa2657212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1ad649-2nfjv" Jul 22 02:14:52.348304 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:52.348272 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/190925df-1eec-40fb-b4b6-705fa2657212-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1ad649-2nfjv\" (UID: \"190925df-1eec-40fb-b4b6-705fa2657212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1ad649-2nfjv" Jul 22 02:14:52.348955 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:52.348935 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/190925df-1eec-40fb-b4b6-705fa2657212-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1ad649-2nfjv\" (UID: \"190925df-1eec-40fb-b4b6-705fa2657212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1ad649-2nfjv" Jul 22 02:14:52.469926 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:52.469896 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1ad649-2nfjv" Jul 22 02:14:52.645757 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:52.645726 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1ad649-2nfjv"] Jul 22 02:14:52.646624 ip-10-0-131-56 kubenswrapper[2561]: W0722 02:14:52.646598 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190925df_1eec_40fb_b4b6_705fa2657212.slice/crio-d8b7b8517b75b40695c006a9c1a6d920966fc35730ffbf6240df63a835a0d1c1 WatchSource:0}: Error finding container d8b7b8517b75b40695c006a9c1a6d920966fc35730ffbf6240df63a835a0d1c1: Status 404 returned error can't find the container with id d8b7b8517b75b40695c006a9c1a6d920966fc35730ffbf6240df63a835a0d1c1 Jul 22 02:14:52.677764 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:52.677734 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1ad649-2nfjv" event={"ID":"190925df-1eec-40fb-b4b6-705fa2657212","Type":"ContainerStarted","Data":"d8b7b8517b75b40695c006a9c1a6d920966fc35730ffbf6240df63a835a0d1c1"} Jul 22 02:14:53.690520 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:53.690468 2561 generic.go:356] "Generic (PLEG): container finished" podID="190925df-1eec-40fb-b4b6-705fa2657212" containerID="1a20049a264df3a4a82694eeeba56908c9a65b12be6391f00e5fdd17d680182d" exitCode=0 Jul 22 02:14:53.690973 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:53.690595 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1ad649-2nfjv" event={"ID":"190925df-1eec-40fb-b4b6-705fa2657212","Type":"ContainerDied","Data":"1a20049a264df3a4a82694eeeba56908c9a65b12be6391f00e5fdd17d680182d"} Jul 22 02:14:54.837989 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:54.837966 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1ad649-2nfjv" Jul 22 02:14:54.869974 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:54.869946 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/190925df-1eec-40fb-b4b6-705fa2657212-maas-api-service-ca\") pod \"190925df-1eec-40fb-b4b6-705fa2657212\" (UID: \"190925df-1eec-40fb-b4b6-705fa2657212\") " Jul 22 02:14:54.870281 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:54.870254 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/190925df-1eec-40fb-b4b6-705fa2657212-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "190925df-1eec-40fb-b4b6-705fa2657212" (UID: "190925df-1eec-40fb-b4b6-705fa2657212"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:14:54.971685 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:54.971614 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/190925df-1eec-40fb-b4b6-705fa2657212-maas-api-service-ca\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:14:55.702827 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:55.702790 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1ad649-2nfjv" Jul 22 02:14:55.702995 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:55.702827 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1ad649-2nfjv" event={"ID":"190925df-1eec-40fb-b4b6-705fa2657212","Type":"ContainerDied","Data":"d8b7b8517b75b40695c006a9c1a6d920966fc35730ffbf6240df63a835a0d1c1"} Jul 22 02:14:55.702995 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:14:55.702874 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8b7b8517b75b40695c006a9c1a6d920966fc35730ffbf6240df63a835a0d1c1" Jul 22 02:15:00.151883 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:00.151844 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744775-5nwbw"] Jul 22 02:15:00.152357 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:00.152342 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="190925df-1eec-40fb-b4b6-705fa2657212" containerName="revoke-keys" Jul 22 02:15:00.152357 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:00.152358 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="190925df-1eec-40fb-b4b6-705fa2657212" containerName="revoke-keys" Jul 22 02:15:00.152468 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:00.152458 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="190925df-1eec-40fb-b4b6-705fa2657212" containerName="revoke-keys" Jul 22 02:15:00.155617 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:00.155596 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744775-5nwbw" Jul 22 02:15:00.158163 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:00.158146 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cqdw8\"" Jul 22 02:15:00.165250 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:00.165225 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744775-5nwbw"] Jul 22 02:15:00.167327 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:00.167307 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744775-5nwbw" Jul 22 02:15:00.309509 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:00.309485 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744775-5nwbw"] Jul 22 02:15:00.311415 ip-10-0-131-56 kubenswrapper[2561]: W0722 02:15:00.311391 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83846c24_5e11_4da6_8628_3b6b62964476.slice/crio-877c285e733f8848670dbb41e0de5226eccfb0e571257edc96024f9675931c3e WatchSource:0}: Error finding container 877c285e733f8848670dbb41e0de5226eccfb0e571257edc96024f9675931c3e: Status 404 returned error can't find the container with id 877c285e733f8848670dbb41e0de5226eccfb0e571257edc96024f9675931c3e Jul 22 02:15:00.732938 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:00.732902 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744775-5nwbw" event={"ID":"83846c24-5e11-4da6-8628-3b6b62964476","Type":"ContainerStarted","Data":"932f48b190dd5fe58a12a99704534818f95296987e308c5bf3774a6d6137f04e"} Jul 22 02:15:00.733065 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:00.732966 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744775-5nwbw" event={"ID":"83846c24-5e11-4da6-8628-3b6b62964476","Type":"ContainerStarted","Data":"877c285e733f8848670dbb41e0de5226eccfb0e571257edc96024f9675931c3e"} Jul 22 02:15:00.821770 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:00.821668 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744775-5nwbw" podStartSLOduration=0.821653199 podStartE2EDuration="821.653199ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:15:00.819172135 +0000 UTC m=+2271.411976498" watchObservedRunningTime="2026-07-22 02:15:00.821653199 +0000 UTC m=+2271.414457566" Jul 22 02:15:01.740546 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:01.740511 2561 generic.go:356] "Generic (PLEG): container finished" podID="83846c24-5e11-4da6-8628-3b6b62964476" containerID="932f48b190dd5fe58a12a99704534818f95296987e308c5bf3774a6d6137f04e" exitCode=0 Jul 22 02:15:01.740929 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:01.740598 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744775-5nwbw" event={"ID":"83846c24-5e11-4da6-8628-3b6b62964476","Type":"ContainerDied","Data":"932f48b190dd5fe58a12a99704534818f95296987e308c5bf3774a6d6137f04e"} Jul 22 02:15:02.908509 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:02.908487 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744775-5nwbw" Jul 22 02:15:03.754463 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:03.754416 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744775-5nwbw" event={"ID":"83846c24-5e11-4da6-8628-3b6b62964476","Type":"ContainerDied","Data":"877c285e733f8848670dbb41e0de5226eccfb0e571257edc96024f9675931c3e"} Jul 22 02:15:03.754463 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:03.754442 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744775-5nwbw" Jul 22 02:15:03.754658 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:03.754472 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="877c285e733f8848670dbb41e0de5226eccfb0e571257edc96024f9675931c3e" Jul 22 02:15:07.283712 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:07.283660 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1ad649-2nfjv"] Jul 22 02:15:07.286273 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:07.286251 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1ad649-2nfjv"] Jul 22 02:15:07.925357 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:07.925322 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="190925df-1eec-40fb-b4b6-705fa2657212" path="/var/lib/kubelet/pods/190925df-1eec-40fb-b4b6-705fa2657212/volumes" Jul 22 02:15:14.379911 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:14.379880 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf"] Jul 22 02:15:14.380309 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:14.380086 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" podUID="ce498ada-4f85-437f-827f-0e5b0a438bd0" containerName="istio-proxy" containerID="cri-o://66f5871c7c32c84b548b44dfc11d2d5f51123221bc330842054a3cf3953fa036" gracePeriod=30 Jul 22 02:15:19.646959 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.646893 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:15:19.801706 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.801663 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce498ada-4f85-437f-827f-0e5b0a438bd0-istiod-ca-cert\") pod \"ce498ada-4f85-437f-827f-0e5b0a438bd0\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " Jul 22 02:15:19.801854 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.801719 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce498ada-4f85-437f-827f-0e5b0a438bd0-istio-podinfo\") pod \"ce498ada-4f85-437f-827f-0e5b0a438bd0\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " Jul 22 02:15:19.801854 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.801747 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce498ada-4f85-437f-827f-0e5b0a438bd0-istio-token\") pod \"ce498ada-4f85-437f-827f-0e5b0a438bd0\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " Jul 22 02:15:19.801854 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.801767 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-istio-envoy\") pod \"ce498ada-4f85-437f-827f-0e5b0a438bd0\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " Jul 22 02:15:19.801854 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.801809 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-credential-socket\") pod \"ce498ada-4f85-437f-827f-0e5b0a438bd0\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " Jul 22 02:15:19.802046 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.801892 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwdzd\" (UniqueName: \"kubernetes.io/projected/ce498ada-4f85-437f-827f-0e5b0a438bd0-kube-api-access-cwdzd\") pod \"ce498ada-4f85-437f-827f-0e5b0a438bd0\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " Jul 22 02:15:19.802046 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.801938 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-workload-certs\") pod \"ce498ada-4f85-437f-827f-0e5b0a438bd0\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " Jul 22 02:15:19.802046 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.802013 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-istio-data\") pod \"ce498ada-4f85-437f-827f-0e5b0a438bd0\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " Jul 22 02:15:19.802199 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.802047 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-workload-socket\") pod \"ce498ada-4f85-437f-827f-0e5b0a438bd0\" (UID: \"ce498ada-4f85-437f-827f-0e5b0a438bd0\") " Jul 22 02:15:19.802199 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.802126 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ce498ada-4f85-437f-827f-0e5b0a438bd0" (UID: "ce498ada-4f85-437f-827f-0e5b0a438bd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:15:19.802199 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.802149 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce498ada-4f85-437f-827f-0e5b0a438bd0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ce498ada-4f85-437f-827f-0e5b0a438bd0" (UID: "ce498ada-4f85-437f-827f-0e5b0a438bd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:15:19.802420 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.802353 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ce498ada-4f85-437f-827f-0e5b0a438bd0" (UID: "ce498ada-4f85-437f-827f-0e5b0a438bd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:15:19.802420 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.802389 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce498ada-4f85-437f-827f-0e5b0a438bd0-istiod-ca-cert\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:15:19.802420 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.802408 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-credential-socket\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:15:19.802813 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.802688 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ce498ada-4f85-437f-827f-0e5b0a438bd0" (UID: "ce498ada-4f85-437f-827f-0e5b0a438bd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:15:19.802929 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.802904 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-istio-data" (OuterVolumeSpecName: "istio-data") pod "ce498ada-4f85-437f-827f-0e5b0a438bd0" (UID: "ce498ada-4f85-437f-827f-0e5b0a438bd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:15:19.804061 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.804041 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ce498ada-4f85-437f-827f-0e5b0a438bd0" (UID: "ce498ada-4f85-437f-827f-0e5b0a438bd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:15:19.804151 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.804135 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce498ada-4f85-437f-827f-0e5b0a438bd0-kube-api-access-cwdzd" (OuterVolumeSpecName: "kube-api-access-cwdzd") pod "ce498ada-4f85-437f-827f-0e5b0a438bd0" (UID: "ce498ada-4f85-437f-827f-0e5b0a438bd0"). InnerVolumeSpecName "kube-api-access-cwdzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:15:19.804590 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.804564 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce498ada-4f85-437f-827f-0e5b0a438bd0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ce498ada-4f85-437f-827f-0e5b0a438bd0" (UID: "ce498ada-4f85-437f-827f-0e5b0a438bd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 02:15:19.804590 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.804570 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce498ada-4f85-437f-827f-0e5b0a438bd0-istio-token" (OuterVolumeSpecName: "istio-token") pod "ce498ada-4f85-437f-827f-0e5b0a438bd0" (UID: "ce498ada-4f85-437f-827f-0e5b0a438bd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:15:19.851775 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.851737 2561 generic.go:356] "Generic (PLEG): container finished" podID="ce498ada-4f85-437f-827f-0e5b0a438bd0" containerID="66f5871c7c32c84b548b44dfc11d2d5f51123221bc330842054a3cf3953fa036" exitCode=0 Jul 22 02:15:19.851892 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.851791 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" Jul 22 02:15:19.851892 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.851826 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" event={"ID":"ce498ada-4f85-437f-827f-0e5b0a438bd0","Type":"ContainerDied","Data":"66f5871c7c32c84b548b44dfc11d2d5f51123221bc330842054a3cf3953fa036"} Jul 22 02:15:19.851968 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.851904 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf" event={"ID":"ce498ada-4f85-437f-827f-0e5b0a438bd0","Type":"ContainerDied","Data":"6b7cf41c78339786b13e6652a7e05777cc952b31327d280d65071aea9568d9b8"} Jul 22 02:15:19.851968 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.851927 2561 scope.go:117] "RemoveContainer" containerID="66f5871c7c32c84b548b44dfc11d2d5f51123221bc330842054a3cf3953fa036" Jul 22 02:15:19.862488 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.862467 2561 scope.go:117] "RemoveContainer" containerID="be07df7a83c28af92859b3ad365d4e865deca2da1a7c3ac7c7e1ea01d887682d" Jul 22 02:15:19.871626 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.871607 2561 scope.go:117] "RemoveContainer" containerID="66f5871c7c32c84b548b44dfc11d2d5f51123221bc330842054a3cf3953fa036" Jul 22 02:15:19.871916 ip-10-0-131-56 kubenswrapper[2561]: E0722 02:15:19.871895 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66f5871c7c32c84b548b44dfc11d2d5f51123221bc330842054a3cf3953fa036\": container with ID starting with 66f5871c7c32c84b548b44dfc11d2d5f51123221bc330842054a3cf3953fa036 not found: ID does not exist" containerID="66f5871c7c32c84b548b44dfc11d2d5f51123221bc330842054a3cf3953fa036" Jul 22 02:15:19.872006 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.871923 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66f5871c7c32c84b548b44dfc11d2d5f51123221bc330842054a3cf3953fa036"} err="failed to get container status \"66f5871c7c32c84b548b44dfc11d2d5f51123221bc330842054a3cf3953fa036\": rpc error: code = NotFound desc = could not find container \"66f5871c7c32c84b548b44dfc11d2d5f51123221bc330842054a3cf3953fa036\": container with ID starting with 66f5871c7c32c84b548b44dfc11d2d5f51123221bc330842054a3cf3953fa036 not found: ID does not exist" Jul 22 02:15:19.872006 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.871940 2561 scope.go:117] "RemoveContainer" containerID="be07df7a83c28af92859b3ad365d4e865deca2da1a7c3ac7c7e1ea01d887682d" Jul 22 02:15:19.872156 ip-10-0-131-56 kubenswrapper[2561]: E0722 02:15:19.872142 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be07df7a83c28af92859b3ad365d4e865deca2da1a7c3ac7c7e1ea01d887682d\": container with ID starting with be07df7a83c28af92859b3ad365d4e865deca2da1a7c3ac7c7e1ea01d887682d not found: ID does not exist" containerID="be07df7a83c28af92859b3ad365d4e865deca2da1a7c3ac7c7e1ea01d887682d" Jul 22 02:15:19.872197 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.872161 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be07df7a83c28af92859b3ad365d4e865deca2da1a7c3ac7c7e1ea01d887682d"} err="failed to get container status \"be07df7a83c28af92859b3ad365d4e865deca2da1a7c3ac7c7e1ea01d887682d\": rpc error: code = NotFound desc = could not find container \"be07df7a83c28af92859b3ad365d4e865deca2da1a7c3ac7c7e1ea01d887682d\": container with ID starting with be07df7a83c28af92859b3ad365d4e865deca2da1a7c3ac7c7e1ea01d887682d not found: ID does not exist" Jul 22 02:15:19.876882 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.876863 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf"] Jul 22 02:15:19.882989 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.882965 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1ad649-openshift-default-5d4cb79b69-77clf"] Jul 22 02:15:19.903140 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.903091 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce498ada-4f85-437f-827f-0e5b0a438bd0-istio-podinfo\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:15:19.903140 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.903112 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce498ada-4f85-437f-827f-0e5b0a438bd0-istio-token\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:15:19.903140 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.903121 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-istio-envoy\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:15:19.903140 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.903130 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwdzd\" (UniqueName: \"kubernetes.io/projected/ce498ada-4f85-437f-827f-0e5b0a438bd0-kube-api-access-cwdzd\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:15:19.903140 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.903139 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-workload-certs\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:15:19.903344 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.903148 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-istio-data\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:15:19.903344 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.903155 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce498ada-4f85-437f-827f-0e5b0a438bd0-workload-socket\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:15:19.924934 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:19.924904 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce498ada-4f85-437f-827f-0e5b0a438bd0" path="/var/lib/kubelet/pods/ce498ada-4f85-437f-827f-0e5b0a438bd0/volumes" Jul 22 02:15:24.192593 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.192564 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r"] Jul 22 02:15:24.193067 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.193052 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce498ada-4f85-437f-827f-0e5b0a438bd0" containerName="istio-proxy" Jul 22 02:15:24.193112 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.193070 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce498ada-4f85-437f-827f-0e5b0a438bd0" containerName="istio-proxy" Jul 22 02:15:24.193112 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.193079 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce498ada-4f85-437f-827f-0e5b0a438bd0" containerName="istio-proxy" Jul 22 02:15:24.193112 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.193084 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce498ada-4f85-437f-827f-0e5b0a438bd0" containerName="istio-proxy" Jul 22 02:15:24.193112 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.193093 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83846c24-5e11-4da6-8628-3b6b62964476" containerName="cleanup" Jul 22 02:15:24.193112 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.193099 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="83846c24-5e11-4da6-8628-3b6b62964476" containerName="cleanup" Jul 22 02:15:24.193263 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.193160 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce498ada-4f85-437f-827f-0e5b0a438bd0" containerName="istio-proxy" Jul 22 02:15:24.193263 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.193170 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce498ada-4f85-437f-827f-0e5b0a438bd0" containerName="istio-proxy" Jul 22 02:15:24.193263 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.193176 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="83846c24-5e11-4da6-8628-3b6b62964476" containerName="cleanup" Jul 22 02:15:24.197764 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.197746 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.202448 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.202426 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-703b60-openshift-default-dockercfg-f8fq2\"" Jul 22 02:15:24.202565 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.202547 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 22 02:15:24.211390 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.211363 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r"] Jul 22 02:15:24.341768 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.341738 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9z5t\" (UniqueName: \"kubernetes.io/projected/467a386f-4a99-40a6-b21f-b36a445f9d2d-kube-api-access-g9z5t\") pod \"e2e-api-b-703b60-openshift-default-58654c7dd9-2426r\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.341916 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.341779 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-credential-socket\") pod \"e2e-api-b-703b60-openshift-default-58654c7dd9-2426r\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.341916 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.341808 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-workload-certs\") pod \"e2e-api-b-703b60-openshift-default-58654c7dd9-2426r\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.341916 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.341868 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-workload-socket\") pod \"e2e-api-b-703b60-openshift-default-58654c7dd9-2426r\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.341916 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.341893 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467a386f-4a99-40a6-b21f-b36a445f9d2d-istiod-ca-cert\") pod \"e2e-api-b-703b60-openshift-default-58654c7dd9-2426r\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.342086 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.341924 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-istio-data\") pod \"e2e-api-b-703b60-openshift-default-58654c7dd9-2426r\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.342086 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.341948 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467a386f-4a99-40a6-b21f-b36a445f9d2d-istio-token\") pod \"e2e-api-b-703b60-openshift-default-58654c7dd9-2426r\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.342086 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.341969 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-istio-envoy\") pod \"e2e-api-b-703b60-openshift-default-58654c7dd9-2426r\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.342086 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.342032 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467a386f-4a99-40a6-b21f-b36a445f9d2d-istio-podinfo\") pod \"e2e-api-b-703b60-openshift-default-58654c7dd9-2426r\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.442752 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.442643 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467a386f-4a99-40a6-b21f-b36a445f9d2d-istio-podinfo\") pod \"e2e-api-b-703b60-openshift-default-58654c7dd9-2426r\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.442752 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.442681 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9z5t\" (UniqueName: \"kubernetes.io/projected/467a386f-4a99-40a6-b21f-b36a445f9d2d-kube-api-access-g9z5t\") pod \"e2e-api-b-703b60-openshift-default-58654c7dd9-2426r\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.442752 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.442733 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-credential-socket\") pod \"e2e-api-b-703b60-openshift-default-58654c7dd9-2426r\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.443006 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.442849 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-workload-certs\") pod \"e2e-api-b-703b60-openshift-default-58654c7dd9-2426r\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.443006 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.442934 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-workload-socket\") pod \"e2e-api-b-703b60-openshift-default-58654c7dd9-2426r\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.443006 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.442977 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467a386f-4a99-40a6-b21f-b36a445f9d2d-istiod-ca-cert\") pod \"e2e-api-b-703b60-openshift-default-58654c7dd9-2426r\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.443181 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.443023 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-istio-data\") pod \"e2e-api-b-703b60-openshift-default-58654c7dd9-2426r\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.443181 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.443051 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-credential-socket\") pod \"e2e-api-b-703b60-openshift-default-58654c7dd9-2426r\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.443181 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.443059 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467a386f-4a99-40a6-b21f-b36a445f9d2d-istio-token\") pod \"e2e-api-b-703b60-openshift-default-58654c7dd9-2426r\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.443181 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.443104 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-istio-envoy\") pod \"e2e-api-b-703b60-openshift-default-58654c7dd9-2426r\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.443348 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.443204 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-workload-certs\") pod \"e2e-api-b-703b60-openshift-default-58654c7dd9-2426r\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.443348 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.443263 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-workload-socket\") pod \"e2e-api-b-703b60-openshift-default-58654c7dd9-2426r\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.443422 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.443372 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-istio-data\") pod \"e2e-api-b-703b60-openshift-default-58654c7dd9-2426r\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.443641 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.443619 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467a386f-4a99-40a6-b21f-b36a445f9d2d-istiod-ca-cert\") pod \"e2e-api-b-703b60-openshift-default-58654c7dd9-2426r\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.445244 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.445221 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-istio-envoy\") pod \"e2e-api-b-703b60-openshift-default-58654c7dd9-2426r\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.445369 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.445353 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467a386f-4a99-40a6-b21f-b36a445f9d2d-istio-podinfo\") pod \"e2e-api-b-703b60-openshift-default-58654c7dd9-2426r\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.452114 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.452088 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9z5t\" (UniqueName: \"kubernetes.io/projected/467a386f-4a99-40a6-b21f-b36a445f9d2d-kube-api-access-g9z5t\") pod \"e2e-api-b-703b60-openshift-default-58654c7dd9-2426r\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.452209 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.452174 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467a386f-4a99-40a6-b21f-b36a445f9d2d-istio-token\") pod \"e2e-api-b-703b60-openshift-default-58654c7dd9-2426r\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.507897 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.507869 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:24.658471 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.658444 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r"] Jul 22 02:15:24.660939 ip-10-0-131-56 kubenswrapper[2561]: W0722 02:15:24.660911 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467a386f_4a99_40a6_b21f_b36a445f9d2d.slice/crio-abb7790796b7bc482641a189d85777801e87b8988681b396aed29df983fa44e5 WatchSource:0}: Error finding container abb7790796b7bc482641a189d85777801e87b8988681b396aed29df983fa44e5: Status 404 returned error can't find the container with id abb7790796b7bc482641a189d85777801e87b8988681b396aed29df983fa44e5 Jul 22 02:15:24.663065 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.663016 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 22 02:15:24.663163 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.663117 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 22 02:15:24.663206 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.663162 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 22 02:15:24.885513 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.885478 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" event={"ID":"467a386f-4a99-40a6-b21f-b36a445f9d2d","Type":"ContainerStarted","Data":"7ddd2dda98cbf7b5d198e9aab63c897b50c7dda7ce44e724dae263547be83e02"} Jul 22 02:15:24.885671 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.885532 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" event={"ID":"467a386f-4a99-40a6-b21f-b36a445f9d2d","Type":"ContainerStarted","Data":"abb7790796b7bc482641a189d85777801e87b8988681b396aed29df983fa44e5"} Jul 22 02:15:24.911912 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:24.911853 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" podStartSLOduration=0.91183843 podStartE2EDuration="911.83843ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:15:24.908568879 +0000 UTC m=+2295.501373243" watchObservedRunningTime="2026-07-22 02:15:24.91183843 +0000 UTC m=+2295.504642794" Jul 22 02:15:25.508875 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:25.508841 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:25.513983 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:25.513953 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:25.891817 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:25.891715 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:25.892805 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:25.892783 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:39.822512 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:39.822477 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae6764-8nvp4"] Jul 22 02:15:39.827762 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:39.827740 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae6764-8nvp4" Jul 22 02:15:39.830175 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:39.830144 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cqdw8\"" Jul 22 02:15:39.843419 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:39.843389 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae6764-8nvp4"] Jul 22 02:15:39.889777 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:39.889748 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4263ea7e-c4ba-418a-aa1d-7e94fd00b4ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ae6764-8nvp4\" (UID: \"4263ea7e-c4ba-418a-aa1d-7e94fd00b4ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae6764-8nvp4" Jul 22 02:15:39.990482 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:39.990448 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4263ea7e-c4ba-418a-aa1d-7e94fd00b4ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ae6764-8nvp4\" (UID: \"4263ea7e-c4ba-418a-aa1d-7e94fd00b4ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae6764-8nvp4" Jul 22 02:15:39.991130 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:39.991111 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4263ea7e-c4ba-418a-aa1d-7e94fd00b4ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ae6764-8nvp4\" (UID: \"4263ea7e-c4ba-418a-aa1d-7e94fd00b4ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae6764-8nvp4" Jul 22 02:15:40.151777 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:40.151662 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae6764-8nvp4" Jul 22 02:15:40.323470 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:40.323441 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae6764-8nvp4"] Jul 22 02:15:40.324387 ip-10-0-131-56 kubenswrapper[2561]: W0722 02:15:40.324361 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4263ea7e_c4ba_418a_aa1d_7e94fd00b4ee.slice/crio-971f7588159f78e3f1e0c122815f3a78c6081d4c51844f6caf7063557d157724 WatchSource:0}: Error finding container 971f7588159f78e3f1e0c122815f3a78c6081d4c51844f6caf7063557d157724: Status 404 returned error can't find the container with id 971f7588159f78e3f1e0c122815f3a78c6081d4c51844f6caf7063557d157724 Jul 22 02:15:40.981674 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:40.981635 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae6764-8nvp4" event={"ID":"4263ea7e-c4ba-418a-aa1d-7e94fd00b4ee","Type":"ContainerStarted","Data":"304f2718deda577d51e724accd220b0c41ac52d1689b2396910b92ddde41cb31"} Jul 22 02:15:40.982087 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:40.981724 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae6764-8nvp4" event={"ID":"4263ea7e-c4ba-418a-aa1d-7e94fd00b4ee","Type":"ContainerStarted","Data":"971f7588159f78e3f1e0c122815f3a78c6081d4c51844f6caf7063557d157724"} Jul 22 02:15:41.002068 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:41.002020 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae6764-8nvp4" podStartSLOduration=2.002006461 podStartE2EDuration="2.002006461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:15:40.999634624 +0000 UTC m=+2311.592439014" watchObservedRunningTime="2026-07-22 02:15:41.002006461 +0000 UTC m=+2311.594810826" Jul 22 02:15:41.989512 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:41.989476 2561 generic.go:356] "Generic (PLEG): container finished" podID="4263ea7e-c4ba-418a-aa1d-7e94fd00b4ee" containerID="304f2718deda577d51e724accd220b0c41ac52d1689b2396910b92ddde41cb31" exitCode=0 Jul 22 02:15:41.989946 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:41.989574 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae6764-8nvp4" event={"ID":"4263ea7e-c4ba-418a-aa1d-7e94fd00b4ee","Type":"ContainerDied","Data":"304f2718deda577d51e724accd220b0c41ac52d1689b2396910b92ddde41cb31"} Jul 22 02:15:43.139841 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:43.139818 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae6764-8nvp4" Jul 22 02:15:43.220865 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:43.220836 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4263ea7e-c4ba-418a-aa1d-7e94fd00b4ee-maas-api-service-ca\") pod \"4263ea7e-c4ba-418a-aa1d-7e94fd00b4ee\" (UID: \"4263ea7e-c4ba-418a-aa1d-7e94fd00b4ee\") " Jul 22 02:15:43.221281 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:43.221252 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4263ea7e-c4ba-418a-aa1d-7e94fd00b4ee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4263ea7e-c4ba-418a-aa1d-7e94fd00b4ee" (UID: "4263ea7e-c4ba-418a-aa1d-7e94fd00b4ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:15:43.321476 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:43.321402 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4263ea7e-c4ba-418a-aa1d-7e94fd00b4ee-maas-api-service-ca\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:15:44.003705 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:44.003658 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae6764-8nvp4" Jul 22 02:15:44.003864 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:44.003710 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae6764-8nvp4" event={"ID":"4263ea7e-c4ba-418a-aa1d-7e94fd00b4ee","Type":"ContainerDied","Data":"971f7588159f78e3f1e0c122815f3a78c6081d4c51844f6caf7063557d157724"} Jul 22 02:15:44.003864 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:44.003757 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="971f7588159f78e3f1e0c122815f3a78c6081d4c51844f6caf7063557d157724" Jul 22 02:15:45.011567 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:45.011537 2561 generic.go:356] "Generic (PLEG): container finished" podID="467a386f-4a99-40a6-b21f-b36a445f9d2d" containerID="7ddd2dda98cbf7b5d198e9aab63c897b50c7dda7ce44e724dae263547be83e02" exitCode=0 Jul 22 02:15:45.012034 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:45.011622 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" event={"ID":"467a386f-4a99-40a6-b21f-b36a445f9d2d","Type":"ContainerDied","Data":"7ddd2dda98cbf7b5d198e9aab63c897b50c7dda7ce44e724dae263547be83e02"} Jul 22 02:15:45.012034 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:45.012004 2561 scope.go:117] "RemoveContainer" containerID="7ddd2dda98cbf7b5d198e9aab63c897b50c7dda7ce44e724dae263547be83e02" Jul 22 02:15:45.012941 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:45.012913 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 22 02:15:45.013055 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:45.012977 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 22 02:15:45.013055 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:45.013010 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 22 02:15:46.019554 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:46.019515 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" event={"ID":"467a386f-4a99-40a6-b21f-b36a445f9d2d","Type":"ContainerStarted","Data":"c279be1e778caf184def3f806886e22c3e488e5fa28aa273c1b57ecf909788e9"} Jul 22 02:15:46.508250 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:46.508213 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:46.508423 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:46.508389 2561 patch_prober.go:28] interesting pod/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Jul 22 02:15:46.508468 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:46.508438 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" podUID="467a386f-4a99-40a6-b21f-b36a445f9d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Jul 22 02:15:47.508461 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:47.508417 2561 patch_prober.go:28] interesting pod/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Jul 22 02:15:47.508954 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:47.508478 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" podUID="467a386f-4a99-40a6-b21f-b36a445f9d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Jul 22 02:15:48.508824 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:48.508789 2561 patch_prober.go:28] interesting pod/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Jul 22 02:15:48.509198 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:48.508843 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" podUID="467a386f-4a99-40a6-b21f-b36a445f9d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Jul 22 02:15:49.508361 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:49.508320 2561 patch_prober.go:28] interesting pod/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Jul 22 02:15:49.508564 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:49.508378 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" podUID="467a386f-4a99-40a6-b21f-b36a445f9d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Jul 22 02:15:50.511543 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:50.511511 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:50.511958 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:50.511928 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:15:50.512718 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:15:50.512666 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:16:00.067796 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:00.067735 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae6764-8nvp4"] Jul 22 02:16:00.070834 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:00.070812 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ae6764-8nvp4"] Jul 22 02:16:01.925256 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:01.925219 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4263ea7e-c4ba-418a-aa1d-7e94fd00b4ee" path="/var/lib/kubelet/pods/4263ea7e-c4ba-418a-aa1d-7e94fd00b4ee/volumes" Jul 22 02:16:07.348715 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:07.348648 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-703b60-r5slm"] Jul 22 02:16:07.349187 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:07.349122 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4263ea7e-c4ba-418a-aa1d-7e94fd00b4ee" containerName="revoke-keys" Jul 22 02:16:07.349187 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:07.349135 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4263ea7e-c4ba-418a-aa1d-7e94fd00b4ee" containerName="revoke-keys" Jul 22 02:16:07.349308 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:07.349210 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4263ea7e-c4ba-418a-aa1d-7e94fd00b4ee" containerName="revoke-keys" Jul 22 02:16:07.352461 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:07.352439 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-703b60-r5slm" Jul 22 02:16:07.355536 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:07.355516 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cqdw8\"" Jul 22 02:16:07.374768 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:07.374743 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-703b60-r5slm"] Jul 22 02:16:07.438959 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:07.438923 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1021afab-4ae3-4348-b053-b4d932b43b53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-703b60-r5slm\" (UID: \"1021afab-4ae3-4348-b053-b4d932b43b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-703b60-r5slm" Jul 22 02:16:07.540182 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:07.540151 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1021afab-4ae3-4348-b053-b4d932b43b53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-703b60-r5slm\" (UID: \"1021afab-4ae3-4348-b053-b4d932b43b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-703b60-r5slm" Jul 22 02:16:07.540803 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:07.540783 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1021afab-4ae3-4348-b053-b4d932b43b53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-703b60-r5slm\" (UID: \"1021afab-4ae3-4348-b053-b4d932b43b53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-703b60-r5slm" Jul 22 02:16:07.663194 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:07.663121 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-703b60-r5slm" Jul 22 02:16:07.807048 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:07.807004 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-703b60-r5slm"] Jul 22 02:16:07.810018 ip-10-0-131-56 kubenswrapper[2561]: W0722 02:16:07.809992 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1021afab_4ae3_4348_b053_b4d932b43b53.slice/crio-9da0161059608db9a9fd057bfb1b431af5bf4577d53e188437d05f1b07ef47e2 WatchSource:0}: Error finding container 9da0161059608db9a9fd057bfb1b431af5bf4577d53e188437d05f1b07ef47e2: Status 404 returned error can't find the container with id 9da0161059608db9a9fd057bfb1b431af5bf4577d53e188437d05f1b07ef47e2 Jul 22 02:16:08.159441 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:08.159413 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-703b60-r5slm" event={"ID":"1021afab-4ae3-4348-b053-b4d932b43b53","Type":"ContainerStarted","Data":"d4c19909241a59bdc1468a96ebb2705de25241f3187fb194599125411389be61"} Jul 22 02:16:08.159594 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:08.159474 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-703b60-r5slm" event={"ID":"1021afab-4ae3-4348-b053-b4d932b43b53","Type":"ContainerStarted","Data":"9da0161059608db9a9fd057bfb1b431af5bf4577d53e188437d05f1b07ef47e2"} Jul 22 02:16:08.175195 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:08.175148 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-703b60-r5slm" podStartSLOduration=1.175133659 podStartE2EDuration="1.175133659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:16:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:16:08.174690254 +0000 UTC m=+2338.767494640" watchObservedRunningTime="2026-07-22 02:16:08.175133659 +0000 UTC m=+2338.767938022" Jul 22 02:16:09.167147 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:09.167113 2561 generic.go:356] "Generic (PLEG): container finished" podID="1021afab-4ae3-4348-b053-b4d932b43b53" containerID="d4c19909241a59bdc1468a96ebb2705de25241f3187fb194599125411389be61" exitCode=0 Jul 22 02:16:09.167514 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:09.167206 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-703b60-r5slm" event={"ID":"1021afab-4ae3-4348-b053-b4d932b43b53","Type":"ContainerDied","Data":"d4c19909241a59bdc1468a96ebb2705de25241f3187fb194599125411389be61"} Jul 22 02:16:10.315095 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:10.315068 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-703b60-r5slm" Jul 22 02:16:10.465787 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:10.465751 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1021afab-4ae3-4348-b053-b4d932b43b53-maas-api-service-ca\") pod \"1021afab-4ae3-4348-b053-b4d932b43b53\" (UID: \"1021afab-4ae3-4348-b053-b4d932b43b53\") " Jul 22 02:16:10.466069 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:10.466047 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1021afab-4ae3-4348-b053-b4d932b43b53-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1021afab-4ae3-4348-b053-b4d932b43b53" (UID: "1021afab-4ae3-4348-b053-b4d932b43b53"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:16:10.466146 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:10.466124 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1021afab-4ae3-4348-b053-b4d932b43b53-maas-api-service-ca\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:16:11.179324 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:11.179285 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-703b60-r5slm" event={"ID":"1021afab-4ae3-4348-b053-b4d932b43b53","Type":"ContainerDied","Data":"9da0161059608db9a9fd057bfb1b431af5bf4577d53e188437d05f1b07ef47e2"} Jul 22 02:16:11.179610 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:11.179344 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9da0161059608db9a9fd057bfb1b431af5bf4577d53e188437d05f1b07ef47e2" Jul 22 02:16:11.179610 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:11.179299 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-703b60-r5slm" Jul 22 02:16:11.233490 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:11.233463 2561 scope.go:117] "RemoveContainer" containerID="e7799e6a7af13c6310cef07fad1151e1e49490d3a35c93535182b7929d8a9615" Jul 22 02:16:27.434274 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:27.434233 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-703b60-r5slm"] Jul 22 02:16:27.437360 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:27.437335 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-703b60-r5slm"] Jul 22 02:16:27.925432 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:27.925392 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1021afab-4ae3-4348-b053-b4d932b43b53" path="/var/lib/kubelet/pods/1021afab-4ae3-4348-b053-b4d932b43b53/volumes" Jul 22 02:16:34.246516 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:34.246482 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r"] Jul 22 02:16:34.246988 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:34.246738 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" podUID="467a386f-4a99-40a6-b21f-b36a445f9d2d" containerName="istio-proxy" containerID="cri-o://c279be1e778caf184def3f806886e22c3e488e5fa28aa273c1b57ecf909788e9" gracePeriod=30 Jul 22 02:16:35.514234 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:35.514195 2561 patch_prober.go:28] interesting pod/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 02:16:35.514604 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:35.514255 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" podUID="467a386f-4a99-40a6-b21f-b36a445f9d2d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 02:16:39.512912 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:39.512857 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:16:39.627313 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:39.627285 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-istio-envoy\") pod \"467a386f-4a99-40a6-b21f-b36a445f9d2d\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " Jul 22 02:16:39.627505 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:39.627349 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-credential-socket\") pod \"467a386f-4a99-40a6-b21f-b36a445f9d2d\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " Jul 22 02:16:39.627505 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:39.627376 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-workload-socket\") pod \"467a386f-4a99-40a6-b21f-b36a445f9d2d\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " Jul 22 02:16:39.627505 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:39.627419 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-workload-certs\") pod \"467a386f-4a99-40a6-b21f-b36a445f9d2d\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " Jul 22 02:16:39.627505 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:39.627443 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467a386f-4a99-40a6-b21f-b36a445f9d2d-istiod-ca-cert\") pod \"467a386f-4a99-40a6-b21f-b36a445f9d2d\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " Jul 22 02:16:39.627505 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:39.627499 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9z5t\" (UniqueName: \"kubernetes.io/projected/467a386f-4a99-40a6-b21f-b36a445f9d2d-kube-api-access-g9z5t\") pod \"467a386f-4a99-40a6-b21f-b36a445f9d2d\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " Jul 22 02:16:39.627944 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:39.627528 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-istio-data\") pod \"467a386f-4a99-40a6-b21f-b36a445f9d2d\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " Jul 22 02:16:39.627944 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:39.627597 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467a386f-4a99-40a6-b21f-b36a445f9d2d-istio-podinfo\") pod \"467a386f-4a99-40a6-b21f-b36a445f9d2d\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " Jul 22 02:16:39.627944 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:39.627625 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467a386f-4a99-40a6-b21f-b36a445f9d2d-istio-token\") pod \"467a386f-4a99-40a6-b21f-b36a445f9d2d\" (UID: \"467a386f-4a99-40a6-b21f-b36a445f9d2d\") " Jul 22 02:16:39.627944 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:39.627655 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "467a386f-4a99-40a6-b21f-b36a445f9d2d" (UID: "467a386f-4a99-40a6-b21f-b36a445f9d2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:16:39.627944 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:39.627669 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "467a386f-4a99-40a6-b21f-b36a445f9d2d" (UID: "467a386f-4a99-40a6-b21f-b36a445f9d2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:16:39.627944 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:39.627686 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "467a386f-4a99-40a6-b21f-b36a445f9d2d" (UID: "467a386f-4a99-40a6-b21f-b36a445f9d2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:16:39.628300 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:39.627996 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-credential-socket\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:16:39.628300 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:39.628018 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-workload-socket\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:16:39.628300 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:39.628032 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-workload-certs\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:16:39.628300 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:39.628204 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/467a386f-4a99-40a6-b21f-b36a445f9d2d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "467a386f-4a99-40a6-b21f-b36a445f9d2d" (UID: "467a386f-4a99-40a6-b21f-b36a445f9d2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:16:39.628707 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:39.628607 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-istio-data" (OuterVolumeSpecName: "istio-data") pod "467a386f-4a99-40a6-b21f-b36a445f9d2d" (UID: "467a386f-4a99-40a6-b21f-b36a445f9d2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:16:39.630408 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:39.630376 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "467a386f-4a99-40a6-b21f-b36a445f9d2d" (UID: "467a386f-4a99-40a6-b21f-b36a445f9d2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:16:39.630510 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:39.630435 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467a386f-4a99-40a6-b21f-b36a445f9d2d-kube-api-access-g9z5t" (OuterVolumeSpecName: "kube-api-access-g9z5t") pod "467a386f-4a99-40a6-b21f-b36a445f9d2d" (UID: "467a386f-4a99-40a6-b21f-b36a445f9d2d"). InnerVolumeSpecName "kube-api-access-g9z5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:16:39.630510 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:39.630453 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467a386f-4a99-40a6-b21f-b36a445f9d2d-istio-token" (OuterVolumeSpecName: "istio-token") pod "467a386f-4a99-40a6-b21f-b36a445f9d2d" (UID: "467a386f-4a99-40a6-b21f-b36a445f9d2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:16:39.630596 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:39.630565 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/467a386f-4a99-40a6-b21f-b36a445f9d2d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "467a386f-4a99-40a6-b21f-b36a445f9d2d" (UID: "467a386f-4a99-40a6-b21f-b36a445f9d2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 02:16:39.729124 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:39.729090 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-istio-envoy\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:16:39.729124 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:39.729125 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467a386f-4a99-40a6-b21f-b36a445f9d2d-istiod-ca-cert\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:16:39.729292 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:39.729140 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9z5t\" (UniqueName: \"kubernetes.io/projected/467a386f-4a99-40a6-b21f-b36a445f9d2d-kube-api-access-g9z5t\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:16:39.729292 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:39.729153 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467a386f-4a99-40a6-b21f-b36a445f9d2d-istio-data\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:16:39.729292 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:39.729166 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467a386f-4a99-40a6-b21f-b36a445f9d2d-istio-podinfo\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:16:39.729292 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:39.729177 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467a386f-4a99-40a6-b21f-b36a445f9d2d-istio-token\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:16:40.353676 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:40.353591 2561 generic.go:356] "Generic (PLEG): container finished" podID="467a386f-4a99-40a6-b21f-b36a445f9d2d" containerID="c279be1e778caf184def3f806886e22c3e488e5fa28aa273c1b57ecf909788e9" exitCode=0 Jul 22 02:16:40.353676 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:40.353639 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" Jul 22 02:16:40.353676 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:40.353667 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" event={"ID":"467a386f-4a99-40a6-b21f-b36a445f9d2d","Type":"ContainerDied","Data":"c279be1e778caf184def3f806886e22c3e488e5fa28aa273c1b57ecf909788e9"} Jul 22 02:16:40.353923 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:40.353751 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r" event={"ID":"467a386f-4a99-40a6-b21f-b36a445f9d2d","Type":"ContainerDied","Data":"abb7790796b7bc482641a189d85777801e87b8988681b396aed29df983fa44e5"} Jul 22 02:16:40.353923 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:40.353783 2561 scope.go:117] "RemoveContainer" containerID="c279be1e778caf184def3f806886e22c3e488e5fa28aa273c1b57ecf909788e9" Jul 22 02:16:40.363469 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:40.363448 2561 scope.go:117] "RemoveContainer" containerID="7ddd2dda98cbf7b5d198e9aab63c897b50c7dda7ce44e724dae263547be83e02" Jul 22 02:16:40.372370 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:40.372193 2561 scope.go:117] "RemoveContainer" containerID="c279be1e778caf184def3f806886e22c3e488e5fa28aa273c1b57ecf909788e9" Jul 22 02:16:40.372494 ip-10-0-131-56 kubenswrapper[2561]: E0722 02:16:40.372475 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c279be1e778caf184def3f806886e22c3e488e5fa28aa273c1b57ecf909788e9\": container with ID starting with c279be1e778caf184def3f806886e22c3e488e5fa28aa273c1b57ecf909788e9 not found: ID does not exist" containerID="c279be1e778caf184def3f806886e22c3e488e5fa28aa273c1b57ecf909788e9" Jul 22 02:16:40.372544 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:40.372502 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c279be1e778caf184def3f806886e22c3e488e5fa28aa273c1b57ecf909788e9"} err="failed to get container status \"c279be1e778caf184def3f806886e22c3e488e5fa28aa273c1b57ecf909788e9\": rpc error: code = NotFound desc = could not find container \"c279be1e778caf184def3f806886e22c3e488e5fa28aa273c1b57ecf909788e9\": container with ID starting with c279be1e778caf184def3f806886e22c3e488e5fa28aa273c1b57ecf909788e9 not found: ID does not exist" Jul 22 02:16:40.372544 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:40.372517 2561 scope.go:117] "RemoveContainer" containerID="7ddd2dda98cbf7b5d198e9aab63c897b50c7dda7ce44e724dae263547be83e02" Jul 22 02:16:40.372785 ip-10-0-131-56 kubenswrapper[2561]: E0722 02:16:40.372767 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ddd2dda98cbf7b5d198e9aab63c897b50c7dda7ce44e724dae263547be83e02\": container with ID starting with 7ddd2dda98cbf7b5d198e9aab63c897b50c7dda7ce44e724dae263547be83e02 not found: ID does not exist" containerID="7ddd2dda98cbf7b5d198e9aab63c897b50c7dda7ce44e724dae263547be83e02" Jul 22 02:16:40.372837 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:40.372793 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ddd2dda98cbf7b5d198e9aab63c897b50c7dda7ce44e724dae263547be83e02"} err="failed to get container status \"7ddd2dda98cbf7b5d198e9aab63c897b50c7dda7ce44e724dae263547be83e02\": rpc error: code = NotFound desc = could not find container \"7ddd2dda98cbf7b5d198e9aab63c897b50c7dda7ce44e724dae263547be83e02\": container with ID starting with 7ddd2dda98cbf7b5d198e9aab63c897b50c7dda7ce44e724dae263547be83e02 not found: ID does not exist" Jul 22 02:16:40.380855 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:40.380826 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r"] Jul 22 02:16:40.384680 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:40.384658 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-703b60-openshift-default-58654c7dd9-2426r"] Jul 22 02:16:41.925043 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:41.925008 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="467a386f-4a99-40a6-b21f-b36a445f9d2d" path="/var/lib/kubelet/pods/467a386f-4a99-40a6-b21f-b36a445f9d2d/volumes" Jul 22 02:16:47.445913 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:47.445881 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2"] Jul 22 02:16:47.446341 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:47.446329 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="467a386f-4a99-40a6-b21f-b36a445f9d2d" containerName="istio-proxy" Jul 22 02:16:47.446387 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:47.446343 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="467a386f-4a99-40a6-b21f-b36a445f9d2d" containerName="istio-proxy" Jul 22 02:16:47.446387 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:47.446352 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="467a386f-4a99-40a6-b21f-b36a445f9d2d" containerName="istio-proxy" Jul 22 02:16:47.446387 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:47.446358 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="467a386f-4a99-40a6-b21f-b36a445f9d2d" containerName="istio-proxy" Jul 22 02:16:47.446387 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:47.446381 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1021afab-4ae3-4348-b053-b4d932b43b53" containerName="revoke-keys" Jul 22 02:16:47.446387 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:47.446386 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1021afab-4ae3-4348-b053-b4d932b43b53" containerName="revoke-keys" Jul 22 02:16:47.446546 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:47.446447 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="467a386f-4a99-40a6-b21f-b36a445f9d2d" containerName="istio-proxy" Jul 22 02:16:47.446546 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:47.446456 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="467a386f-4a99-40a6-b21f-b36a445f9d2d" containerName="istio-proxy" Jul 22 02:16:47.446546 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:47.446466 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1021afab-4ae3-4348-b053-b4d932b43b53" containerName="revoke-keys" Jul 22 02:16:47.449478 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:47.449460 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2" Jul 22 02:16:47.452748 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:47.452681 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-c6960c-serving-cert\"" Jul 22 02:16:47.476271 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:47.476247 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2"] Jul 22 02:16:47.593567 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:47.593534 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h54m7\" (UniqueName: \"kubernetes.io/projected/a1debdc1-148e-4aee-bcab-791cdc9c6001-kube-api-access-h54m7\") pod \"maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2\" (UID: \"a1debdc1-148e-4aee-bcab-791cdc9c6001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2" Jul 22 02:16:47.593743 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:47.593587 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1debdc1-148e-4aee-bcab-791cdc9c6001-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2\" (UID: \"a1debdc1-148e-4aee-bcab-791cdc9c6001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2" Jul 22 02:16:47.593743 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:47.593635 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1debdc1-148e-4aee-bcab-791cdc9c6001-maas-api-tls\") pod \"maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2\" (UID: \"a1debdc1-148e-4aee-bcab-791cdc9c6001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2" Jul 22 02:16:47.695144 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:47.695109 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1debdc1-148e-4aee-bcab-791cdc9c6001-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2\" (UID: \"a1debdc1-148e-4aee-bcab-791cdc9c6001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2" Jul 22 02:16:47.695313 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:47.695158 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1debdc1-148e-4aee-bcab-791cdc9c6001-maas-api-tls\") pod \"maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2\" (UID: \"a1debdc1-148e-4aee-bcab-791cdc9c6001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2" Jul 22 02:16:47.695313 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:47.695242 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h54m7\" (UniqueName: \"kubernetes.io/projected/a1debdc1-148e-4aee-bcab-791cdc9c6001-kube-api-access-h54m7\") pod \"maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2\" (UID: \"a1debdc1-148e-4aee-bcab-791cdc9c6001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2" Jul 22 02:16:47.695792 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:47.695767 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1debdc1-148e-4aee-bcab-791cdc9c6001-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2\" (UID: \"a1debdc1-148e-4aee-bcab-791cdc9c6001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2" Jul 22 02:16:47.697709 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:47.697641 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1debdc1-148e-4aee-bcab-791cdc9c6001-maas-api-tls\") pod \"maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2\" (UID: \"a1debdc1-148e-4aee-bcab-791cdc9c6001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2" Jul 22 02:16:47.704531 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:47.704512 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h54m7\" (UniqueName: \"kubernetes.io/projected/a1debdc1-148e-4aee-bcab-791cdc9c6001-kube-api-access-h54m7\") pod \"maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2\" (UID: \"a1debdc1-148e-4aee-bcab-791cdc9c6001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2" Jul 22 02:16:47.762752 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:47.762723 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2" Jul 22 02:16:47.909023 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:47.909000 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2"] Jul 22 02:16:47.910178 ip-10-0-131-56 kubenswrapper[2561]: W0722 02:16:47.910154 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1debdc1_148e_4aee_bcab_791cdc9c6001.slice/crio-ed3f3fdbe13796f62afd982093da9516504413ce0aa24474880fa85caab539b6 WatchSource:0}: Error finding container ed3f3fdbe13796f62afd982093da9516504413ce0aa24474880fa85caab539b6: Status 404 returned error can't find the container with id ed3f3fdbe13796f62afd982093da9516504413ce0aa24474880fa85caab539b6 Jul 22 02:16:47.911608 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:47.911592 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 02:16:48.413562 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:48.413521 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2" event={"ID":"a1debdc1-148e-4aee-bcab-791cdc9c6001","Type":"ContainerStarted","Data":"ed3f3fdbe13796f62afd982093da9516504413ce0aa24474880fa85caab539b6"} Jul 22 02:16:52.447156 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:52.447116 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2" event={"ID":"a1debdc1-148e-4aee-bcab-791cdc9c6001","Type":"ContainerStarted","Data":"7973d764e9de006cef1e4928c979c8a7ae4e49d9fc3b66334fc50af567c9e5b1"} Jul 22 02:16:52.447630 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:52.447254 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2" Jul 22 02:16:52.468791 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:52.468745 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2" podStartSLOduration=1.63745296 podStartE2EDuration="5.468731349s" totalImagesPullingTime="3.831278389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:16:52.467013898 +0000 UTC m=+2383.059818272" watchObservedRunningTime="2026-07-22 02:16:52.468731349 +0000 UTC m=+2383.061535709" Jul 22 02:16:58.460123 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:16:58.460091 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2" Jul 22 02:17:11.262891 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:17:11.262858 2561 scope.go:117] "RemoveContainer" containerID="e71213406191e93fbdf401a376c07690f5f72b06a8387d7bdb9f751de252e532" Jul 22 02:21:11.314849 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:21:11.314818 2561 scope.go:117] "RemoveContainer" containerID="1a20049a264df3a4a82694eeeba56908c9a65b12be6391f00e5fdd17d680182d" Jul 22 02:21:11.323887 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:21:11.323866 2561 scope.go:117] "RemoveContainer" containerID="6fa5c994ac72f619d6bc292b6db80de88c72567a2ef80ba9f76af9e598125806" Jul 22 02:22:11.349895 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:11.349822 2561 scope.go:117] "RemoveContainer" containerID="d4c19909241a59bdc1468a96ebb2705de25241f3187fb194599125411389be61" Jul 22 02:22:11.358857 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:11.358837 2561 scope.go:117] "RemoveContainer" containerID="304f2718deda577d51e724accd220b0c41ac52d1689b2396910b92ddde41cb31" Jul 22 02:22:34.276770 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:34.276734 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7697-139835625815856"] Jul 22 02:22:34.280659 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:34.280642 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-139835625815856" Jul 22 02:22:34.283206 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:34.283175 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jp4bj\"" Jul 22 02:22:34.285947 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:34.285921 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-139835625815856"] Jul 22 02:22:34.334112 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:34.334085 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5hjg\" (UniqueName: \"kubernetes.io/projected/99d8e054-03a7-425b-a5ec-a941084a93a6-kube-api-access-x5hjg\") pod \"test-curl-7697-139835625815856\" (UID: \"99d8e054-03a7-425b-a5ec-a941084a93a6\") " pod="opendatahub/test-curl-7697-139835625815856" Jul 22 02:22:34.435066 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:34.435040 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5hjg\" (UniqueName: \"kubernetes.io/projected/99d8e054-03a7-425b-a5ec-a941084a93a6-kube-api-access-x5hjg\") pod \"test-curl-7697-139835625815856\" (UID: \"99d8e054-03a7-425b-a5ec-a941084a93a6\") " pod="opendatahub/test-curl-7697-139835625815856" Jul 22 02:22:34.443380 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:34.443357 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5hjg\" (UniqueName: \"kubernetes.io/projected/99d8e054-03a7-425b-a5ec-a941084a93a6-kube-api-access-x5hjg\") pod \"test-curl-7697-139835625815856\" (UID: \"99d8e054-03a7-425b-a5ec-a941084a93a6\") " pod="opendatahub/test-curl-7697-139835625815856" Jul 22 02:22:34.592088 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:34.592010 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-139835625815856" Jul 22 02:22:34.728839 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:34.728806 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-139835625815856"] Jul 22 02:22:34.730717 ip-10-0-131-56 kubenswrapper[2561]: W0722 02:22:34.730665 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99d8e054_03a7_425b_a5ec_a941084a93a6.slice/crio-a212ac3eae966564d31a80ab85b2225d13bdaa3a7b652af24e607bd4e653b59f WatchSource:0}: Error finding container a212ac3eae966564d31a80ab85b2225d13bdaa3a7b652af24e607bd4e653b59f: Status 404 returned error can't find the container with id a212ac3eae966564d31a80ab85b2225d13bdaa3a7b652af24e607bd4e653b59f Jul 22 02:22:34.732414 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:34.732394 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 02:22:35.479368 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:35.479345 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-139835625815856" event={"ID":"99d8e054-03a7-425b-a5ec-a941084a93a6","Type":"ContainerStarted","Data":"a212ac3eae966564d31a80ab85b2225d13bdaa3a7b652af24e607bd4e653b59f"} Jul 22 02:22:36.486978 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:36.486945 2561 generic.go:356] "Generic (PLEG): container finished" podID="99d8e054-03a7-425b-a5ec-a941084a93a6" containerID="53ba98fc5446591bbf970daf1b6a62c5dd517e18baf54f42a68c1255ab2dcc63" exitCode=0 Jul 22 02:22:36.487349 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:36.487027 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-139835625815856" event={"ID":"99d8e054-03a7-425b-a5ec-a941084a93a6","Type":"ContainerDied","Data":"53ba98fc5446591bbf970daf1b6a62c5dd517e18baf54f42a68c1255ab2dcc63"} Jul 22 02:22:37.629483 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:37.629458 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-139835625815856" Jul 22 02:22:37.666127 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:37.666090 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7697-139835625815856_99d8e054-03a7-425b-a5ec-a941084a93a6/test-curl-7697-139835625815856/0.log" Jul 22 02:22:37.691085 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:37.691052 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7697-139835625815856"] Jul 22 02:22:37.699420 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:37.699399 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7697-139835625815856"] Jul 22 02:22:37.764829 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:37.764756 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5hjg\" (UniqueName: \"kubernetes.io/projected/99d8e054-03a7-425b-a5ec-a941084a93a6-kube-api-access-x5hjg\") pod \"99d8e054-03a7-425b-a5ec-a941084a93a6\" (UID: \"99d8e054-03a7-425b-a5ec-a941084a93a6\") " Jul 22 02:22:37.767328 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:37.767300 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d8e054-03a7-425b-a5ec-a941084a93a6-kube-api-access-x5hjg" (OuterVolumeSpecName: "kube-api-access-x5hjg") pod "99d8e054-03a7-425b-a5ec-a941084a93a6" (UID: "99d8e054-03a7-425b-a5ec-a941084a93a6"). InnerVolumeSpecName "kube-api-access-x5hjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:22:37.838457 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:37.838430 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7697-139835625815344"] Jul 22 02:22:37.838911 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:37.838897 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99d8e054-03a7-425b-a5ec-a941084a93a6" containerName="test-curl-7697-139835625815856" Jul 22 02:22:37.838962 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:37.838913 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="99d8e054-03a7-425b-a5ec-a941084a93a6" containerName="test-curl-7697-139835625815856" Jul 22 02:22:37.839015 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:37.839005 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="99d8e054-03a7-425b-a5ec-a941084a93a6" containerName="test-curl-7697-139835625815856" Jul 22 02:22:37.842163 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:37.842148 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-139835625815344" Jul 22 02:22:37.847435 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:37.847407 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-139835625815344"] Jul 22 02:22:37.866075 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:37.866049 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5hjg\" (UniqueName: \"kubernetes.io/projected/99d8e054-03a7-425b-a5ec-a941084a93a6-kube-api-access-x5hjg\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:22:37.925660 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:37.925636 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99d8e054-03a7-425b-a5ec-a941084a93a6" path="/var/lib/kubelet/pods/99d8e054-03a7-425b-a5ec-a941084a93a6/volumes" Jul 22 02:22:37.967145 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:37.967118 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6r9n\" (UniqueName: \"kubernetes.io/projected/3c859785-fa87-4ee8-93c6-5283b13f5d17-kube-api-access-s6r9n\") pod \"test-curl-7697-139835625815344\" (UID: \"3c859785-fa87-4ee8-93c6-5283b13f5d17\") " pod="opendatahub/test-curl-7697-139835625815344" Jul 22 02:22:38.068441 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:38.068384 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6r9n\" (UniqueName: \"kubernetes.io/projected/3c859785-fa87-4ee8-93c6-5283b13f5d17-kube-api-access-s6r9n\") pod \"test-curl-7697-139835625815344\" (UID: \"3c859785-fa87-4ee8-93c6-5283b13f5d17\") " pod="opendatahub/test-curl-7697-139835625815344" Jul 22 02:22:38.077105 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:38.077081 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6r9n\" (UniqueName: \"kubernetes.io/projected/3c859785-fa87-4ee8-93c6-5283b13f5d17-kube-api-access-s6r9n\") pod \"test-curl-7697-139835625815344\" (UID: \"3c859785-fa87-4ee8-93c6-5283b13f5d17\") " pod="opendatahub/test-curl-7697-139835625815344" Jul 22 02:22:38.152619 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:38.152593 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-139835625815344" Jul 22 02:22:38.299360 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:38.299323 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-139835625815344"] Jul 22 02:22:38.300498 ip-10-0-131-56 kubenswrapper[2561]: W0722 02:22:38.300471 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c859785_fa87_4ee8_93c6_5283b13f5d17.slice/crio-0aec88b86c135c8abcf51a3197af67f4b6f8b65d8b657a0e3a47851d6bdd40da WatchSource:0}: Error finding container 0aec88b86c135c8abcf51a3197af67f4b6f8b65d8b657a0e3a47851d6bdd40da: Status 404 returned error can't find the container with id 0aec88b86c135c8abcf51a3197af67f4b6f8b65d8b657a0e3a47851d6bdd40da Jul 22 02:22:38.499505 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:38.499476 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-139835625815856" Jul 22 02:22:38.499716 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:38.499517 2561 scope.go:117] "RemoveContainer" containerID="53ba98fc5446591bbf970daf1b6a62c5dd517e18baf54f42a68c1255ab2dcc63" Jul 22 02:22:38.500920 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:38.500827 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-139835625815344" event={"ID":"3c859785-fa87-4ee8-93c6-5283b13f5d17","Type":"ContainerStarted","Data":"0aec88b86c135c8abcf51a3197af67f4b6f8b65d8b657a0e3a47851d6bdd40da"} Jul 22 02:22:39.508948 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:39.508909 2561 generic.go:356] "Generic (PLEG): container finished" podID="3c859785-fa87-4ee8-93c6-5283b13f5d17" containerID="0b73229043ad5a47d7eccdc63d862d3b06513ecbe1359aa5e63eeff93d24eb1b" exitCode=0 Jul 22 02:22:39.509344 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:39.509000 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-139835625815344" event={"ID":"3c859785-fa87-4ee8-93c6-5283b13f5d17","Type":"ContainerDied","Data":"0b73229043ad5a47d7eccdc63d862d3b06513ecbe1359aa5e63eeff93d24eb1b"} Jul 22 02:22:40.657565 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:40.657537 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-139835625815344" Jul 22 02:22:40.676568 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:40.676544 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7697-139835625815344_3c859785-fa87-4ee8-93c6-5283b13f5d17/test-curl-7697-139835625815344/0.log" Jul 22 02:22:40.716242 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:40.716213 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7697-139835625815344"] Jul 22 02:22:40.718732 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:40.718711 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7697-139835625815344"] Jul 22 02:22:40.791261 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:40.791200 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6r9n\" (UniqueName: \"kubernetes.io/projected/3c859785-fa87-4ee8-93c6-5283b13f5d17-kube-api-access-s6r9n\") pod \"3c859785-fa87-4ee8-93c6-5283b13f5d17\" (UID: \"3c859785-fa87-4ee8-93c6-5283b13f5d17\") " Jul 22 02:22:40.793394 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:40.793373 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c859785-fa87-4ee8-93c6-5283b13f5d17-kube-api-access-s6r9n" (OuterVolumeSpecName: "kube-api-access-s6r9n") pod "3c859785-fa87-4ee8-93c6-5283b13f5d17" (UID: "3c859785-fa87-4ee8-93c6-5283b13f5d17"). InnerVolumeSpecName "kube-api-access-s6r9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:22:40.892658 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:40.892631 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6r9n\" (UniqueName: \"kubernetes.io/projected/3c859785-fa87-4ee8-93c6-5283b13f5d17-kube-api-access-s6r9n\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:22:40.928989 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:40.928959 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7697-139835625649184"] Jul 22 02:22:40.929655 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:40.929633 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c859785-fa87-4ee8-93c6-5283b13f5d17" containerName="test-curl-7697-139835625815344" Jul 22 02:22:40.929655 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:40.929660 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c859785-fa87-4ee8-93c6-5283b13f5d17" containerName="test-curl-7697-139835625815344" Jul 22 02:22:40.929805 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:40.929791 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c859785-fa87-4ee8-93c6-5283b13f5d17" containerName="test-curl-7697-139835625815344" Jul 22 02:22:40.933332 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:40.933314 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-139835625649184" Jul 22 02:22:40.939255 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:40.939223 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-139835625649184"] Jul 22 02:22:40.993256 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:40.993228 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwzx2\" (UniqueName: \"kubernetes.io/projected/493c2c41-cb94-4746-8cef-7d67dc868662-kube-api-access-cwzx2\") pod \"test-curl-7697-139835625649184\" (UID: \"493c2c41-cb94-4746-8cef-7d67dc868662\") " pod="opendatahub/test-curl-7697-139835625649184" Jul 22 02:22:41.094236 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:41.094163 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwzx2\" (UniqueName: \"kubernetes.io/projected/493c2c41-cb94-4746-8cef-7d67dc868662-kube-api-access-cwzx2\") pod \"test-curl-7697-139835625649184\" (UID: \"493c2c41-cb94-4746-8cef-7d67dc868662\") " pod="opendatahub/test-curl-7697-139835625649184" Jul 22 02:22:41.104539 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:41.104516 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwzx2\" (UniqueName: \"kubernetes.io/projected/493c2c41-cb94-4746-8cef-7d67dc868662-kube-api-access-cwzx2\") pod \"test-curl-7697-139835625649184\" (UID: \"493c2c41-cb94-4746-8cef-7d67dc868662\") " pod="opendatahub/test-curl-7697-139835625649184" Jul 22 02:22:41.245224 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:41.245195 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-139835625649184" Jul 22 02:22:41.404415 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:41.404374 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-139835625649184"] Jul 22 02:22:41.405817 ip-10-0-131-56 kubenswrapper[2561]: W0722 02:22:41.405790 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod493c2c41_cb94_4746_8cef_7d67dc868662.slice/crio-f2a4e9f1af24c0781e76b096792874bbc64bf5e07dda30e7536c8e1ae326aa6c WatchSource:0}: Error finding container f2a4e9f1af24c0781e76b096792874bbc64bf5e07dda30e7536c8e1ae326aa6c: Status 404 returned error can't find the container with id f2a4e9f1af24c0781e76b096792874bbc64bf5e07dda30e7536c8e1ae326aa6c Jul 22 02:22:41.522071 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:41.522049 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-139835625815344" Jul 22 02:22:41.522164 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:41.522080 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0aec88b86c135c8abcf51a3197af67f4b6f8b65d8b657a0e3a47851d6bdd40da" Jul 22 02:22:41.523327 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:41.523300 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-139835625649184" event={"ID":"493c2c41-cb94-4746-8cef-7d67dc868662","Type":"ContainerStarted","Data":"f2a4e9f1af24c0781e76b096792874bbc64bf5e07dda30e7536c8e1ae326aa6c"} Jul 22 02:22:41.533033 ip-10-0-131-56 kubenswrapper[2561]: E0722 02:22:41.533008 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7697-139835625815344\" is forbidden: User \"system:node:ip-10-0-131-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-56.ec2.internal' and this object" podUID="3c859785-fa87-4ee8-93c6-5283b13f5d17" pod="opendatahub/test-curl-7697-139835625815344" Jul 22 02:22:41.925470 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:41.925439 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c859785-fa87-4ee8-93c6-5283b13f5d17" path="/var/lib/kubelet/pods/3c859785-fa87-4ee8-93c6-5283b13f5d17/volumes" Jul 22 02:22:42.530143 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:42.530105 2561 generic.go:356] "Generic (PLEG): container finished" podID="493c2c41-cb94-4746-8cef-7d67dc868662" containerID="7217d90e2639c05656c51bee5a670a7d0079672b3d4d3bcfdabf41a0ca07e3d4" exitCode=0 Jul 22 02:22:42.530317 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:42.530167 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-139835625649184" event={"ID":"493c2c41-cb94-4746-8cef-7d67dc868662","Type":"ContainerDied","Data":"7217d90e2639c05656c51bee5a670a7d0079672b3d4d3bcfdabf41a0ca07e3d4"} Jul 22 02:22:43.680974 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:43.680949 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-139835625649184" Jul 22 02:22:43.703215 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:43.703192 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7697-139835625649184_493c2c41-cb94-4746-8cef-7d67dc868662/test-curl-7697-139835625649184/0.log" Jul 22 02:22:43.731623 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:43.731598 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7697-139835625649184"] Jul 22 02:22:43.736898 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:43.736866 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7697-139835625649184"] Jul 22 02:22:43.821910 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:43.821852 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwzx2\" (UniqueName: \"kubernetes.io/projected/493c2c41-cb94-4746-8cef-7d67dc868662-kube-api-access-cwzx2\") pod \"493c2c41-cb94-4746-8cef-7d67dc868662\" (UID: \"493c2c41-cb94-4746-8cef-7d67dc868662\") " Jul 22 02:22:43.824029 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:43.824003 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493c2c41-cb94-4746-8cef-7d67dc868662-kube-api-access-cwzx2" (OuterVolumeSpecName: "kube-api-access-cwzx2") pod "493c2c41-cb94-4746-8cef-7d67dc868662" (UID: "493c2c41-cb94-4746-8cef-7d67dc868662"). InnerVolumeSpecName "kube-api-access-cwzx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:22:43.890620 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:43.890596 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7697-139835625649184"] Jul 22 02:22:43.891061 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:43.891047 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="493c2c41-cb94-4746-8cef-7d67dc868662" containerName="test-curl-7697-139835625649184" Jul 22 02:22:43.891103 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:43.891063 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="493c2c41-cb94-4746-8cef-7d67dc868662" containerName="test-curl-7697-139835625649184" Jul 22 02:22:43.891149 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:43.891139 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="493c2c41-cb94-4746-8cef-7d67dc868662" containerName="test-curl-7697-139835625649184" Jul 22 02:22:43.894399 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:43.894385 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-139835625649184" Jul 22 02:22:43.900871 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:43.900839 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-139835625649184"] Jul 22 02:22:43.922720 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:43.922682 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwzx2\" (UniqueName: \"kubernetes.io/projected/493c2c41-cb94-4746-8cef-7d67dc868662-kube-api-access-cwzx2\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:22:43.925970 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:43.925948 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="493c2c41-cb94-4746-8cef-7d67dc868662" path="/var/lib/kubelet/pods/493c2c41-cb94-4746-8cef-7d67dc868662/volumes" Jul 22 02:22:44.023214 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:44.023185 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tthrn\" (UniqueName: \"kubernetes.io/projected/d5722054-bf32-4223-a89a-ea9385d07bac-kube-api-access-tthrn\") pod \"test-curl-7697-139835625649184\" (UID: \"d5722054-bf32-4223-a89a-ea9385d07bac\") " pod="opendatahub/test-curl-7697-139835625649184" Jul 22 02:22:44.123768 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:44.123711 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tthrn\" (UniqueName: \"kubernetes.io/projected/d5722054-bf32-4223-a89a-ea9385d07bac-kube-api-access-tthrn\") pod \"test-curl-7697-139835625649184\" (UID: \"d5722054-bf32-4223-a89a-ea9385d07bac\") " pod="opendatahub/test-curl-7697-139835625649184" Jul 22 02:22:44.135397 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:44.135365 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tthrn\" (UniqueName: \"kubernetes.io/projected/d5722054-bf32-4223-a89a-ea9385d07bac-kube-api-access-tthrn\") pod \"test-curl-7697-139835625649184\" (UID: \"d5722054-bf32-4223-a89a-ea9385d07bac\") " pod="opendatahub/test-curl-7697-139835625649184" Jul 22 02:22:44.204852 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:44.204827 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-139835625649184" Jul 22 02:22:44.341792 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:44.341756 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-139835625649184"] Jul 22 02:22:44.341996 ip-10-0-131-56 kubenswrapper[2561]: W0722 02:22:44.341971 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5722054_bf32_4223_a89a_ea9385d07bac.slice/crio-e5c84e01f074e6d01bd484340620ca1c2163ac64b13ac5dffe944bfb3d9e9ebe WatchSource:0}: Error finding container e5c84e01f074e6d01bd484340620ca1c2163ac64b13ac5dffe944bfb3d9e9ebe: Status 404 returned error can't find the container with id e5c84e01f074e6d01bd484340620ca1c2163ac64b13ac5dffe944bfb3d9e9ebe Jul 22 02:22:44.545609 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:44.545576 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-139835625649184" event={"ID":"d5722054-bf32-4223-a89a-ea9385d07bac","Type":"ContainerStarted","Data":"e5c84e01f074e6d01bd484340620ca1c2163ac64b13ac5dffe944bfb3d9e9ebe"} Jul 22 02:22:44.546835 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:44.546815 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-139835625649184" Jul 22 02:22:44.546926 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:44.546824 2561 scope.go:117] "RemoveContainer" containerID="7217d90e2639c05656c51bee5a670a7d0079672b3d4d3bcfdabf41a0ca07e3d4" Jul 22 02:22:45.554434 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:45.554403 2561 generic.go:356] "Generic (PLEG): container finished" podID="d5722054-bf32-4223-a89a-ea9385d07bac" containerID="e6a2ca972efa70cb66e71dd8ac5aafc36b8a6d1ae9d99eb88dcf1e7dc1ccc1f0" exitCode=0 Jul 22 02:22:45.554927 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:45.554491 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-139835625649184" event={"ID":"d5722054-bf32-4223-a89a-ea9385d07bac","Type":"ContainerDied","Data":"e6a2ca972efa70cb66e71dd8ac5aafc36b8a6d1ae9d99eb88dcf1e7dc1ccc1f0"} Jul 22 02:22:46.700397 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:46.700373 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-139835625649184" Jul 22 02:22:46.720543 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:46.720517 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7697-139835625649184_d5722054-bf32-4223-a89a-ea9385d07bac/test-curl-7697-139835625649184/0.log" Jul 22 02:22:46.745369 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:46.745345 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7697-139835625649184"] Jul 22 02:22:46.750289 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:46.750261 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7697-139835625649184"] Jul 22 02:22:46.848979 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:46.848919 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tthrn\" (UniqueName: \"kubernetes.io/projected/d5722054-bf32-4223-a89a-ea9385d07bac-kube-api-access-tthrn\") pod \"d5722054-bf32-4223-a89a-ea9385d07bac\" (UID: \"d5722054-bf32-4223-a89a-ea9385d07bac\") " Jul 22 02:22:46.851151 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:46.851130 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5722054-bf32-4223-a89a-ea9385d07bac-kube-api-access-tthrn" (OuterVolumeSpecName: "kube-api-access-tthrn") pod "d5722054-bf32-4223-a89a-ea9385d07bac" (UID: "d5722054-bf32-4223-a89a-ea9385d07bac"). InnerVolumeSpecName "kube-api-access-tthrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:22:46.895242 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:46.895210 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7697-139835625650192"] Jul 22 02:22:46.895664 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:46.895651 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5722054-bf32-4223-a89a-ea9385d07bac" containerName="test-curl-7697-139835625649184" Jul 22 02:22:46.895757 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:46.895665 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5722054-bf32-4223-a89a-ea9385d07bac" containerName="test-curl-7697-139835625649184" Jul 22 02:22:46.895806 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:46.895771 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5722054-bf32-4223-a89a-ea9385d07bac" containerName="test-curl-7697-139835625649184" Jul 22 02:22:46.898805 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:46.898790 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-139835625650192" Jul 22 02:22:46.906243 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:46.906215 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-139835625650192"] Jul 22 02:22:46.950351 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:46.950319 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tthrn\" (UniqueName: \"kubernetes.io/projected/d5722054-bf32-4223-a89a-ea9385d07bac-kube-api-access-tthrn\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:22:47.051171 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:47.051132 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkxr6\" (UniqueName: \"kubernetes.io/projected/445487e3-f0a7-4217-bbcb-222447b44fa2-kube-api-access-jkxr6\") pod \"test-curl-7697-139835625650192\" (UID: \"445487e3-f0a7-4217-bbcb-222447b44fa2\") " pod="opendatahub/test-curl-7697-139835625650192" Jul 22 02:22:47.152579 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:47.152497 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkxr6\" (UniqueName: \"kubernetes.io/projected/445487e3-f0a7-4217-bbcb-222447b44fa2-kube-api-access-jkxr6\") pod \"test-curl-7697-139835625650192\" (UID: \"445487e3-f0a7-4217-bbcb-222447b44fa2\") " pod="opendatahub/test-curl-7697-139835625650192" Jul 22 02:22:47.161810 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:47.161788 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkxr6\" (UniqueName: \"kubernetes.io/projected/445487e3-f0a7-4217-bbcb-222447b44fa2-kube-api-access-jkxr6\") pod \"test-curl-7697-139835625650192\" (UID: \"445487e3-f0a7-4217-bbcb-222447b44fa2\") " pod="opendatahub/test-curl-7697-139835625650192" Jul 22 02:22:47.209572 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:47.209548 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-139835625650192" Jul 22 02:22:47.343415 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:47.343389 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-139835625650192"] Jul 22 02:22:47.345287 ip-10-0-131-56 kubenswrapper[2561]: W0722 02:22:47.345260 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod445487e3_f0a7_4217_bbcb_222447b44fa2.slice/crio-9912d3f9f5715ae2a3a909c98567bcc07616293ddacd62f0fb853c891648fe62 WatchSource:0}: Error finding container 9912d3f9f5715ae2a3a909c98567bcc07616293ddacd62f0fb853c891648fe62: Status 404 returned error can't find the container with id 9912d3f9f5715ae2a3a909c98567bcc07616293ddacd62f0fb853c891648fe62 Jul 22 02:22:47.567480 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:47.567447 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-139835625649184" Jul 22 02:22:47.567650 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:47.567447 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5c84e01f074e6d01bd484340620ca1c2163ac64b13ac5dffe944bfb3d9e9ebe" Jul 22 02:22:47.568751 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:47.568722 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-139835625650192" event={"ID":"445487e3-f0a7-4217-bbcb-222447b44fa2","Type":"ContainerStarted","Data":"9912d3f9f5715ae2a3a909c98567bcc07616293ddacd62f0fb853c891648fe62"} Jul 22 02:22:47.577883 ip-10-0-131-56 kubenswrapper[2561]: E0722 02:22:47.577854 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7697-139835625649184\" is forbidden: User \"system:node:ip-10-0-131-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-56.ec2.internal' and this object" podUID="d5722054-bf32-4223-a89a-ea9385d07bac" pod="opendatahub/test-curl-7697-139835625649184" Jul 22 02:22:47.925186 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:47.925155 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5722054-bf32-4223-a89a-ea9385d07bac" path="/var/lib/kubelet/pods/d5722054-bf32-4223-a89a-ea9385d07bac/volumes" Jul 22 02:22:48.576483 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:48.576450 2561 generic.go:356] "Generic (PLEG): container finished" podID="445487e3-f0a7-4217-bbcb-222447b44fa2" containerID="446aebe8d0c7328d8c5e9cd11c4eef80680be669f0ac0b1683e828069d64d936" exitCode=0 Jul 22 02:22:48.576659 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:48.576515 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-139835625650192" event={"ID":"445487e3-f0a7-4217-bbcb-222447b44fa2","Type":"ContainerDied","Data":"446aebe8d0c7328d8c5e9cd11c4eef80680be669f0ac0b1683e828069d64d936"} Jul 22 02:22:49.728065 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:49.728042 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-139835625650192" Jul 22 02:22:49.748777 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:49.748754 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7697-139835625650192_445487e3-f0a7-4217-bbcb-222447b44fa2/test-curl-7697-139835625650192/0.log" Jul 22 02:22:49.776047 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:49.776025 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7697-139835625650192"] Jul 22 02:22:49.778521 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:49.778500 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7697-139835625650192"] Jul 22 02:22:49.877066 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:49.877006 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkxr6\" (UniqueName: \"kubernetes.io/projected/445487e3-f0a7-4217-bbcb-222447b44fa2-kube-api-access-jkxr6\") pod \"445487e3-f0a7-4217-bbcb-222447b44fa2\" (UID: \"445487e3-f0a7-4217-bbcb-222447b44fa2\") " Jul 22 02:22:49.879126 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:49.879100 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/445487e3-f0a7-4217-bbcb-222447b44fa2-kube-api-access-jkxr6" (OuterVolumeSpecName: "kube-api-access-jkxr6") pod "445487e3-f0a7-4217-bbcb-222447b44fa2" (UID: "445487e3-f0a7-4217-bbcb-222447b44fa2"). InnerVolumeSpecName "kube-api-access-jkxr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:22:49.925413 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:49.925389 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="445487e3-f0a7-4217-bbcb-222447b44fa2" path="/var/lib/kubelet/pods/445487e3-f0a7-4217-bbcb-222447b44fa2/volumes" Jul 22 02:22:49.978244 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:49.978221 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkxr6\" (UniqueName: \"kubernetes.io/projected/445487e3-f0a7-4217-bbcb-222447b44fa2-kube-api-access-jkxr6\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:22:50.588844 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:50.588811 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-139835625650192" Jul 22 02:22:50.588844 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:50.588832 2561 scope.go:117] "RemoveContainer" containerID="446aebe8d0c7328d8c5e9cd11c4eef80680be669f0ac0b1683e828069d64d936" Jul 22 02:22:52.990189 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:52.990155 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eba9c8-dcpxq"] Jul 22 02:22:52.990902 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:52.990882 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="445487e3-f0a7-4217-bbcb-222447b44fa2" containerName="test-curl-7697-139835625650192" Jul 22 02:22:52.990958 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:52.990906 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="445487e3-f0a7-4217-bbcb-222447b44fa2" containerName="test-curl-7697-139835625650192" Jul 22 02:22:52.991033 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:52.991019 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="445487e3-f0a7-4217-bbcb-222447b44fa2" containerName="test-curl-7697-139835625650192" Jul 22 02:22:52.996022 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:52.995994 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eba9c8-dcpxq" Jul 22 02:22:52.998551 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:52.998516 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cqdw8\"" Jul 22 02:22:52.999537 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:52.999513 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eba9c8-dcpxq"] Jul 22 02:22:53.104804 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:53.104778 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4627d3e-6276-458a-b3fa-df172c837a69-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-eba9c8-dcpxq\" (UID: \"c4627d3e-6276-458a-b3fa-df172c837a69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eba9c8-dcpxq" Jul 22 02:22:53.205734 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:53.205676 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4627d3e-6276-458a-b3fa-df172c837a69-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-eba9c8-dcpxq\" (UID: \"c4627d3e-6276-458a-b3fa-df172c837a69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eba9c8-dcpxq" Jul 22 02:22:53.206302 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:53.206280 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4627d3e-6276-458a-b3fa-df172c837a69-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-eba9c8-dcpxq\" (UID: \"c4627d3e-6276-458a-b3fa-df172c837a69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eba9c8-dcpxq" Jul 22 02:22:53.309233 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:53.309169 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eba9c8-dcpxq" Jul 22 02:22:53.449114 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:53.449080 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eba9c8-dcpxq"] Jul 22 02:22:53.450008 ip-10-0-131-56 kubenswrapper[2561]: W0722 02:22:53.449973 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4627d3e_6276_458a_b3fa_df172c837a69.slice/crio-9de22951a2264d7d06ae7c03f48172a6df62e7b6981a38307700a0cddfd86eeb WatchSource:0}: Error finding container 9de22951a2264d7d06ae7c03f48172a6df62e7b6981a38307700a0cddfd86eeb: Status 404 returned error can't find the container with id 9de22951a2264d7d06ae7c03f48172a6df62e7b6981a38307700a0cddfd86eeb Jul 22 02:22:53.608819 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:53.608744 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eba9c8-dcpxq" event={"ID":"c4627d3e-6276-458a-b3fa-df172c837a69","Type":"ContainerStarted","Data":"9de22951a2264d7d06ae7c03f48172a6df62e7b6981a38307700a0cddfd86eeb"} Jul 22 02:22:54.616862 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:54.616824 2561 generic.go:356] "Generic (PLEG): container finished" podID="c4627d3e-6276-458a-b3fa-df172c837a69" containerID="a8f556c9ecf5d428c84c7972d5c7c6df45f610b135ab6239748ef1f1509c7dcc" exitCode=0 Jul 22 02:22:54.617249 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:54.616915 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eba9c8-dcpxq" event={"ID":"c4627d3e-6276-458a-b3fa-df172c837a69","Type":"ContainerDied","Data":"a8f556c9ecf5d428c84c7972d5c7c6df45f610b135ab6239748ef1f1509c7dcc"} Jul 22 02:22:55.763258 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:55.763236 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eba9c8-dcpxq" Jul 22 02:22:55.927924 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:55.927899 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4627d3e-6276-458a-b3fa-df172c837a69-maas-api-service-ca\") pod \"c4627d3e-6276-458a-b3fa-df172c837a69\" (UID: \"c4627d3e-6276-458a-b3fa-df172c837a69\") " Jul 22 02:22:55.928267 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:55.928247 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4627d3e-6276-458a-b3fa-df172c837a69-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c4627d3e-6276-458a-b3fa-df172c837a69" (UID: "c4627d3e-6276-458a-b3fa-df172c837a69"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:22:56.029076 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:56.029047 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4627d3e-6276-458a-b3fa-df172c837a69-maas-api-service-ca\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:22:56.630564 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:56.630526 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eba9c8-dcpxq" event={"ID":"c4627d3e-6276-458a-b3fa-df172c837a69","Type":"ContainerDied","Data":"9de22951a2264d7d06ae7c03f48172a6df62e7b6981a38307700a0cddfd86eeb"} Jul 22 02:22:56.630564 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:56.630555 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eba9c8-dcpxq" Jul 22 02:22:56.630767 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:22:56.630574 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9de22951a2264d7d06ae7c03f48172a6df62e7b6981a38307700a0cddfd86eeb" Jul 22 02:23:08.085186 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:08.085152 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eba9c8-dcpxq"] Jul 22 02:23:08.088256 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:08.088232 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-eba9c8-dcpxq"] Jul 22 02:23:09.925297 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:09.925266 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4627d3e-6276-458a-b3fa-df172c837a69" path="/var/lib/kubelet/pods/c4627d3e-6276-458a-b3fa-df172c837a69/volumes" Jul 22 02:23:15.288380 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:15.288338 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6960c-kpfzj"] Jul 22 02:23:15.289874 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:15.289845 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4627d3e-6276-458a-b3fa-df172c837a69" containerName="revoke-keys" Jul 22 02:23:15.290029 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:15.290017 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4627d3e-6276-458a-b3fa-df172c837a69" containerName="revoke-keys" Jul 22 02:23:15.290388 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:15.290370 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4627d3e-6276-458a-b3fa-df172c837a69" containerName="revoke-keys" Jul 22 02:23:15.296814 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:15.296785 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6960c-kpfzj" Jul 22 02:23:15.298219 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:15.298190 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6960c-kpfzj"] Jul 22 02:23:15.299521 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:15.299498 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cqdw8\"" Jul 22 02:23:15.398146 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:15.398118 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1be654e6-72e1-4ab8-9a2a-e2ad9ce29451-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-c6960c-kpfzj\" (UID: \"1be654e6-72e1-4ab8-9a2a-e2ad9ce29451\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6960c-kpfzj" Jul 22 02:23:15.498930 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:15.498896 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1be654e6-72e1-4ab8-9a2a-e2ad9ce29451-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-c6960c-kpfzj\" (UID: \"1be654e6-72e1-4ab8-9a2a-e2ad9ce29451\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6960c-kpfzj" Jul 22 02:23:15.499470 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:15.499450 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1be654e6-72e1-4ab8-9a2a-e2ad9ce29451-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-c6960c-kpfzj\" (UID: \"1be654e6-72e1-4ab8-9a2a-e2ad9ce29451\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6960c-kpfzj" Jul 22 02:23:15.609225 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:15.609152 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6960c-kpfzj" Jul 22 02:23:15.749722 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:15.749676 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6960c-kpfzj"] Jul 22 02:23:15.751448 ip-10-0-131-56 kubenswrapper[2561]: W0722 02:23:15.751421 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be654e6_72e1_4ab8_9a2a_e2ad9ce29451.slice/crio-ff1732b0e59a533fc2ef25ef126cb270096928278cba8c7b291335ca804fa818 WatchSource:0}: Error finding container ff1732b0e59a533fc2ef25ef126cb270096928278cba8c7b291335ca804fa818: Status 404 returned error can't find the container with id ff1732b0e59a533fc2ef25ef126cb270096928278cba8c7b291335ca804fa818 Jul 22 02:23:16.752760 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:16.752730 2561 generic.go:356] "Generic (PLEG): container finished" podID="1be654e6-72e1-4ab8-9a2a-e2ad9ce29451" containerID="bc823984b0cea0a96783d7d2c7954c2b906f284635c060c6a0dceec12358dcc3" exitCode=0 Jul 22 02:23:16.753123 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:16.752818 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6960c-kpfzj" event={"ID":"1be654e6-72e1-4ab8-9a2a-e2ad9ce29451","Type":"ContainerDied","Data":"bc823984b0cea0a96783d7d2c7954c2b906f284635c060c6a0dceec12358dcc3"} Jul 22 02:23:16.753123 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:16.752870 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6960c-kpfzj" event={"ID":"1be654e6-72e1-4ab8-9a2a-e2ad9ce29451","Type":"ContainerStarted","Data":"ff1732b0e59a533fc2ef25ef126cb270096928278cba8c7b291335ca804fa818"} Jul 22 02:23:17.898658 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:17.898633 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6960c-kpfzj" Jul 22 02:23:18.023436 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:18.023361 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1be654e6-72e1-4ab8-9a2a-e2ad9ce29451-maas-api-service-ca\") pod \"1be654e6-72e1-4ab8-9a2a-e2ad9ce29451\" (UID: \"1be654e6-72e1-4ab8-9a2a-e2ad9ce29451\") " Jul 22 02:23:18.023813 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:18.023784 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1be654e6-72e1-4ab8-9a2a-e2ad9ce29451-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1be654e6-72e1-4ab8-9a2a-e2ad9ce29451" (UID: "1be654e6-72e1-4ab8-9a2a-e2ad9ce29451"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:23:18.124392 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:18.124364 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1be654e6-72e1-4ab8-9a2a-e2ad9ce29451-maas-api-service-ca\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:23:18.765251 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:18.765216 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6960c-kpfzj" event={"ID":"1be654e6-72e1-4ab8-9a2a-e2ad9ce29451","Type":"ContainerDied","Data":"ff1732b0e59a533fc2ef25ef126cb270096928278cba8c7b291335ca804fa818"} Jul 22 02:23:18.765251 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:18.765237 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6960c-kpfzj" Jul 22 02:23:18.765483 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:18.765274 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff1732b0e59a533fc2ef25ef126cb270096928278cba8c7b291335ca804fa818" Jul 22 02:23:25.831926 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:25.831895 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2"] Jul 22 02:23:25.832375 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:25.832131 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2" podUID="a1debdc1-148e-4aee-bcab-791cdc9c6001" containerName="maas-api" containerID="cri-o://7973d764e9de006cef1e4928c979c8a7ae4e49d9fc3b66334fc50af567c9e5b1" gracePeriod=30 Jul 22 02:23:26.093011 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:26.092953 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2" Jul 22 02:23:26.193623 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:26.193597 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1debdc1-148e-4aee-bcab-791cdc9c6001-openshift-service-ca\") pod \"a1debdc1-148e-4aee-bcab-791cdc9c6001\" (UID: \"a1debdc1-148e-4aee-bcab-791cdc9c6001\") " Jul 22 02:23:26.193813 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:26.193664 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1debdc1-148e-4aee-bcab-791cdc9c6001-maas-api-tls\") pod \"a1debdc1-148e-4aee-bcab-791cdc9c6001\" (UID: \"a1debdc1-148e-4aee-bcab-791cdc9c6001\") " Jul 22 02:23:26.193813 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:26.193776 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h54m7\" (UniqueName: \"kubernetes.io/projected/a1debdc1-148e-4aee-bcab-791cdc9c6001-kube-api-access-h54m7\") pod \"a1debdc1-148e-4aee-bcab-791cdc9c6001\" (UID: \"a1debdc1-148e-4aee-bcab-791cdc9c6001\") " Jul 22 02:23:26.193995 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:26.193963 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1debdc1-148e-4aee-bcab-791cdc9c6001-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a1debdc1-148e-4aee-bcab-791cdc9c6001" (UID: "a1debdc1-148e-4aee-bcab-791cdc9c6001"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:23:26.194101 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:26.194085 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1debdc1-148e-4aee-bcab-791cdc9c6001-openshift-service-ca\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:23:26.195942 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:26.195914 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1debdc1-148e-4aee-bcab-791cdc9c6001-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a1debdc1-148e-4aee-bcab-791cdc9c6001" (UID: "a1debdc1-148e-4aee-bcab-791cdc9c6001"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:23:26.196055 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:26.196026 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1debdc1-148e-4aee-bcab-791cdc9c6001-kube-api-access-h54m7" (OuterVolumeSpecName: "kube-api-access-h54m7") pod "a1debdc1-148e-4aee-bcab-791cdc9c6001" (UID: "a1debdc1-148e-4aee-bcab-791cdc9c6001"). InnerVolumeSpecName "kube-api-access-h54m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:23:26.295328 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:26.295303 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h54m7\" (UniqueName: \"kubernetes.io/projected/a1debdc1-148e-4aee-bcab-791cdc9c6001-kube-api-access-h54m7\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:23:26.295328 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:26.295326 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1debdc1-148e-4aee-bcab-791cdc9c6001-maas-api-tls\") on node \"ip-10-0-131-56.ec2.internal\" DevicePath \"\"" Jul 22 02:23:26.812267 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:26.812233 2561 generic.go:356] "Generic (PLEG): container finished" podID="a1debdc1-148e-4aee-bcab-791cdc9c6001" containerID="7973d764e9de006cef1e4928c979c8a7ae4e49d9fc3b66334fc50af567c9e5b1" exitCode=0 Jul 22 02:23:26.812456 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:26.812276 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2" Jul 22 02:23:26.812456 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:26.812321 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2" event={"ID":"a1debdc1-148e-4aee-bcab-791cdc9c6001","Type":"ContainerDied","Data":"7973d764e9de006cef1e4928c979c8a7ae4e49d9fc3b66334fc50af567c9e5b1"} Jul 22 02:23:26.812456 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:26.812381 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2" event={"ID":"a1debdc1-148e-4aee-bcab-791cdc9c6001","Type":"ContainerDied","Data":"ed3f3fdbe13796f62afd982093da9516504413ce0aa24474880fa85caab539b6"} Jul 22 02:23:26.812456 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:26.812405 2561 scope.go:117] "RemoveContainer" containerID="7973d764e9de006cef1e4928c979c8a7ae4e49d9fc3b66334fc50af567c9e5b1" Jul 22 02:23:26.826671 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:26.826646 2561 scope.go:117] "RemoveContainer" containerID="7973d764e9de006cef1e4928c979c8a7ae4e49d9fc3b66334fc50af567c9e5b1" Jul 22 02:23:26.827119 ip-10-0-131-56 kubenswrapper[2561]: E0722 02:23:26.827090 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7973d764e9de006cef1e4928c979c8a7ae4e49d9fc3b66334fc50af567c9e5b1\": container with ID starting with 7973d764e9de006cef1e4928c979c8a7ae4e49d9fc3b66334fc50af567c9e5b1 not found: ID does not exist" containerID="7973d764e9de006cef1e4928c979c8a7ae4e49d9fc3b66334fc50af567c9e5b1" Jul 22 02:23:26.827220 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:26.827130 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7973d764e9de006cef1e4928c979c8a7ae4e49d9fc3b66334fc50af567c9e5b1"} err="failed to get container status \"7973d764e9de006cef1e4928c979c8a7ae4e49d9fc3b66334fc50af567c9e5b1\": rpc error: code = NotFound desc = could not find container \"7973d764e9de006cef1e4928c979c8a7ae4e49d9fc3b66334fc50af567c9e5b1\": container with ID starting with 7973d764e9de006cef1e4928c979c8a7ae4e49d9fc3b66334fc50af567c9e5b1 not found: ID does not exist" Jul 22 02:23:26.843762 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:26.843738 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2"] Jul 22 02:23:26.849322 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:26.849292 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c6960c-6d5cccc7bb-tjwl2"] Jul 22 02:23:27.925408 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:27.925373 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1debdc1-148e-4aee-bcab-791cdc9c6001" path="/var/lib/kubelet/pods/a1debdc1-148e-4aee-bcab-791cdc9c6001/volumes" Jul 22 02:23:30.379318 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:30.379240 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6960c-kpfzj"] Jul 22 02:23:30.388406 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:30.388375 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6960c-kpfzj"] Jul 22 02:23:31.925560 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:31.925526 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1be654e6-72e1-4ab8-9a2a-e2ad9ce29451" path="/var/lib/kubelet/pods/1be654e6-72e1-4ab8-9a2a-e2ad9ce29451/volumes" Jul 22 02:23:44.440524 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:44.440494 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-t4pkb_8b563dcf-e028-4eaa-b33c-164306146411/manager/0.log" Jul 22 02:23:44.566492 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:44.566463 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-j9twf_17416ded-8f9b-4822-8800-d65e3bf062b3/manager/0.log" Jul 22 02:23:44.698033 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:44.697961 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9cc45ddbb-qtw4v_2f068eb0-f72d-408d-af6c-ccc299da8009/manager/0.log" Jul 22 02:23:44.825926 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:44.825901 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-pnbqj_7fc9fa98-e5a6-4662-8a43-97c9373f5938/server/0.log" Jul 22 02:23:44.952436 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:44.952363 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-f724l_233fa350-ced9-4ddb-8ddd-9aa88e1d32d1/manager/0.log" Jul 22 02:23:45.211505 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:45.211438 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c578697c6-lfq9r_7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135/manager/0.log" Jul 22 02:23:45.224673 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:45.224647 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c578697c6-lfq9r_7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135/copy-manifests/0.log" Jul 22 02:23:45.751139 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:45.751110 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6f898ff5d6-5kx2g_b65d3c5b-99a3-4526-b567-79106c0c9217/maas-api/0.log" Jul 22 02:23:46.016854 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:46.016778 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29744775-5nwbw_83846c24-5e11-4da6-8628-3b6b62964476/cleanup/0.log" Jul 22 02:23:46.152989 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:46.152960 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-24rfr_4f1e2188-e71a-4611-a144-722d4f9a2454/postgres/0.log" Jul 22 02:23:47.062782 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:47.062753 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc_e48954b4-33a3-4744-893d-cceb6c6c6ed0/util/0.log" Jul 22 02:23:47.070647 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:47.070622 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc_e48954b4-33a3-4744-893d-cceb6c6c6ed0/pull/0.log" Jul 22 02:23:47.080310 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:47.080290 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc_e48954b4-33a3-4744-893d-cceb6c6c6ed0/extract/0.log" Jul 22 02:23:47.197199 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:47.197173 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8_2dba0d85-6bd2-4b3a-9811-eb5b58d6355b/util/0.log" Jul 22 02:23:47.204063 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:47.204044 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8_2dba0d85-6bd2-4b3a-9811-eb5b58d6355b/pull/0.log" Jul 22 02:23:47.211616 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:47.211593 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8_2dba0d85-6bd2-4b3a-9811-eb5b58d6355b/extract/0.log" Jul 22 02:23:47.337070 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:47.336992 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb_97270ed7-a6bb-4bb3-898e-b08205c7ef5e/util/0.log" Jul 22 02:23:47.344431 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:47.344406 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb_97270ed7-a6bb-4bb3-898e-b08205c7ef5e/pull/0.log" Jul 22 02:23:47.351682 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:47.351663 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb_97270ed7-a6bb-4bb3-898e-b08205c7ef5e/extract/0.log" Jul 22 02:23:47.489375 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:47.489344 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf_3ec8f212-80ab-497b-8f4f-da31514f063e/util/0.log" Jul 22 02:23:47.497821 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:47.497803 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf_3ec8f212-80ab-497b-8f4f-da31514f063e/pull/0.log" Jul 22 02:23:47.505531 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:47.505510 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf_3ec8f212-80ab-497b-8f4f-da31514f063e/extract/0.log" Jul 22 02:23:47.792322 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:47.792299 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-hl4dc_ff897a45-88e9-48e8-a2e7-b9f33d926399/manager/0.log" Jul 22 02:23:47.941030 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:47.941010 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-xn275_8e9c947f-fe51-4870-b616-8fe740c067e5/manager/0.log" Jul 22 02:23:48.206959 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:48.206932 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-z2n5w_7b31df87-eb54-4fb0-9b11-a4abcb0287d1/registry-server/0.log" Jul 22 02:23:48.359287 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:48.359256 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-glwnx_54c07509-dea0-4ab0-a2fc-e90694a5ca50/manager/0.log" Jul 22 02:23:48.641414 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:48.641342 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-5nzlv_09fcdea7-4413-45df-a08b-6e05a3bc4243/manager/0.log" Jul 22 02:23:49.317742 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:49.317711 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5975cd7b94-4zh5g_32e6dba1-d57f-4e7c-bed0-2264b16668ea/kube-auth-proxy/0.log" Jul 22 02:23:49.724075 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:49.724045 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-hqg49_c084ceb9-3866-47e4-888c-1a79d2a91e5f/payload-pre-processing/0.log" Jul 22 02:23:49.858682 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:49.858644 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-tsdbn_011f7fd0-7031-49ab-8eb2-ea5dc4a3f6ad/payload-processing/0.log" Jul 22 02:23:50.538483 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:50.538440 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-nl64t_9be25cbc-bb3b-43f0-b158-a3ab08b7d6a9/main/0.log" Jul 22 02:23:50.935486 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:50.935461 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-zvrdx_0cbbc5fd-55ba-4f44-ac9b-bf9ccc347424/main/0.log" Jul 22 02:23:57.547183 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:57.547147 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-knlbt/must-gather-zwwdm"] Jul 22 02:23:57.547622 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:57.547608 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1debdc1-148e-4aee-bcab-791cdc9c6001" containerName="maas-api" Jul 22 02:23:57.547622 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:57.547622 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1debdc1-148e-4aee-bcab-791cdc9c6001" containerName="maas-api" Jul 22 02:23:57.547738 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:57.547642 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1be654e6-72e1-4ab8-9a2a-e2ad9ce29451" containerName="revoke-keys" Jul 22 02:23:57.547738 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:57.547647 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1be654e6-72e1-4ab8-9a2a-e2ad9ce29451" containerName="revoke-keys" Jul 22 02:23:57.547738 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:57.547728 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1debdc1-148e-4aee-bcab-791cdc9c6001" containerName="maas-api" Jul 22 02:23:57.547940 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:57.547741 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1be654e6-72e1-4ab8-9a2a-e2ad9ce29451" containerName="revoke-keys" Jul 22 02:23:57.551061 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:57.551045 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-knlbt/must-gather-zwwdm" Jul 22 02:23:57.553493 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:57.553465 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-knlbt\"/\"kube-root-ca.crt\"" Jul 22 02:23:57.553623 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:57.553536 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-knlbt\"/\"openshift-service-ca.crt\"" Jul 22 02:23:57.554482 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:57.554468 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-knlbt\"/\"default-dockercfg-9xmg5\"" Jul 22 02:23:57.562253 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:57.562229 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-knlbt/must-gather-zwwdm"] Jul 22 02:23:57.573056 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:57.573032 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dff2af19-c67a-483b-8768-4da1d26c5208-must-gather-output\") pod \"must-gather-zwwdm\" (UID: \"dff2af19-c67a-483b-8768-4da1d26c5208\") " pod="openshift-must-gather-knlbt/must-gather-zwwdm" Jul 22 02:23:57.573166 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:57.573078 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n97ft\" (UniqueName: \"kubernetes.io/projected/dff2af19-c67a-483b-8768-4da1d26c5208-kube-api-access-n97ft\") pod \"must-gather-zwwdm\" (UID: \"dff2af19-c67a-483b-8768-4da1d26c5208\") " pod="openshift-must-gather-knlbt/must-gather-zwwdm" Jul 22 02:23:57.674092 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:57.674071 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dff2af19-c67a-483b-8768-4da1d26c5208-must-gather-output\") pod \"must-gather-zwwdm\" (UID: \"dff2af19-c67a-483b-8768-4da1d26c5208\") " pod="openshift-must-gather-knlbt/must-gather-zwwdm" Jul 22 02:23:57.674220 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:57.674104 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n97ft\" (UniqueName: \"kubernetes.io/projected/dff2af19-c67a-483b-8768-4da1d26c5208-kube-api-access-n97ft\") pod \"must-gather-zwwdm\" (UID: \"dff2af19-c67a-483b-8768-4da1d26c5208\") " pod="openshift-must-gather-knlbt/must-gather-zwwdm" Jul 22 02:23:57.674397 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:57.674379 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dff2af19-c67a-483b-8768-4da1d26c5208-must-gather-output\") pod \"must-gather-zwwdm\" (UID: \"dff2af19-c67a-483b-8768-4da1d26c5208\") " pod="openshift-must-gather-knlbt/must-gather-zwwdm" Jul 22 02:23:57.682529 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:57.682498 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n97ft\" (UniqueName: \"kubernetes.io/projected/dff2af19-c67a-483b-8768-4da1d26c5208-kube-api-access-n97ft\") pod \"must-gather-zwwdm\" (UID: \"dff2af19-c67a-483b-8768-4da1d26c5208\") " pod="openshift-must-gather-knlbt/must-gather-zwwdm" Jul 22 02:23:57.860713 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:57.860625 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-knlbt/must-gather-zwwdm" Jul 22 02:23:58.001576 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:58.001546 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-knlbt/must-gather-zwwdm"] Jul 22 02:23:58.002765 ip-10-0-131-56 kubenswrapper[2561]: W0722 02:23:58.002743 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddff2af19_c67a_483b_8768_4da1d26c5208.slice/crio-3617c378a946d9151b49a15ecf366d4517a31e9fe019e2034f23d2f646693ed0 WatchSource:0}: Error finding container 3617c378a946d9151b49a15ecf366d4517a31e9fe019e2034f23d2f646693ed0: Status 404 returned error can't find the container with id 3617c378a946d9151b49a15ecf366d4517a31e9fe019e2034f23d2f646693ed0 Jul 22 02:23:59.013586 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:59.013411 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-knlbt/must-gather-zwwdm" event={"ID":"dff2af19-c67a-483b-8768-4da1d26c5208","Type":"ContainerStarted","Data":"7842fa71975cb6437b05f2a7fe4616c759cd0eb5496b247ac4054d338f1b8078"} Jul 22 02:23:59.013586 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:59.013463 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-knlbt/must-gather-zwwdm" event={"ID":"dff2af19-c67a-483b-8768-4da1d26c5208","Type":"ContainerStarted","Data":"d90d4aac89f040487e8504ccb485b51a44e858f5013983f3610721a81a8000fa"} Jul 22 02:23:59.013586 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:59.013482 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-knlbt/must-gather-zwwdm" event={"ID":"dff2af19-c67a-483b-8768-4da1d26c5208","Type":"ContainerStarted","Data":"3617c378a946d9151b49a15ecf366d4517a31e9fe019e2034f23d2f646693ed0"} Jul 22 02:23:59.030325 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:23:59.030270 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-knlbt/must-gather-zwwdm" podStartSLOduration=1.244118666 podStartE2EDuration="2.030252389s" totalImagesPullingTime="786.133723ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:23:59.028866892 +0000 UTC m=+2809.621671257" watchObservedRunningTime="2026-07-22 02:23:59.030252389 +0000 UTC m=+2809.623056752" Jul 22 02:24:00.516598 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:00.516563 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ghl8b_448e76a1-df97-4a26-8433-7a51cdeb4a5a/global-pull-secret-syncer/0.log" Jul 22 02:24:00.664676 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:00.664647 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gzgpx_50a4581a-21ba-4900-aee7-648e18e7abe5/konnectivity-agent/0.log" Jul 22 02:24:00.751409 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:00.751350 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-56.ec2.internal_8f30de953d78a535da92683887f7e533/haproxy/0.log" Jul 22 02:24:04.319272 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:04.319236 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc_e48954b4-33a3-4744-893d-cceb6c6c6ed0/extract/0.log" Jul 22 02:24:04.367800 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:04.367773 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc_e48954b4-33a3-4744-893d-cceb6c6c6ed0/util/0.log" Jul 22 02:24:04.401387 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:04.401352 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zfnlc_e48954b4-33a3-4744-893d-cceb6c6c6ed0/pull/0.log" Jul 22 02:24:04.502244 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:04.502188 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8_2dba0d85-6bd2-4b3a-9811-eb5b58d6355b/extract/0.log" Jul 22 02:24:04.547312 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:04.547282 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8_2dba0d85-6bd2-4b3a-9811-eb5b58d6355b/util/0.log" Jul 22 02:24:04.576578 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:04.576478 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mlzz8_2dba0d85-6bd2-4b3a-9811-eb5b58d6355b/pull/0.log" Jul 22 02:24:04.610313 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:04.610284 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb_97270ed7-a6bb-4bb3-898e-b08205c7ef5e/extract/0.log" Jul 22 02:24:04.640489 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:04.640462 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb_97270ed7-a6bb-4bb3-898e-b08205c7ef5e/util/0.log" Jul 22 02:24:04.675433 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:04.675371 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735mpjb_97270ed7-a6bb-4bb3-898e-b08205c7ef5e/pull/0.log" Jul 22 02:24:04.710715 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:04.710665 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf_3ec8f212-80ab-497b-8f4f-da31514f063e/extract/0.log" Jul 22 02:24:04.738980 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:04.738952 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf_3ec8f212-80ab-497b-8f4f-da31514f063e/util/0.log" Jul 22 02:24:04.769558 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:04.769530 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1msqwf_3ec8f212-80ab-497b-8f4f-da31514f063e/pull/0.log" Jul 22 02:24:05.342755 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:05.342632 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-hl4dc_ff897a45-88e9-48e8-a2e7-b9f33d926399/manager/0.log" Jul 22 02:24:05.373140 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:05.372980 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-xn275_8e9c947f-fe51-4870-b616-8fe740c067e5/manager/0.log" Jul 22 02:24:05.485089 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:05.485058 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-z2n5w_7b31df87-eb54-4fb0-9b11-a4abcb0287d1/registry-server/0.log" Jul 22 02:24:05.666772 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:05.664641 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-glwnx_54c07509-dea0-4ab0-a2fc-e90694a5ca50/manager/0.log" Jul 22 02:24:05.803224 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:05.803175 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-5nzlv_09fcdea7-4413-45df-a08b-6e05a3bc4243/manager/0.log" Jul 22 02:24:07.543228 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:07.543197 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6db543ab-b8f6-4f88-a53b-7f37db4799fb/alertmanager/0.log" Jul 22 02:24:07.600670 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:07.600579 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6db543ab-b8f6-4f88-a53b-7f37db4799fb/config-reloader/0.log" Jul 22 02:24:07.652249 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:07.650564 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6db543ab-b8f6-4f88-a53b-7f37db4799fb/kube-rbac-proxy-web/0.log" Jul 22 02:24:07.731443 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:07.731373 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6db543ab-b8f6-4f88-a53b-7f37db4799fb/kube-rbac-proxy/0.log" Jul 22 02:24:07.815322 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:07.815295 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6db543ab-b8f6-4f88-a53b-7f37db4799fb/kube-rbac-proxy-metric/0.log" Jul 22 02:24:07.874707 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:07.874581 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6db543ab-b8f6-4f88-a53b-7f37db4799fb/prom-label-proxy/0.log" Jul 22 02:24:07.932109 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:07.932084 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6db543ab-b8f6-4f88-a53b-7f37db4799fb/init-config-reloader/0.log" Jul 22 02:24:08.209572 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:08.209543 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7ccd8878ff-lsgj6_35a74ac3-5716-4629-8eb8-cf115f89234b/metrics-server/0.log" Jul 22 02:24:08.265480 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:08.265450 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-cz8pk_5ac1bc03-0f09-4262-9160-1c19119eb8f4/monitoring-plugin/0.log" Jul 22 02:24:08.616320 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:08.616281 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tslm7_6acfad03-6cc4-4291-a5d4-97256ea86c87/node-exporter/0.log" Jul 22 02:24:08.670119 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:08.670088 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tslm7_6acfad03-6cc4-4291-a5d4-97256ea86c87/kube-rbac-proxy/0.log" Jul 22 02:24:08.726826 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:08.726798 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tslm7_6acfad03-6cc4-4291-a5d4-97256ea86c87/init-textfile/0.log" Jul 22 02:24:08.783618 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:08.783537 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-ncwrs_e08c73c4-2356-4d3e-add0-16acc7193db7/kube-rbac-proxy-main/0.log" Jul 22 02:24:08.819372 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:08.819336 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-ncwrs_e08c73c4-2356-4d3e-add0-16acc7193db7/kube-rbac-proxy-self/0.log" Jul 22 02:24:08.867948 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:08.867875 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-ncwrs_e08c73c4-2356-4d3e-add0-16acc7193db7/openshift-state-metrics/0.log" Jul 22 02:24:09.322503 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:09.322451 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-thj9m_2c9c232c-a93d-47aa-88a2-90fa05b1bc47/prometheus-operator-admission-webhook/0.log" Jul 22 02:24:09.363632 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:09.363594 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f5dd86dd9-dqjhz_7262c89e-c2bf-4fc2-915d-b91ee295628e/telemeter-client/0.log" Jul 22 02:24:09.398218 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:09.398190 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f5dd86dd9-dqjhz_7262c89e-c2bf-4fc2-915d-b91ee295628e/reload/0.log" Jul 22 02:24:09.427243 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:09.427215 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f5dd86dd9-dqjhz_7262c89e-c2bf-4fc2-915d-b91ee295628e/kube-rbac-proxy/0.log" Jul 22 02:24:09.484376 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:09.484350 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bc54c4b65-8rdms_fdbe34c0-975c-4776-863b-5c06b0b5b850/thanos-query/0.log" Jul 22 02:24:09.523799 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:09.523762 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bc54c4b65-8rdms_fdbe34c0-975c-4776-863b-5c06b0b5b850/kube-rbac-proxy-web/0.log" Jul 22 02:24:09.570813 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:09.570788 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bc54c4b65-8rdms_fdbe34c0-975c-4776-863b-5c06b0b5b850/kube-rbac-proxy/0.log" Jul 22 02:24:09.611489 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:09.611411 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bc54c4b65-8rdms_fdbe34c0-975c-4776-863b-5c06b0b5b850/prom-label-proxy/0.log" Jul 22 02:24:09.643584 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:09.643546 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bc54c4b65-8rdms_fdbe34c0-975c-4776-863b-5c06b0b5b850/kube-rbac-proxy-rules/0.log" Jul 22 02:24:09.679280 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:09.679254 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bc54c4b65-8rdms_fdbe34c0-975c-4776-863b-5c06b0b5b850/kube-rbac-proxy-metrics/0.log" Jul 22 02:24:10.112529 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:10.112486 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-knlbt/perf-node-gather-daemonset-pfjvk"] Jul 22 02:24:10.123509 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:10.122176 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-pfjvk" Jul 22 02:24:10.131955 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:10.131925 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-knlbt/perf-node-gather-daemonset-pfjvk"] Jul 22 02:24:10.211133 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:10.211092 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsbn4\" (UniqueName: \"kubernetes.io/projected/fcb138ae-7951-485f-943d-841f5f468520-kube-api-access-jsbn4\") pod \"perf-node-gather-daemonset-pfjvk\" (UID: \"fcb138ae-7951-485f-943d-841f5f468520\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-pfjvk" Jul 22 02:24:10.211323 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:10.211154 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fcb138ae-7951-485f-943d-841f5f468520-podres\") pod \"perf-node-gather-daemonset-pfjvk\" (UID: \"fcb138ae-7951-485f-943d-841f5f468520\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-pfjvk" Jul 22 02:24:10.211323 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:10.211238 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcb138ae-7951-485f-943d-841f5f468520-sys\") pod \"perf-node-gather-daemonset-pfjvk\" (UID: \"fcb138ae-7951-485f-943d-841f5f468520\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-pfjvk" Jul 22 02:24:10.211323 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:10.211281 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fcb138ae-7951-485f-943d-841f5f468520-lib-modules\") pod \"perf-node-gather-daemonset-pfjvk\" (UID: \"fcb138ae-7951-485f-943d-841f5f468520\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-pfjvk" Jul 22 02:24:10.211469 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:10.211376 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fcb138ae-7951-485f-943d-841f5f468520-proc\") pod \"perf-node-gather-daemonset-pfjvk\" (UID: \"fcb138ae-7951-485f-943d-841f5f468520\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-pfjvk" Jul 22 02:24:10.312483 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:10.312448 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fcb138ae-7951-485f-943d-841f5f468520-proc\") pod \"perf-node-gather-daemonset-pfjvk\" (UID: \"fcb138ae-7951-485f-943d-841f5f468520\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-pfjvk" Jul 22 02:24:10.312809 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:10.312789 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsbn4\" (UniqueName: \"kubernetes.io/projected/fcb138ae-7951-485f-943d-841f5f468520-kube-api-access-jsbn4\") pod \"perf-node-gather-daemonset-pfjvk\" (UID: \"fcb138ae-7951-485f-943d-841f5f468520\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-pfjvk" Jul 22 02:24:10.312942 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:10.312929 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fcb138ae-7951-485f-943d-841f5f468520-podres\") pod \"perf-node-gather-daemonset-pfjvk\" (UID: \"fcb138ae-7951-485f-943d-841f5f468520\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-pfjvk" Jul 22 02:24:10.313094 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:10.313082 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcb138ae-7951-485f-943d-841f5f468520-sys\") pod \"perf-node-gather-daemonset-pfjvk\" (UID: \"fcb138ae-7951-485f-943d-841f5f468520\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-pfjvk" Jul 22 02:24:10.313201 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:10.313189 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fcb138ae-7951-485f-943d-841f5f468520-lib-modules\") pod \"perf-node-gather-daemonset-pfjvk\" (UID: \"fcb138ae-7951-485f-943d-841f5f468520\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-pfjvk" Jul 22 02:24:10.313426 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:10.313412 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fcb138ae-7951-485f-943d-841f5f468520-lib-modules\") pod \"perf-node-gather-daemonset-pfjvk\" (UID: \"fcb138ae-7951-485f-943d-841f5f468520\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-pfjvk" Jul 22 02:24:10.313545 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:10.312589 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fcb138ae-7951-485f-943d-841f5f468520-proc\") pod \"perf-node-gather-daemonset-pfjvk\" (UID: \"fcb138ae-7951-485f-943d-841f5f468520\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-pfjvk" Jul 22 02:24:10.313680 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:10.313551 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcb138ae-7951-485f-943d-841f5f468520-sys\") pod \"perf-node-gather-daemonset-pfjvk\" (UID: \"fcb138ae-7951-485f-943d-841f5f468520\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-pfjvk" Jul 22 02:24:10.313680 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:10.313610 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fcb138ae-7951-485f-943d-841f5f468520-podres\") pod \"perf-node-gather-daemonset-pfjvk\" (UID: \"fcb138ae-7951-485f-943d-841f5f468520\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-pfjvk" Jul 22 02:24:10.325614 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:10.325584 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsbn4\" (UniqueName: \"kubernetes.io/projected/fcb138ae-7951-485f-943d-841f5f468520-kube-api-access-jsbn4\") pod \"perf-node-gather-daemonset-pfjvk\" (UID: \"fcb138ae-7951-485f-943d-841f5f468520\") " pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-pfjvk" Jul 22 02:24:10.441294 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:10.441043 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-pfjvk" Jul 22 02:24:10.646528 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:10.646495 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-knlbt/perf-node-gather-daemonset-pfjvk"] Jul 22 02:24:10.651081 ip-10-0-131-56 kubenswrapper[2561]: W0722 02:24:10.651048 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfcb138ae_7951_485f_943d_841f5f468520.slice/crio-e97c5644cd3a8e12c38726e6e59fe0eb73d35e7ceff766b725e07e449662ca14 WatchSource:0}: Error finding container e97c5644cd3a8e12c38726e6e59fe0eb73d35e7ceff766b725e07e449662ca14: Status 404 returned error can't find the container with id e97c5644cd3a8e12c38726e6e59fe0eb73d35e7ceff766b725e07e449662ca14 Jul 22 02:24:11.137023 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:11.136929 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-pfjvk" event={"ID":"fcb138ae-7951-485f-943d-841f5f468520","Type":"ContainerStarted","Data":"3fafe8125fba2344ad79c550faeda8096a44965a49c04f01e398452ee95e8782"} Jul 22 02:24:11.137023 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:11.136996 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-pfjvk" event={"ID":"fcb138ae-7951-485f-943d-841f5f468520","Type":"ContainerStarted","Data":"e97c5644cd3a8e12c38726e6e59fe0eb73d35e7ceff766b725e07e449662ca14"} Jul 22 02:24:11.137260 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:11.137054 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-pfjvk" Jul 22 02:24:11.169210 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:11.169159 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-pfjvk" podStartSLOduration=1.169144223 podStartE2EDuration="1.169144223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:24:11.168637009 +0000 UTC m=+2821.761441371" watchObservedRunningTime="2026-07-22 02:24:11.169144223 +0000 UTC m=+2821.761948586" Jul 22 02:24:12.152762 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:12.152731 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-r6jqj_cd37a5f0-67e8-4a2c-ade0-528e281a7a3e/download-server/0.log" Jul 22 02:24:13.963510 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:13.963480 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-drsdb_8510529c-89e5-48b1-bbc0-3f3ccd912896/dns/0.log" Jul 22 02:24:13.988625 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:13.988574 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-drsdb_8510529c-89e5-48b1-bbc0-3f3ccd912896/kube-rbac-proxy/0.log" Jul 22 02:24:14.069372 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:14.069335 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-92dh5_6a4a5252-c595-42f9-834a-0c26e27adaf8/dns-node-resolver/0.log" Jul 22 02:24:14.688626 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:14.688592 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-67d86b484b-dtqd8_ac01ce0a-32b3-458f-b280-b170447dcc21/registry/0.log" Jul 22 02:24:14.712359 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:14.712334 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j78lg_a184b54b-d2a6-4c78-aea6-714b9f0a6a1f/node-ca/0.log" Jul 22 02:24:15.948476 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:15.948444 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5975cd7b94-4zh5g_32e6dba1-d57f-4e7c-bed0-2264b16668ea/kube-auth-proxy/0.log" Jul 22 02:24:16.068571 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:16.068511 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-hqg49_c084ceb9-3866-47e4-888c-1a79d2a91e5f/payload-pre-processing/0.log" Jul 22 02:24:16.103634 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:16.103591 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-tsdbn_011f7fd0-7031-49ab-8eb2-ea5dc4a3f6ad/payload-processing/0.log" Jul 22 02:24:16.821466 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:16.821436 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ddbnn_8f8e967e-e9b7-47c7-83b0-9fedc09ce1d5/serve-healthcheck-canary/0.log" Jul 22 02:24:17.159285 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:17.159199 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-knlbt/perf-node-gather-daemonset-pfjvk" Jul 22 02:24:17.655785 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:17.655742 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vmk5j_1a297949-cd40-49e4-8643-0f3610080c45/kube-rbac-proxy/0.log" Jul 22 02:24:17.679666 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:17.679639 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vmk5j_1a297949-cd40-49e4-8643-0f3610080c45/exporter/0.log" Jul 22 02:24:17.702935 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:17.702909 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vmk5j_1a297949-cd40-49e4-8643-0f3610080c45/extractor/0.log" Jul 22 02:24:20.201489 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:20.201457 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-t4pkb_8b563dcf-e028-4eaa-b33c-164306146411/manager/0.log" Jul 22 02:24:20.436340 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:20.436312 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-j9twf_17416ded-8f9b-4822-8800-d65e3bf062b3/manager/0.log" Jul 22 02:24:20.586195 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:20.586163 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9cc45ddbb-qtw4v_2f068eb0-f72d-408d-af6c-ccc299da8009/manager/0.log" Jul 22 02:24:20.610006 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:20.609981 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-pnbqj_7fc9fa98-e5a6-4662-8a43-97c9373f5938/server/0.log" Jul 22 02:24:20.691945 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:20.691859 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-f724l_233fa350-ced9-4ddb-8ddd-9aa88e1d32d1/manager/0.log" Jul 22 02:24:20.698861 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:20.698840 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5975cd7b94-4zh5g_32e6dba1-d57f-4e7c-bed0-2264b16668ea/kube-auth-proxy/0.log" Jul 22 02:24:20.803926 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:20.803898 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-hqg49_c084ceb9-3866-47e4-888c-1a79d2a91e5f/payload-pre-processing/0.log" Jul 22 02:24:20.835906 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:20.835874 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-tsdbn_011f7fd0-7031-49ab-8eb2-ea5dc4a3f6ad/payload-processing/0.log" Jul 22 02:24:20.886925 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:20.886892 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c578697c6-lfq9r_7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135/manager/0.log" Jul 22 02:24:20.912313 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:20.912288 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c578697c6-lfq9r_7e2fecd4-5e9c-4d1a-a66e-65e2ffa3c135/copy-manifests/0.log" Jul 22 02:24:22.220458 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:22.220427 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5975cd7b94-4zh5g_32e6dba1-d57f-4e7c-bed0-2264b16668ea/kube-auth-proxy/0.log" Jul 22 02:24:22.312358 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:22.312333 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-hqg49_c084ceb9-3866-47e4-888c-1a79d2a91e5f/payload-pre-processing/0.log" Jul 22 02:24:22.338865 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:22.338796 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-tsdbn_011f7fd0-7031-49ab-8eb2-ea5dc4a3f6ad/payload-processing/0.log" Jul 22 02:24:27.256982 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:27.256955 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-wbzgd_d1b2434a-a884-4c5b-aff6-338544f7c60b/migrator/0.log" Jul 22 02:24:27.282081 ip-10-0-131-56 kubenswrapper[2561]: I0722 02:24:27.282052 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-wbzgd_d1b2434a-a884-4c5b-aff6-338544f7c60b/graceful-termination/0.log"